How to Measure without measuring? Are you measuring what’s Not important?
Posted by: adonis49 on: May 27, 2018
How to Measure without measuring? Are you measuring what’s Not important?
Note: In experiment, testing or evaluation, what is being measured is called dependent variable.
Seth Godin had to say this June 5, 2013 on “Measuring without measuring”
As an organization grows and industrializes, it’s tempting to simplify things for the troops.
Find a goal, make it a number and measure it until it gets better. In most organizations, the thing you measure is the thing that will improve.
Colleges decided that the SAT were a useful shortcut, a way to measure future performance in college.
And nervous parents and competitive kids everywhere embraced the metric, and stick with it, even after seeing (again and again) that all the SAT measures is how well you do on the SAT (Nothing to do with intelligence or effective performance).
It’s easier to focus on one number than it is to focus on a life.
Paypal and Chase and countless other organizations do precisely this: they figure out a metric, decide it’s important and then create a department to improve that metric.
Consider the Chase Fraud Prevention department.
It costs a credit card company (and especially their merchants) a lot of money when fraudulent charges are made, because they often have to eat the cost.
So this department of thousands of people works to make the number of fraudulent charges go down at the same time they keep expenses low.
Which sounds great until you realize that the easiest way to do this is to flag false positives, annoy honest customers and provide little or no fallback when a mistake is made.
Simple example: I regularly get an automated phone call from the bank with an urgent warning. But even when I answer the phone, the system doesn’t let me ring through to an operator.
Instead, I have to write every detail down, then call, wait on hold, prove it’s me, type in all the information, and THEN explain to them that in fact, the charge was mine.
And this department has no incentive to fix this interaction, because ‘annoying’ is not a metric that the bosses have decided to measure.
Someone is busy watching one number, but it’s the wrong one.
Or consider the similar problem at Paypal.
Stories of good (or great) customers being totally shut down, sometimes to the point of bankruptcy, are legion. There may be people at Paypal who care about this, but the security people don’t. That’s because they’re not measuring the right thing.
Measurement is fabulous. Unless you’re busy measuring what’s easy to measure as opposed to what’s important.
Adonis Diaries 
Leave a Reply