Posts Tagged ‘dependent variable’

How to Measure without measuring? Are you measuring what’s Not important?

Posted on: May 27, 2018

How to Measure without measuring? Are you measuring what’s Not important?

Note: In experiment, testing or evaluation, what is being measured is called dependent variable.

Seth Godin had to say this June 5, 2013 on “Measuring without measuring”

As an organization grows and industrializes, it’s tempting to simplify things for the troops.

Find a goal, make it a number and measure it until it gets better. In most organizations, the thing you measure is the thing that will improve.

Colleges decided that the SAT were a useful shortcut, a way to measure future performance in college.

And nervous parents and competitive kids everywhere embraced the metric, and stick with it, even after seeing (again and again) that all the SAT measures is how well you do on the SAT (Nothing to do with intelligence or effective performance).

It’s easier to focus on one number than it is to focus on a life.

Paypal and Chase and countless other organizations do precisely this: they figure out a metric, decide it’s important and then create a department to improve that metric.

Consider the Chase Fraud Prevention department.

It costs a credit card company (and especially their merchants) a lot of money when fraudulent charges are made, because they often have to eat the cost.

So this department of thousands of people works to make the number of fraudulent charges go down at the same time they keep expenses low.

Which sounds great until you realize that the easiest way to do this is to flag false positives, annoy honest customers and provide little or no fallback when a mistake is made.

Simple example: I regularly get an automated phone call from the bank with an urgent warning. But even when I answer the phone, the system doesn’t let me ring through to an operator.

Instead, I have to write every detail down, then call, wait on hold, prove it’s me, type in all the information, and THEN explain to them that in fact, the charge was mine.

And this department has no incentive to fix this interaction, because ‘annoying’ is not a metric that the bosses have decided to measure.

Someone is busy watching one number, but it’s the wrong one.

Or consider the similar problem at Paypal.

Stories of good (or great) customers being totally shut down, sometimes to the point of bankruptcy, are legion. There may be people at Paypal who care about this, but the security people don’t. That’s because they’re not measuring the right thing.

Measurement is fabulous. Unless you’re busy measuring what’s easy to measure as opposed to what’s important.

Controlled experiment methods: In Human Factors studies

Posted on: July 16, 2009

“Fundamentals of controlled experimentation methods” (Article 39, April 1st, 2006)

An experiment is designed to study the behavior of the values/responses of a dependent variable (for example data collected) as the values/stimuli of an independent variable/factor are changed, manipulated, or presented randomly or in fixed manner.

Besides the independent variables, there are other factors that need to be controlled because they could have serious effects on the behavior of the selected dependent variable, and if the researcher fails to hold these factors constant or fixed by appropriate techniques, procedures, instructions, experimental setting, and environmental conditions, the study will most likely have confounding results.

Controlled experimentation methods are versions of current simulations methods, but are essentially more structured and physically controlled.  In a nut shell, an experimental method is a series of controlled observations undertaken in an artificial situation with the deliberate manipulation of variables in order to answer specific hypotheses.

In general, a scientist plans, controls and describes all the circumstances surrounding his tests in a way they can be repeated by anyone else, a condition that offer dependability for validation.

The requisite of repeatability encourages artificial settings that can be controlled, especially because:

1. The participants/subjects in the experiment are not usually involved or engrossed in their tasks,

2. and it enables a scientist to try combinations of conditions that have not yet occurred.

Controlled experiments are time-consuming, expensive, and require a staff of skilled researchers and investigators so that they are conducted for basic research, publishing scientific papers, and when sponsored by deep pocket private companies and well-funded public institutions.

There are different types of experiments, some are designed to extract cause and effects among the variables and, especially their interactions in the performance of a system, and others are not so well structured and are intended to explore a phenomenon at an initial phase in order to comprehend the subject matter…

Experiments varies in their design purposes and levels of control:  there are experiments on inanimate objects, natural phenomena that follow fixed trends and do not change much with time, and experiments using human subjects to select the better performing system or product, and experiments intended to study the cognitive concepts of people such as attitudes, mental abilities, problem solving aptitudes, attention span and the like.

The next article entitled “Controlled experimentation: natural sciences versus people’s behavior sciences” is intended to compare the complexity, differences, and levels of difficulties among the various experiments.

This article is striving to establish the fundamental processes or necessary structured steps to conducting a controlled experiment. In the spectrum of complexity, innovation and difficulty, the experiments in natural sciences are the easiest, and psychology experiments the hardest. Within the human-targeted research, fall experiments in the disciplines of agriculture, econometrics, education, social sciences, and marketing.

Early researchers in the phenomenon of electricity had to experiment with simple methods of one dependent and one independent variable, rudimentary equipments, and to rely on an exploratory knowledge of how electricity works and what are the factors that cause definite change in the behavior of certain criteria.

For example, scientists observed that there are relationships among voltage/power, the intensity of the current and the material the current is flowing through, then a scientist set up an experiment to study how the voltage changes when the intensity of the current varies or when the resistance of a material changes.  By conducting several experiments, first by working with a specific conducting material, thus fixing the resistance, and varying the intensity of the current and repeating this simple experiment many times and, second by fixing the current at a certain level and working with different kinds of conducting materials, then the scientist managed to observe a steady mathematical relationship among these three variables. As the body of knowledge in electricity expanded and more experiments were undertaken, the physical science of electricity discovered many more factors that entered into the mathematical relationship with varying degrees of importance and consequences.

Obviously, physical scientists can now enjoy more powerful, time-saving, and effective experimental designs that can employ several independent variables and several dependent variables in the same experiment, thanks to the development in statistical/mathematical modeling and the number crunching computers.

These developments in controlled experimentations allow observations of the interactions among the various variables simultaneously, if physical scientists deign to apply them!

Controlled experimentation methods have a set of requisite structured steps that are common to both natural and social studies. Usually, an investigator has to review the research papers on the topic to be investigated, sort out the articles that are scientifically valid and experimentally sound, consider the variables that have been satisfactorily examined and those that were controlled, or not even considered…

Or the researcher may explore the topic by systematic observation of the problem, then he has to propose a hypothesis that could be rejected but never accepted no matter how often it was not rejected, then has to conceive a design for the experiment such as the types, numbers of variables, their levels, and how to manipulate the trials, then he has to decide on the best method for selecting the subjects, the materials, or products to be tested, the setting conditions, the procedures, the operations or tasks to be performed, the instructions, the equipments, the appropriate statistical model, then conducting the experiment, running the data, analyzing the results, interpreting the results, and finally providing guidelines or practical suggestions to be applied in engineering projects.

The motors of statistical packages used to analyze data are mathematical models or sets of algebraic equations with as many equations as unknown variables and relying on the two main statistical concepts of means and variances among data.

The purpose of controlled experimentation methods is to strictly control systematic errors due to biases and then to sort out the errors that are due to differences among the independent variables and those introduced randomly by human variability.  Once the size of random errors is accounted for then it is possible to study the relationships among the independent variables and to claim that a hypothesis could or could not be rejected at a criterion level of statistical significance, set frequently at 5%.  This criterion level of 5% of statistical significance means that there is still a 5% chance that an amount of random error might be the cause in the differences of the results.

Types of errors and mistakes committed in controlled experimentations will be reviewed in article #45.  However, it is important to differentiate between evaluation/testing methods and strictly controlled experimentation.

In human factors discipline, evaluation methods are applied to compare the effectiveness of several products or systems by measuring end-users behaviors, like/dislike, acceptance/rejection, or satisfying rules and regulations with the purpose that management would be able to decide on the choice among the products offered within specifications.

Controlled experimental methods are mainly applied to study the cause and effects of the main factors on objective measurements that represents valid behaviors of representative samples of end-users with the purpose of reaching design guidelines for products or systems planned for productions.

Fundamentals of controlled experimentation methods: What’s Human Factors in design?

Posted on: October 14, 2008

What is Human Factors in design? (Article #39, April 1st, 2006)

Fundamentals of controlled experimentation methods

An experiment is designed to study the behavior of the responses of subjects (dependent variables or what are measured as performance), as the values/stimuli of an independent variables or factors are changed, manipulated, or presented randomly or in fixed manner.

There are other factors that need to be controlled because they could have serious effects on the behavior of the selected dependent variables, and thus are held constant or fixed by appropriate techniques, procedures, instructions, experimental setting, and environmental conditions.

Controlled experimentation methods are versions of current simulations methods, but are essentially more structured and controlled.

In a nut shell, an experimental method is a series of controlled observations undertaken in an artificial situation with the deliberate manipulation of variables in order to answer specific hypotheses.

In general, a scientist plans, controls and describes all the circumstances surrounding his tests, in a way they can be repeated by anyone else, which offer dependability for validation.

The requisite of repeatability encourages artificial settings that can be controlled, especially because the participants/subjects in the experiment are not usually involved or engrossed in their tasks, and because it enables a scientist to try combinations of conditions that have not yet occurred.

Controlled experimentation are time-consuming, expensive, and require a staff of skilled researchers and investigators so that they are conducted for basic research, publishing scientific papers, and when sponsored by deep pockets private companies and well-funded public institutions.

There are different types of experiments, some are designed to extract cause and effects among the variables, and especially their interactions in the performance of a system.

Others experiments are not so well structured and are intended to explore a phenomenon at an initial phase.

Experiments varies in their design purposes and levels of control:  there are experiments on inanimate objects or natural phenomena that follow fixed trends and do not change much with time.

Experiments using human subjects in order to select the better performing system or product, and experiments intended to study the cognitive concepts of people such as attitudes, mental abilities, problem solving aptitudes, attention span and the like are very complex, very intricate,  highly time-consuming and expensive to conduct.

The next article entitled “Controlled experimentation: natural sciences versus people’s behavior sciences” is intended to compare the complexity, differences, and levels of difficulties among the various experiments.

In the spectrum of complexity, innovation, and difficulty the experiments in natural sciences are the easiest, and psychology the hardest; within that spectrum fall experiments in the disciplines of agriculture, econometrics, education, social sciences, and marketing.

Early researchers in the phenomenon of electricity had to experiment with simple methods of one dependent and one independent variable, rudimentary equipments, and to rely on an exploratory knowledge of how electricity works and what are the factors that cause definite change in the behavior of certain criteria.

For example, scientists observed that there are relationships among voltage/power, the intensity of the current and the material the current is flowing through, then a scientist set up an experiment to study how the voltage changes when the intensity of the current varies or when the resistance of a material varies.  By conducting several experiments, first by working with a specific conducting material, thus fixing the resistance, and varying the intensity of the current and repeating this simple experiment many times and, second by fixing the current at a certain level and working with different kinds of conducting materials, then the scientist managed to observe a steady mathematical relationship among these three variables.

As the body of knowledge in electricity expanded and more experiments were undertaken, the physical science of electricity discovered many more factors that entered into the mathematical relationship with varying degrees of importance and consequences.

Obviously, physical scientists can now enjoy more powerful, time-saving, and effective experimental designs that can employ several independent variables and several dependent variables in the same experiment thanks to the development in statistical/mathematical modeling and the number crunching computers; these developments in controlled experimentation allow observations of the interactions among the various variables simultaneously, if physical scientists deign to apply them!

Controlled experimentation methods have a set of requisite structured steps that are common to both natural and social studies. Usually, an investigator has to review the research papers on the topic to be investigated, sort out the articles that are scientifically valid and experimentally sound, consider the variables that have been satisfactorily examined and those that were controlled, or not even considered.   Or the scientist may explore the topic by systematic observation of the problem, then he has to propose a hypothesis that could be rejected, but never accepted no matter how often it was not rejected.  He has to conceive a design for the experiment such as the types, numbers of variables, their levels, and how to manipulate the trials, then he has to decide on the best method for selecting the subjects, the materials, or products to be tested, the setting conditions, the procedures, the operations or tasks to be performed, the instructions, the equipments, the appropriate statistical model.

Then the scientist has to conduct the experiment, running the data, analyzing the results, interpreting the results, and finally providing guidelines or practical suggestions to be applied in engineering projects.

The motors of statistical packages used to analyze data are mathematical models or sets of algebraic equations with as many equations as unknown variables and relying on the two main statistical concepts of means and variances among data.

The purpose of controlled experimentation methods is to strictly control systematic errors due to biases and then to sort out the errors that are due to differences among the independent variables and those introduced randomly by human variability.  Once the size of random errors is accounted for then it is possible to study the relationships among the independent variables and to claim that a hypothesis could or could not be rejected at a criterion level of statistical significance, set frequently at 5%.  This criterion level of 5% of statistical significance, means that there is still a 5% chance that an amount of random error might be the cause in the differences of the results.

Types of errors and mistakes committed in controlled experimentation will be reviewed in article #45.   However, it is important to differentiate between evaluation/testing methods and strictly controlled experimentation. In human factors discipline, evaluation methods are applied to compare the effectiveness of several products or systems by measuring end-users behaviors, like/dislike, acceptance/rejection, or satisfying rules and regulations with the purpose that management would be able to decide on the choice among the products offered within specifications.

Controlled experimental methods are mainly applied to study the cause and effects of the main factors on objective measurements that represents valid behaviors of representative samples of end-users with the purpose of reaching design guidelines for products or systems planned for productions.

Blog Stats

• 1,521,894 hits