Adonis Diaries

Main errors and mistakes in controlled experimen

Posted on: October 17, 2008

Article #45, (April 15, 2006)

“Main errors and mistakes in controlled experimentations”

This article targets controlled experimentations investigated in the social and psychological sciences or when human participants or living subjects such as animals are used.  The natural physical science experimentations are much more immune to generating random errors or systematic bias errors because these kinds of errors are due principally to the variability of human subjects.

As a reminder, controlled experimentations study the variables/factors that may cause statistically significant effects on the behavior of a set of valid and representative dependent variables in order to validate a theoretical model that describes and defines the processes and the working of a phenomena or a concept. The basis of the analyses of the data obtained in the experiment is to sort out the various types of errors that result from the differences and variety of effects on the behavior of the system/model.  What the scientist wants is the effects of the factors under study and their interactions due to the experimental differences and this can only be done by trying to control errors generated from systematic biases and by accounting for the size of random errors introduced in the experiment due primarily to the changing behavior of the subject/participants from moment to moment, activity to activity, and trial to trial.

            The major step that can achieve this goal is a sound design for the experiment which is basically an experimental structure of the model and a milestone toward the proper direction of the investigation. This planning phase of a formal pattern for collecting observations has 3 principal objectives: first, to identify all the variables that the investigator decided to test, second to decide on the number of subjects and trials to be tested, and third to decide on the techniques for the order and schedule of the experimental trials, primarily to counterbalance the trials sequence in order to control any systematic biases.

            Systematic errors are virulent viruses that can pop up at every step of an experiment. For example, if we were testing the design of bottle lids and the subjects were consistently trying to twist lid #1 and then lid #2 we will have serious doubt if the difference in the observations are not due to fatigue from the effort of opening lid #1, the excitement to exerting more efforts with lid #2, or some other physical or psychological changes in attitudes and expectations. Selecting subjects by not considering their gender, age, skills, level of education and other differences in capabilities and limitations is a serious cause for systematic errors if these variability are not factored in the design.  Procedures and instructions may be sources for systematic errors if they were not properly tested with a different set of subjects for comprehension, clarity, and potential mistakes.  The attitude, voice, and gesture of the investigator may offer the subjects clues that generate systematic errors. The best technique to avoiding systematic errors is to counterbalance trials by using scheduled order of trials; for example the second test would start with lid #2 then lid #1.  Frequently, counterbalancing technique might be cumbersome, impractical, or not feasible due to lack of participants then the next best technique is to randomize the order of trials or subjects by assigning test trials or the designated subjects according to a list of numbers generated from a mathematically randomized lists of numbers.

Consequently, systematic errors that are the product of biases affect the internal validity of an experiment and the expert investigator’s job is to control the introduction of these systematic biases through counterbalancing or randomization of stimuli, trials, selection of subjects, design of the experiment, testing of instruction materials and procedures. 

            Once systematic errors are strictly controlled then the design of an experiment should yield a measure of random error (residual) in the experiment due to human variability between trials and between themselves.  The experimental error must come from the same experiment as is used to test the primary factors of interest.

Another problem that can affect the validity of the results of an experiment is when an important factor that can have a substantial effects on the dependent variables/data was not considered or the investigator failed to control its effects; for example, when the products under study are sensible to temperature and the investigator did not conduct the trials within an acceptable range of temperatures.  In such a case, it would be difficult to claim that the relationships are due to the variables manipulated in the experiment or to the variable that was not accounted for; these kinds of invalid experiments are said to have confounding effects.  The investigator has thus to know the subject matter thoroughly in order to avoid dubious results and be candid about his mistakes and offer justifications if the effects of the confounding factor is proven not be significant enough in this particular experiment.  That is why it is important to consider a post experimentation phase targeted at validating any anomalies not designed purposely in the experiment. Once all the main factors are accounted for the investigator can control the factors that he is not interested to test in his experiment and consequently he will control them either in the setting conditions, the procedures, the instructions, and the selection of the participants.

There are two other types of errors; error of type I is when the investigator does not reject a hypothesis when he should have, and the type II error when the investigator reject a hypothesis when in fact he shouldn’t have.  The power of an experiment is represented by the level of type II error selected:  the least it is the more powerful the experiment is and that is why many subjects are employed or trials are conducted.

While a sound design for an experiment allows the statistical package to compute the various random errors such as the within and the between errors and to decide whether to reject or not reject a hypothesis at a criterion level of statistical significance, eliminating or avoiding the errors of systematic biases and confounding effects of uncontrolled variables are solely the  responsibility of a professional investigator who know the subject matter with a long an practical expertise of conducting controlled experimentations.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

adonis49

adonis49

adonis49

October 2008
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  

Blog Stats

  • 1,473,958 hits

Enter your email address to subscribe to this blog and receive notifications of new posts by email.adonisbouh@gmail.com

Join 807 other followers

%d bloggers like this: