Adonis Diaries

“Fundamentals of controlled experimentation methods” (Article 39, April 1st, 2006)

An experiment is designed to study the behavior of the values/responses of a dependent variable (for example data collected) as the values/stimuli of an independent variable/factor are changed, manipulated, or presented randomly or in fixed manner.

Besides the independent variables, there are other factors that need to be controlled because they could have serious effects on the behavior of the selected dependent variable, and if the researcher fails to hold these factors constant or fixed by appropriate techniques, procedures, instructions, experimental setting, and environmental conditions, the study will most likely have confounding results.

Controlled experimentation methods are versions of current simulations methods, but are essentially more structured and physically controlled.  In a nut shell, an experimental method is a series of controlled observations undertaken in an artificial situation with the deliberate manipulation of variables in order to answer specific hypotheses.

In general, a scientist plans, controls and describes all the circumstances surrounding his tests in a way they can be repeated by anyone else, a condition that offer dependability for validation.

The requisite of repeatability encourages artificial settings that can be controlled, especially because:

1. The participants/subjects in the experiment are not usually involved or engrossed in their tasks,

2. and it enables a scientist to try combinations of conditions that have not yet occurred.

Controlled experiments are time-consuming, expensive, and require a staff of skilled researchers and investigators so that they are conducted for basic research, publishing scientific papers, and when sponsored by deep pocket private companies and well-funded public institutions.

            There are different types of experiments, some are designed to extract cause and effects among the variables and, especially their interactions in the performance of a system, and others are not so well structured and are intended to explore a phenomenon at an initial phase in order to comprehend the subject matter…

Experiments varies in their design purposes and levels of control:  there are experiments on inanimate objects, natural phenomena that follow fixed trends and do not change much with time, and experiments using human subjects to select the better performing system or product, and experiments intended to study the cognitive concepts of people such as attitudes, mental abilities, problem solving aptitudes, attention span and the like.

The next article entitled “Controlled experimentation: natural sciences versus people’s behavior sciences” is intended to compare the complexity, differences, and levels of difficulties among the various experiments.

This article is striving to establish the fundamental processes or necessary structured steps to conducting a controlled experiment. In the spectrum of complexity, innovation and difficulty, the experiments in natural sciences are the easiest, and psychology experiments the hardest. Within the human-targeted research, fall experiments in the disciplines of agriculture, econometrics, education, social sciences, and marketing.

Early researchers in the phenomenon of electricity had to experiment with simple methods of one dependent and one independent variable, rudimentary equipments, and to rely on an exploratory knowledge of how electricity works and what are the factors that cause definite change in the behavior of certain criteria.

For example, scientists observed that there are relationships among voltage/power, the intensity of the current and the material the current is flowing through, then a scientist set up an experiment to study how the voltage changes when the intensity of the current varies or when the resistance of a material changes.  By conducting several experiments, first by working with a specific conducting material, thus fixing the resistance, and varying the intensity of the current and repeating this simple experiment many times and, second by fixing the current at a certain level and working with different kinds of conducting materials, then the scientist managed to observe a steady mathematical relationship among these three variables. As the body of knowledge in electricity expanded and more experiments were undertaken, the physical science of electricity discovered many more factors that entered into the mathematical relationship with varying degrees of importance and consequences.

            Obviously, physical scientists can now enjoy more powerful, time-saving, and effective experimental designs that can employ several independent variables and several dependent variables in the same experiment, thanks to the development in statistical/mathematical modeling and the number crunching computers.

These developments in controlled experimentations allow observations of the interactions among the various variables simultaneously, if physical scientists deign to apply them!

Controlled experimentation methods have a set of requisite structured steps that are common to both natural and social studies. Usually, an investigator has to review the research papers on the topic to be investigated, sort out the articles that are scientifically valid and experimentally sound, consider the variables that have been satisfactorily examined and those that were controlled, or not even considered…

Or the researcher may explore the topic by systematic observation of the problem, then he has to propose a hypothesis that could be rejected but never accepted no matter how often it was not rejected, then has to conceive a design for the experiment such as the types, numbers of variables, their levels, and how to manipulate the trials, then he has to decide on the best method for selecting the subjects, the materials, or products to be tested, the setting conditions, the procedures, the operations or tasks to be performed, the instructions, the equipments, the appropriate statistical model, then conducting the experiment, running the data, analyzing the results, interpreting the results, and finally providing guidelines or practical suggestions to be applied in engineering projects.

The motors of statistical packages used to analyze data are mathematical models or sets of algebraic equations with as many equations as unknown variables and relying on the two main statistical concepts of means and variances among data.

The purpose of controlled experimentation methods is to strictly control systematic errors due to biases and then to sort out the errors that are due to differences among the independent variables and those introduced randomly by human variability.  Once the size of random errors is accounted for then it is possible to study the relationships among the independent variables and to claim that a hypothesis could or could not be rejected at a criterion level of statistical significance, set frequently at 5%.  This criterion level of 5% of statistical significance means that there is still a 5% chance that an amount of random error might be the cause in the differences of the results.

Types of errors and mistakes committed in controlled experimentations will be reviewed in article #45.  However, it is important to differentiate between evaluation/testing methods and strictly controlled experimentation.

In human factors discipline, evaluation methods are applied to compare the effectiveness of several products or systems by measuring end-users behaviors, like/dislike, acceptance/rejection, or satisfying rules and regulations with the purpose that management would be able to decide on the choice among the products offered within specifications.

Controlled experimental methods are mainly applied to study the cause and effects of the main factors on objective measurements that represents valid behaviors of representative samples of end-users with the purpose of reaching design guidelines for products or systems planned for productions.