Adonis Diaries

Human Factors in Engineering (Article #43)

“Controlled experimentation versus Evaluation and Testing methods”

The methods of evaluation and testing are basically subset of controlled experimentation: They differ on the purpose, the types of relevant variables, the level of control, and the logical and chronological phase in a scientific study.

Usually, any scientific undertaking need a theoretical model, the variables and procedures are extracted from exploratory studies, earlier research, and observations that might discover analogies with other more developed concepts.

The next phase is to conduct varied controlled experimentations in artificial settings for repeatability by other researchers with the purpose of generating the main factors/independent variables that have significant effects on the behavior of the selected and valid dependent variables.

The results of controlled experimentations may be used to either validate/modify a theoretical model that could offer explanations for the behavior/concept under study, or provide practical design guidelines for manufacturing efficient products/processes or developing systems such as educational, training, or safety programs.

The logical next phase is to test the theoretical model, product, or system in a real world context or real life setting in order to validate the extent of its effective performance when operated by target users.  Consequently, the purpose of testing is to study the effectiveness of the specifications of the product when applied in a real life setting so that further modifications and redesigns are undertaken; the question is: “Does the product/system actually works? Does it deliver as claimed?”  

Testing methods are the debugging procedures; The logical criteria/dependent variables are the type of errors, malfunction, dysfunctional mismatch with operators’ skills, number of errors, seriousness of errors, probability distribution of occurrence of defects, and training time required. The independent variables/factors are the comparisons among the kind of real world settings, types of operators, sort of idiosyncrasies, and the processes.

The final phase is the evaluation among competing products/systems to enable management deciding on the better choice on the market that satisfy its requirements.

The question is: “Can the product/system be accepted, acquired, and purchased?” 

What is compared is not just the product but the whole package that comes with it; mainly, the added features, the instruction manuals, audio-visual training programs, maintainability, availability of spare parts, client-marketing responses to claims, malfunctions, misuses, plaintiffs’ cases in product liability, and on time delivery.

Although the final output of an evaluation is the decision of management of (Yes/No), there are a prior output for rating the various specifications with relative weights attached to each factor and thus, the evaluation method is a series of evaluations for each specification that differ in the level of control and objectivity and measurement accuracy of the responses.

For example, in comparing warning alarm systems, management might want a system that forces all workers to vacate within standard safety duration and within an acceptable frequency of false warning activation.

Any additional features such as visual feedback in police cars, possibility of connecting to a central fire station, and regular maintenance can tip the balance toward a product that was as performing as others.