Double Check that Checklist

By Emilie Worthen Sommeroff
Managing Editor

Download Your Checklist

Do you know how to test your facility for possible ergonomic failings? It may not be as straight forward as you think.

If you don’t know by now, the National Ergonomics Conference and Exposition in New York at the end of May was there to remind you—“Ergonomics is not just good for your employees’ health and comfort. It’s good for your business.” Monitoring the effects of the work environment on employee well-being is a noble goal. Actually undertaking this task properly, however, so that valid, reliable results are obtained, can be a complicated process—at least in the delivery and analysis of symptom surveys and risk factor checklists.

During her session at the conference, Joy M. Ebben, Ph.D., CPE, a human factors and ergonomics technical advisor based in Alta Loma, California, discussed the possible errors involved in administering such questionnaires. While symptom surveys (used to identify possible musculoskeletal disorders—MSDs—that might otherwise go unnoticed) and risk factor checklists (used to determine potential MSD risk factors, such as awkward posture and excessive force) provide a standardized way of identifying possible problems. Ebben notes that these methods rely on responses that are subject to change and can be highly influenced by outside factors. In addition, analyzing the data is not a straightforward process.

Nevertheless, Ebben says, for those companies that cannot afford to consult and/ or employ a professional ergonomist, managing such a survey in-house “is a lot better than doing nothing.” For those considering administering either a symptom survey or risk factor checklist, Ebben offers a few suggestions.

One Size Does Not Fit All.

No one survey will satisfy all situations, which, says Ebben, is why it has been difficult to design a good comprehensive survey/ checklist for sale. A company, therefore, must develop a questionnaire that suits its specific job tasks and environment— often done by combining several checklists to create a form appropriate to the situation.

In developing this questionnaire, Ebben notes the importance of word choice. One must be aware of how descriptive words like “normal,” “sometimes,” and “easily adjustable” can mean different things to different people completing the survey/ checklist. Also seemingly quantitative questions like “how much,” “how many,” and “how often” can be answered unreliably, since respondents are often required to use estimates rather than actual measurements. The survey’s language must also be accessible to those taking the survey/ checklists—Will there be any non-English speakers completing the survey or respondents with remedial reading skills? Too many questions in too little time may tax an employee’s attention, while questions that are irrelevant to the survey/ checklisttakers will likely elicit unreliable answers. Notes Ebben, “They probably don’t know or don’t care about the question or answer, so not much thought is put into generating an answer.” Even details of format, grammar, and printing, or the way in which an interview is conducted, can motivate a respondent to take more time and therefore provide more accurate, useable data.

Administering the tests is equally sticky. According to Nation Institute for Occupational Safety & Health (NIOSH), tests should be anonymous, voluntary in nature, completed by an employee on his/ own work time. The administrator of the survey should also be aware of factors in the work environment—downsizing, striking, a disagreement with a superior— as these may influence the results of the survey.

Whether a questionnaire is self-administered or taken in a group will also affect the reliability of the results. A self-administered survey will take the least employee time and can be totally anonymous, says Ebben, but may also contribute to a low response rate or selection bias. (Only those with a problem may respond, hoping for action; those without a problem may ignore the questionnaire. This response bias may make the problem appear worse that it actually is.) A group-conducted survey may be prone to peer pressure, whereas a one-on-one administration is labor intensive and costly for the company.

It is also important to observe several employees doing the same task, in order to verify if the job is risky or the person is risky in that job.

Finally, once you have the completed surveys how do you analyze the data? In the case of symptom surveys, many more employees will report physical symptoms than those that will actually develop MSDs. Likewise, the ability of risk factor checklists—which, as mentioned above, are proactive and usually administered before there are noticeable problems in order to assess the potential for accumulative trauma illnesses—to differentiate non-hazardous and hazardous conditions has been debated. In order to verify the accuracy of the survey and checklist results, Ebben suggests that administrators of the test ask themselves two questions:

1. Is it valid? Is the survey actually measuring what you think it is measuring?
2. Is it reliable? Is it a device that provides consistent answers?

A survey or checklist should demonstrate four different types of validity:

• Face validity, which means a device appears to measure what it says it is measuring—Can respondents relate to the survey/checklist to the problem, or will they think the questions are off base?
• Content validity—Does a questionnaire deal with all aspects of the subject about which the conclusions are going to be drawn? Ebben uses the example of a mechanic that, in checking for the reasons a car won’t run, forgets to check the gas gauge.
• Criterion-related validity—How well do a survey’s results compare to variables that are direct measures, such as which workstations really do need more analysis and which do not?
• Construct validity—How compatible are the survey results with “certain explanatory concepts or qualities” such as the opinion of a consultant?

Likewise, the reliability of a survey must also be determined. The survey should yield the same results when given to the same respondent or observer repeatedly. “No matter how you look at it, you should get the same answer,” says Ebben. “The answers should not be dependent on who gives it.” Ebben notes that vague word choice, irrelevant questions, or varied observers, interviewers, or response methods may contribute to unreliable with results. Likewise, inconsistencies in the environment or time of day the device was administered will also affect its reliability. Did an employee take the first survey/ checklist in his office and the second at home?

Not The Final Word.

While an important first step in determining the existence of ergonomic problems in the workplace, symptom surveys and risk factor checklists should not be the final word on the subject. Ebben warns administrators against marching into the offices of senior management with the results of one round of surveys. Several follow-up surveys are essential—though keep in mind that the more time between the surveys, the more variables that have been introduced. In addition, try gathering the information

For advice on developing an ergonomics program, or for examples of surveys and checklists, see:

1. NIOSH, Elements of Ergonomics Programs, 1997. A comprehensive—and free—guide to developing a checklist and an ergonomics program, available at the NIOSH Web site: www.cdc.gov/niosh/homepage.html
2. National Safety Council, Ergonomics: A Practical Guide, 1993. A reader-friendly guide that discusses how to analyze work methods and environments, establish priorities, and resolve ergonomic problems.
3. Keyserling, W.M., Brouwer, M., & Silverstein, B.A. International Journal of Industrial Ergonomics, 1992, pp.298-299. A checklist for evaluating ergonomic risk factors resulting from awkward postures of the legs, trunk, and neck.
4. Keyserling, W.M., Brouwer, M., & Silverstein, B.A. International Journal of Industrial Ergonomics, 1993, pp.828-831. A checklist for evaluating ergonmic risk factors associated with upper extremity cumulative trauma disorders.
5. Humanics ErgoSystems, Inc, The Ergo Expert. A software package offering ergonomic workplace assessment. Visit www.humanics-es.com for an overview of the product.