EAP Handbook - Chapter 5 - PROGRAM EVALUATION

V. PROGRAM EVALUATION

The subject of program evaluation becomes increasingly important during periods of transition. In today's era of shrinking resources, when the "old way" of conducting business has become obsolete, and no "new way" has yet emerged, discovering what does and does not work helps lessen the impact of continuous adjustment. Performing program evaluations can provide much needed direction with regard to current operations and future courses of action, both in general, and for the employee assistance field specifically.

The increasing diversity in population demographics in the United States, with its attendant change in the nature of social problems and organizational evolution, will only speed the process of change EAPs have undergone since the 1980s. These issues will also undoubtedly affect the assessment of EAPs in the future (Tramm, 1990).

However, EAP evaluation remains a controversial subject. Some professionals in the field view any request for evaluation as an attempt to cast doubt on the legitimacy of the EAP concept and as an excuse to withhold management support unless evaluation results prove acceptable. On the other hand, too many organizations accept their EAPs on faith and rarely ask for operational evaluation of any kind.

Despite the concerns employee assistance professionals have had, many have initiated evaluation projects to document program operations on their own (Blair, 1990). For the most effective and useful study, however, all parties involved in evaluation must view the process less as an investigation than as an opportunity. Program evaluation can even be incorporated into a total quality management approach that utilizes statistical (as well as other) analyses to improve systems. As with managed care, employee assistance programs exist, at least in part, to provide the best possible service at the most effective price, and program evaluation affords a useful tool for assessing quality of care and suggesting strategies for meeting specific, well-defined goals in a financially efficient manner.

Most of the numerous efforts to evaluate EAPs to date have not been rigorously executed and have failed to employ sophisticated methods of analysis. One exception is a recent study conducted for the EAP of California Edison Company (Amaral and Kelly, 1989) which has won high marks for its excellent research methodology and advanced level of data analysis. Efforts like this one will contribute substantially to the status and efficiency of the employee assistance field.

This chapter examines why evaluation of EAPs is so important, why resistance to the process exists, how to facilitate evaluation procedures (including various roles EAP professionals can play), and the key components of a successful study. The chapter concludes with an effective and comprehensible approach for conducting EAP evaluations.

A. Reasons for Program Evaluation

Program evaluation is a broad area of research activity devoted to collecting, analyzing, and interpreting vital information on the needs for implementation and impact of a given undertaking for the purpose of improvement and/or justification of the project (Kim, 1988). The first, and perhaps most compelling reason for conducting assessments of a program's effectiveness are almost impossible without evaluation. A proper study will describe the activities and outcomes of an organization's EAP quantitatively and qualitatively.

Organizations seeking ways to reduce expenditures on benefits will also find program evaluation useful. Such a process can highlight the specific contributions of employee assistance and justify the cost of the EAP. Determining return on investment is always a sound business practice, but it becomes essential during difficult economic times. Moreover, the advent of managed care and the perceived overlap of some services between managed care and employee assistance demands examination of the value that employee assistance programming adds to an organization (Amaral and Kelly, 1989).

Another reason for conducting EAP evaluations is to determine whether such efforts have fulfilled their purpose and duties properly and well, thus providing quality assurance. A thorough survey should identify areas in need of improvement and suggest alternatives for accomplishing stated goals (Blair, 1990).

Organizations can also identify problems that might otherwise remain hidden through these procedures. For example, EAP evaluation can be included in broader survey efforts that examine a wide variety of organizational undertakings (e.g., all HR-related activities). Data analyses of program evaluation in a specific non-EAP area will often reveal overall emerging needs (a change in the benefits plan, for example) that affects EAPs as well (Kim, 1988).

Some organizations may wish to compare their EAP to some predefined benchmark. This quality improvement method has received such attention during the 1990s. Popularized by Xerox's 1979 study of the methods of the Japanese and other industry competitors, it now stands as one of the criteria used for scoring the Malcolm Baldridge National Quality Award. After first emerging in the manufacturing sector, benchmarking has since cropped up in the service and financial industries, the government sector, and, most recently, in the health care sector.

Benchmarking offers a systematic assessment of how an organization's business practices compare with a standard of best-in-class practices in a certain area. Companies can thereby establish goals and strategies for improvement, as well as identify and adopt methods other organizations use to achieve excellence in performance.

Benchmarking allows organizations evaluating their EAPs to compare programs with highly regarded EAPs in similar industries. It can also be an internal process. Periodic evaluations can be used to record program practices and measure their rate of improvement over time. These kinds of evaluations serve as either historical reference points, an approximate current base case, or a means of setting future goals. Without baseline or benchmark data, EAP assessment will only describe current practices without tracking improvements or making comparisons (Ulrich, Brockbank, and Yeung, 1989; Jennings and Westfall, 1992).

Finally, EAP evaluation can assist an organization with regard to meeting and demonstrating legal and fiscal responsibilities (Walker, 1989). For example, evaluation can document an organization's efforts to comply with drug-free workplace guidelines while providing a resource for those suffering from drug-related problems.

B. Reasons for Resistance to Program Evaluation

The above account provides undeniable evidence for the value of program evaluation and begs the question of why evaluations have not been performed on a regular basis. The cost of these procedures appears to be one primary explanation. The resources required for proper study discourage many HR and EAP managers. Further, since HR professionals typically lack training in research methods, many fail to anticipate a proposed evaluation's value. This, however, has begun to change. Increasing professionalization of the EAP field and the requiring of academic credentialing for employee assistance practitioners has boosted recognition of program evaluation as a valuable tool (Tramm, 1990). Moreover, EAPA now includes periodic program evaluation as one of the elements in its standards for EAPs (EAPA, 1990).

However, costs are not the only deterrent to program evaluation. The second most often cited reason for not pursuing EAP evaluation is confidentiality (Knott and Knott, 1989). An evaluation process that compromises an EAP's confidentiality may well undermine the program's credibility. But evaluations do not necessarily pose a threat to confidentiality as some often contend (Farkas and Hirshberg, 1990).

A number of alternative methods allow for the conducting of studies of a program's effectiveness without sacrificing its essentially classified nature. Addressing this issue before implementation of an EAP program makes evaluation efforts easier to set up. That way the EAP can prepare itself for future evaluation by taking steps to protect its clients' privacy. For example, names of clients of the employee assistance program can be easily masked with assigned numbers. Employee assistance professionals and/or the outside contractor selected to conduct the evaluation can handle all contact with clients, thereby screening organizational representatives from involvement. Moreover, most information about program evaluation is solicited anonymously. Names may be requested, but the program has no obligation to supply them. Standard practice in research investigations dictate the reporting of findings in terms of group data. This process makes obtaining information about a specific individual difficult, unless the appraisal has as a stated goal the examination of a particular person.

EAP requests to access records of other organizational functions, such as medical plans or benefits, may raise a new set of confidentiality issues. However, the reaction of other organizational functions to such requests usually has more to do with the EAP's relationship to related functions within the organization than with confidentiality concerns. Building alliances and relationships with other internal departments before specific circumstances arise, therefore, will promote proper cooperation (Blair, 1990).

Useful program evaluations can be managed even after an EAP has become fully operational without granting permission for such procedures in advance. Employee assistance personnel can contact past clients to secure their participation. This process may be time-consuming, however, and reduce the number of participants in the survey, effectively skewing the results. On the other hand, many clients often seem pleased to receive an additional contact from the EAP to have their opinions solicited (Blair, 1990; Farkas and Hirshberg, 1990).

The hesitancy of employee assistance staffs to cooperate has also contributed to the irregularity of program evaluations (Knott and Knott, 1989). Naturally, some concern exists about negative outcomes and no guarantee exists that all findings will be positive. However, most EAP professionals do want to deliver high quality service, and they recognize that their program will be more successful if problems are identified. Envisioning evaluations as trouble-shooting helps allay fears about any negative portrayal of program activities.

Many employee assistance personnel are already overworked and have limited resources, and these factors may make them reluctant to participate in evaluations. They view the extra work necessary to ensure successful evaluation as one more task they must handle. Extra work may include, for example, supplying more descriptive and understandable data. Given the limited support systems most EAPs work with, data often gets backlogged, incorrectly entered, badly organized or all of the above. Sorting out all that information can consume valuable time, a commodity most employee assistance professionals find in short supply.

Measurement problems can also make EAP evaluations more difficult to perform than other assessment efforts. Problems that often come to the attention of EAPS, such as employee morale, the quality of the work environment, and the long-term impacts of people with personal problems on the organization's overall health are difficult to quantify and therefore measure effectively. Comparisons to other programs can be problematic as well since EAPs often have different approaches and function under different models at various points in time. The considerable turnover in employee assistance staffs, too, makes replication of program evaluation methodology more difficult than it might be under more stable conditions.

Even with all of the above problems, though, and despite resistance from HR managers due to the perceived drain on resources, even brief and inexpensive evaluation projects can provide meaningful assessments of how an EAP is implementing and meeting its goals (Blair, 1990; Farkas and Hirshberg, 1990). Further, such evaluations can be structured to explore other organizational questions as well. Always, the goals of any evaluation need to be defined up front.

C. Conducting Process and Outcome Program Evaluations

Many organizations do, at some point, decide to conduct program evaluations. Sometimes problems surface and prompt investigation. More often, EAPs themselves press for some sort of survey to track their own progress. Several strategies will help predispose an evaluation to success. The first involves setting the stage. The agency conducting such a procedure needs to obtain as much commitment and cooperation from the organization and the EAP staff as possible right from the beginning. Securing commitment and cooperation becomes easier when (Mani, 1992):

• There are stated goals, i.e. the organization defines and agrees upon what it wants the evaluation to accomplish;

• The goals thus defined and agreed upon are stated in measurable terms;

• Information necessary for the evaluation is both recorded and retrievable;

• The evaluation process and tools reflect the goals of the assessment;

• All participants understand their roles in the process prior to the design of the study; and

• The evaluation process will provide useful information to all participants.

Even if minimal resources, lack of data, or any other difficulty limits the possible accomplishment of an initial program evaluation, the process can still provide valuable information. A well defined evaluation with reasonable scope will, at the very least, yield a benchmark for comparison with future evaluation efforts and provide a ground floor on which later efforts can build.

Different members of an organization will likely have different goals for a proposed program evaluation. Employee assistance staff members, for example, might want to study details of program functioning while labor representatives seek to gauge the effect of program operations on the number of grievances filed. Compliance with drug-free workplace mandates and guidelines will probably interest security personnel, while benefits managers look for information about the EAP's impact on utilization and expenditure of benefits dollars (Blair, 1990). HR managers, of course, will likely be interested in all the areas described.

Whatever the goals may be, staffs from all parts of the organization must agree on them for the program evaluation to be successful. Participants must realize all of their needs will probably not get met, but agreement on certain mutually acceptable goals will lead to the addressing of subjects important to all.

The questions organizational representatives settle upon will determine the kind of evaluation required. Numerous evaluation methods exist, but two of the most common are process and outcome evaluations. Process evaluations focus on the EAP's everyday work, typically looking at:

• The number of clients seen;

• The promptness with which requests for service are accommodated;

• Courtesy;

• Utilization and penetrations rates across organizational groups and populations;

• Kinds of problems handled;

• Types of referrals;

• The number of cases involving mandatory reporting requirements;

• Distribution of employee assistance information (leaflets, brochures, newsletters, etc.);

• The level of and system for follow-up services;

• The demographic characteristics of clients; and

• Consultation with other organizational entities (labor relations, benefits, training, etc.)

Outcome evaluations, on the other hand, examine the impact and effects of EAPs on clients and the organization. These generally more sophisticated efforts require a higher level of skill, and probe areas in which quantification and measurement may prove difficult. Inquiries often focus on job performance. Some typical questions posed include:

• Do identified problems improve?

• Are treatment plans followed?

• Are gains sustained over time?

• What variables are correlated with satisfaction and dissatisfaction?

• What variables impact any or all of the above?

A second way of approaching program evaluations involves looking at the various stages of EAP functioning. EAP intervention, planning, program monitoring, impact assessment, and economic efficiency commonly provide the focus for this approach. Program intervention planning concentrates on the establishment and initial development of employee assistance efforts once organizational needs have been determined and goals set. Program monitoring addresses ongoing operations. Impact assessment deals with program effectiveness, examining the degree to which the program actually produces changes within the objectives defined. Benefit/cost analysis assesses economic efficiency in terms of value returned (benefit) for resources expended (cost) (Kim, 1988).

Cost-effectiveness analysis is another impact assessment technique that may prove easier to use than benefit/cost analysis. This technique is most commonly used to test efficiency where benefits do not translate easily into dollar figures. With cost-effectiveness analysis, program costs and benefits are quantified in terms of reaching set goals compared to the monetary value of resources invested in the program. The methods and principles are the same as those in benefit/cost analysis. Instead of using dollar amounts as a common denominator, though, this technique permits rank-ordering and comparison of programs in terms of costs for achieving set goals or different degrees of goals. EAP efficiency can be assessed by a relative comparison of cost for achieving the same or similar benefits or by examining the different levels of benefits achievable for the same or similar costs (Kim, 1988).

Tools for investigations usually remain the same regardless of structural approach. Process evaluations, for example, typically use program records and simple self-report surveys. Outcome evaluations use in-depth employee surveys, detailed client follow-up studies, treatment/cost analyses, sophisticated supervisor surveys, structured qualitative interviews, benefit/cost analysis methods, and overall qualitative and quantitative analyses to examine an EAP's functioning within its organizational context.

Deciding which method will best provide answers to the questions that need to be addressed should help simplify the selection of an evaluator. The employee assistance staff itself can often perform process evaluations provided they have basic research skills and the time to conduct such an evaluation. Using employee personnel for investigations of this kind has several advantages: it is less expensive than hiring an outside consultant; performing the operation gives staff members an opportunity to gain valuable insight into program operations; EAP staff members can share positive and negative information secured in the manner they deem most appropriate and useful; and inquiries can be targeted inexpensively to individual needs in a situationally specific manner.

However, self-evaluation has disadvantages too. Such appraisals present at least the appearance of a conflict of interest since results reflect on those performing the evaluation. A neutral external evaluator will find objectivity more easily attainable. As noted, employee assistance personnel also rarely have either time or the necessary research skills to ensure that the evaluation is rigorously conducted and its results accurately analyzed and interpreted. Limited research skills also narrow the range of topics an evaluation can address (Mani, 1992; Blair, 1990).

Certain circumstances, therefore, call for organizations to secure outside evaluators. Any procedure that is large in scope, technically complex, examines subtle questions with potentially threatening answers, and studies sensitive information and situations usually requires an impartial, external survey. EAP clients sometimes feel more comfortable with outside evaluators as well. Contractors may inspire more frankness in responses to surveys because clients feel confident that their privacy will be protected.

Outcome evaluations conducted by contract evaluators tend to yield richer information than the concrete results process evaluations supply. The outside consultants' skill level allows for the examination of more issues and relationships. The multiple validations and more sophisticated statistical analyses employed in such studies lead to more reliable and supportable conclusions. The information obtained can, therefore, be utilized across a broader spectrum of needs, applying not only to current operations, but also to future directions the organization might wish to consider.

D. Elements of a Successful Program Evaluation

While evaluation processes can take many different forms, a few elements usually form components of the most successful efforts. These include a needs assessment, the development of an implementation plan, the proper execution of the plan selected, feedback, and the generation of reports and strategies.

The needs assessment portion of program evaluation efforts determines what ideally should be done, what can be done given whatever resource limitations exist, and how to most efficiently accomplish stated goals. Evaluators should secure input and assurances of cooperation from all participants within the different organizational functions involved in the survey through a qualitative interviewing process. Based on the data interviewing reveals, an evaluation plan can be devised to address important issues, clarify the focus of the procedure, and describe possible uses for evaluations results. Evaluators should guarantee feedback to anyone asked to participate in the investigation and answer any questions those organization members paying for the survey might have (Blair, 1990).

Evaluation plans can include quantitative and qualitative techniques described earlier that the evaluators plan to employ. Once the methodology, which should flow from the types of issues requiring study, is determined, the plan must be executed. Obviously, a valid evaluation process relies on a sound record keeping system and data base. Many aspects of EAP functioning is easy to examine if the EAP's staff has collected and collated information pertaining to the program on a regular basis. Sometimes someone must start looking for an answer before discovering that the answer is not there. Incomplete treatment records, for example, make the conducting of clinical peer reviews, or commenting on the quality of care EAP clients receive, difficult. Appropriate evaluation and research generally produces many strategies for solving whatever problems the process has uncovered. Assessment and advice on corrective measures is an essential part of the evaluator's duties (Blair, 1990).

A mechanism for ensuring follow-through is also recommended once strategies for impacting EAP goals have been devised. Evaluation defines areas of effectiveness, areas that need improvement, and future directions that might prove profitable. Today's organizations, beset by constant change, place a high value on any concrete input into planning for the future. Program evaluation can provide such input, and therefore must be incorporated into standard EAP operations to ensure efficiency, effectiveness, and growth.

E. An Example of Effective Program Evaluation: A Benefit/Cost Structure Analysis

What follows is a summary of one effective yet easily understood approach to program evaluation. Several different practitioners have applied this process, or variations of it. In an article in The Employee Assistance Quarterly in 1988, Hide Yamatani presented one of the most detailed examples of a review of basic estimation methods geared for assessment of service benefits and cost structures of EAPS.

The benefit/cost analysis approach requires an assessment of demonstrable benefits an EAP provides to its organization, as well as the costs associated with operating the EAP. In simple terms, any dollar amount remaining after subtracting the cost of EAP operation from the estimated cost savings (benefits) provided by the EAP is considered a net financial gain. Dividing total cost savings by total operational costs yields another benefit/cost ratio. If the derived figure is greater than one, then the total savings produced by having an EAP are greater than the cost of operating one, while the reverse is true if the ratio drops below one. For example, a benefit/cost ratio of 3.0 means the organization received a return of $3 for every dollar spent by and on the EAP (Yamatani, 1998).

Benefits associated with EAP operation fall into two major categories: tangible and intangible gains. Tangible gains can most easily be measured and attributed directly to dollar savings. They include reductions in absenteeism, tardiness, grievance filing, and benefits utilization. Intangible benefits do no relate so obviously to the so-called "bottom-line," but affect it nevertheless by contributing to the productivity and overall well-being of the workforce. Some examples of intangible benefits include more cooperative workplace relationships, more positive attitudes toward work assigned, improvements in job performance as measured by qualitative supervisor assessment, greater commitments to expanding employee skills, and reported decreases in family problems. The benefit/cost model described here only estimates savings attributable to tangible benefits due to EAP intervention.

Operational costs for EAPs fall into two similar categories: fixed and variable. Fixed cost such as rent, equipment purchases, and administrative necessities remain stable over time. The EAP's operating budget generally identifies and defines these kinds of expenditures. Variable costs, on the other hand, differ from month to month or year to year, and are usually heavily influenced by the number of clients the EAP serves in a given period. The greater the number of clients seen, the higher the variable costs.

The benefit/cost analysis presented here can be undertaken at various levels of statistical sophistication, depending on the needs and available resources of the organization under evaluation. As this approach illustrates, though, even simple program evaluation, when conducted appropriately, can contribute to assessing return on investment.

F. Concluding Remarks

EAP evaluation should be an ongoing process, supplemented periodically by more specific investigations to answer timely questions about the program. Increasingly, the emphasis of evaluation has been on outcome indicators, examining whether the EAP delivers the results it should and answering the question, "What is a good EAP?" (Blair, 1990).

However, because EAPs exist within the context of organizations and the changing factors affecting these organizations, future evaluations may more likely try to figure out just what, exactly, an EAP is or should be given new and different and unique circumstances. Shifting demographics in the workforce, discussed in Chapter VI of this book, will also affect the form and function of future EAP services, possibly leading to wider variations between programs. This will ultimately complicate EAP evaluations and demand increased flexibility and cooperation from all those involved in the investigative process.

REFERENCES

Amaral, Thomas M., and Mana A. Kelly (1989), "EAP Cost-Benefit Analysis: Exploring Assumptions," Research Track, 18th EAPA Annual Conference, October 29-November 1, pp. 1-10.

Blair, Brenda R. (1990), "Before the Evaluation," EAPA Exchange, October, pp. 20-25.

Employee Assistance Professional Association, Inc. (EAPA, 1990), EA Program Standards, Arlington, Virginia.

Farkas, A.J. and Hirshberg, L. (1990). "Using Evaluation To Improve Services" EAP Digest, March/April, pp. 30-38.

Jennings, Kenneth, and Frederick Westfall (1992), "Benchmarking for Strategic Action," Journal of Business Strategies, May/June, pp. 22-25.

Kim, Dong Soo (1988), "Assessing Employment Assistance Programs: Evaluation Typology and Models," Evaluation of Employee Assistance Programs, M.J. Holosko and M.D. Feit, editors, The Haworth Press, New York, New York.

Knott, Tara D., and David H. Knott (1989) , "EAP Professionals Erect Barriers to Evaluation," Employee Assistance, December, p. 21.

Mani, Bonnie G. (1992), "Developing an Evaluation Design," Employee Assistance, January, pp. 40-50.

Tramm, Madeleine (1990), "Future Evaluations," EAPA Exchange, October, pp. 50-53.

Ulrich, Dave, Wayne Brockbank, and Arthur Yeung (1989), "Beyond Belief: A Benchmark for Human Resources," Human Resources Management, Vol. 28, No. 3, Fall, pp. 311-335.

Walker, Paul L. (1989), "Evaluating Your EAP," Employee Assistance, December, pp. 19-24.

Yamatani, Hide (1988), "EAP Benefit and Cost Structure Analysis: A Suggested Estimation Method," Employee Assistance Quarterly, Vol. 3, Nos. 3/4, pp. 129-149.

Continue to: EAP Handbook - Chapter 6 - THE ROLE OF EMPLOYEE ASSISTANCE IN A CHANGING WORLD (Reprise)

Return to: Table of Contents