Study of the effectiveness of benchmarks on early terminations at ABC Grocery.
An Example of Performance Benchmarking

Submitted by:
Mike Williamson, Ph.D., VP and Director of Research and Development
Jason Clark, BA (Hons.), PhD (Candid.), Director of Business Development
Precision Human Development Ltd.

Executive Summary

In the fall of 2004, ABC Grocery commissioned Precision Human Development Ltd. (PHD) to conduct a research study designed to reduce unwanted turnover in the Cashier and Clerk positions. 

Approximately 63 percent of the newly-hired Cashiers and Clerks (hereafter referred to as Clerks) voluntarily terminated employment within the first six months of their start dates.   That amount of turnover was very expensive and imposed a constant inconvenience and burden on the already heavy workloads of managers and trainers.   

Five stores in the Vancouver, B.C. area participated in the study.  The sample consisted of 134 employed Clerks, which included approximately 50 percent of all Clerks that qualified for participation at the five stores.

Methods

The study was designed so that double-blind tests could be made of the effectiveness of customized benchmarks that were developed by the study for the purpose of selecting candidates that would be long-term Clerks.  Two control and two experimental groups were formed, separately, each of which was divided into long-term and short-term subgroups based on tenure in the position, where 6.5 months was the cut-point for the classifications. 

The study began on the first day of data collection, November 1, 2004, and ended on the last day of data collection, April 3, 2006.  Each member of the sample completed the SURE Behavioral Survey. Other data provided months of employment in the position.

Throughout the period of data collection, hiring managers were instructed to use long-standing hiring procedures, which included results from a selection tool previously used by the Company called Reid Assessments. 

Following data collection, a benchmark was created on each control group and applied blind to its corresponding experimental group to predict each member’s mathematical probability of being a long-term employee.  Those benchmarks were based on the strength of correlations between a set of behavioral competencies, measured by the Survey, and Months of Employment.  The procedure provided a double-blind validation of the benchmarks under conditions that simulated the hiring process.   

Subsequently, the two hiring procedures (Reid Assessments versus PHD Benchmarks) were evaluated for effectiveness in reducing unwanted turnover. 

Abridged Results

Results are tabulated and displayed graphically in Charts 1 through 4. 

Of the 33 Clerks in the first experimental group (Experimental Group 1) that were classified as long-term employees by the first benchmark (Benchmark 1), 32 Clerks actually were long-term employees for a predictive accuracy of 97.0 percent.  Of the 33 Clerks in Experimental Group 2 that were classified as long-term employees by Benchmark 2, thirty-one (31) Clerks actually were long-term employees for a predictive accuracy of 93.9 percent.  Neither level of accuracy would have occurred by chance 1 time in over 10,000 similarly conducted studies (p < 0.0000). 

During the data collection period, 40 Clerks were hired by the standard procedures, which included information provided by the Reid Assessments instrument.  Of those, 25 (62.5 percent) voluntarily quit within 6.5 months of employment.  Benchmark 1 predicted short-term employment for 23 of the 25 Clerks, indicating that they should not have been hired. 

Nine other Clerks met the qualifications for the study, but did so after April 3, 2005, too late to be included in the research sample.  Benchmark 1 predicted long-term employment for all of them, and 8 actually were long-term employees.  

Whereas, standard procedures resulted in 63 percent early terminations, Benchmarks 1 and 2 predicted rates of termination of 3 and 6 percent, respectively.  Realistic application of the Benchmarks over time should reduce rates of termination by at least 40 percent from their current levels and the return on investment would be highly cost-effective. 

Conclusions

The results of the study provided compelling evidence that the Benchmarks will do what they purport to do.  However, the investigators warn against the presumption that real application of the Benchmarks will result in 97 percent or even 94 percent retention over the long term, as suggested by the results.  As tenure increases so do reasons to quit, as does the economy and organizational changes in policy, all of which are beyond the control of the Benchmarks and could attenuate long-term employment.  Also, the conditions are right for the law of the regression toward the mean to make a modest reduction in the currently high percentages.            

Nevertheless, the results of the study justified the following conclusions:    

1. Benchmarks 1 and 2 are effective tools for reducing early terminations significantly below their current levels among Clerks at the five participating stores.   

2. Benchmarks 1 and 2 are significantly more effective for reducing early terminations among Clerks at the five participating stores than is the Reid Assessments instrument when conditions are relatively the same at the time of both applications.        

3. Benchmarks 1 and 2 are sufficiently effective that their use in the selection of Clerks at the five participating stores would yield a highly cost-effective return on investment. 

Return on Investment

For a Benchmark to be cost-effective, it would need only to reduce turnover by four percentage points, from 63 to 59 percent.  On the other hand, to illustrate potential benefits, 300 new hires at a 35 percent rate of turnover, which would be realistic, would yield a net savings of $345,446 and an ROI of $1,135 per employee. 

(Chart 5, attached, has been programmed so that it calculates net savings and returns on investment for various entries of “New Hires,”  “Rate of Turnover” (Rate of TO) and “With Benchmark” (With BM).  The reader simply needs to double click on the Chart, make the entries and observe the results.) 

Guarantee

Precision Human Development is so confident of the benefits of its services that it will guarantee the cost-effectiveness of its benchmarks, whereby, any benchmark that is less than cost effective will be adjusted at no cost to the Company. 

Recommendations

The highly significant results obtained in the study suggest that a broad-based application of the benchmarks within Marketplace IGA stores would be prudent as an effective strategy for reducing current rates of early terminations and providing ABC Grocery with a positive, and potentially significant, Return On Investment per employee.
Unabridged Results

Results in Chart 1 show the classifications for the two groups.  The overall accuracy in classifications (including predictions for both long and short tenure employees) for Control Group 1 was 84.4 percent.  This finding was
so strong that it would not occur as a chance outcome 1 time in more than 10,000 similarly conducted studies
(p < 0.0000).  The overall accuracy in blind classifications for the Experimental Group 1 was 88.6 percent.

The results that are most practically significant, however, are the accuracies of predictions for long-term employment (highlighted in yellow in Chart 1).  Thus, in the application of Benchmark 1 for selecting job candidates, only those predicted for long-term employment would be considered for selection. (On the Website, these individuals would appear beside green bars on the graph and would have Percentage Fit scores of 50 percent or higher.)  Conversely, individuals in Chart 1 predicted to be short-term would not be recommended for the position.  (These individuals would appear beside red bars on the PHD Website graph and would have Percentage Fit scores below 50 percent.) 

Chart 1

Classification Results (a,b)

a  84.4% of all cases in Control Group 1 correctly classified.
b  88.6% of all cases in Experimental Group 1 correctly classified.

Benchmark Accuracy for Identifying Long-Tenure Employees

Chart 2 graphically displays the percentages of Hits and Misses for Benchmark 1.  Benchmark 1 accurately classified 96.8 percent of the long-term members of Control Group 1 and 97.0 percent of the long-term members in Experimental Group 1.  The predicted long-term classification for both Groups was 96.8 percent.

A second benchmark (Benchmark 2) was developed on Control Group 2 and applied in the blind classification of cases in Experimental Group 2. 

Results in Chart 3 show the overall classifications produced by Benchmark 2. 

Chart 3

Classification Results (a,b)

                                a  93.3% of all cases in Control Group 2 correctly classified.
                                b  84.4% of all cases in Experimental Group 1 correctly classified.

Chart 4 graphically displays the percentages of Hits and Misses for Benchmark 2.  Benchmark 2 accurately classified 98.5 percent of the long-term members of Control Group 2 and 93.9 percent of the long-term members in Experimental Group 2.  The accuracy for the predicted long-term classification for both Groups was 97.0 percent.

Following the creation of the Benchmarks, an empirical test of the effectiveness of Benchmark 1 was conducted.  During the data collection period 40 Clerks were hired by the standard procedure.   Of those 40 Clerks, 25 (62.5 percent) voluntarily ended their employment prior to the 6.5 month threshold.  Benchmark 1 predicted short-term employment for 23 of the 25 Clerks, indicating that they should never have been hired.

The start dates for nine Clerks did not qualify them for the study until after April 3, 2006.  When Benchmark 1 was applied to their Survey data, it predicted long-term employment for all of them, and eight of the nine actually passed the 6.5 month threshold, as predicted.  Clearly, the number of cases in this analysis was far too small to draw conclusions, but the observation was quite interesting, nevertheless

Conclusions

The results of the study provided compelling evidence that the Benchmarks will do what they purport to do, identify job candidates whose tenure is significantly longer than those selected by the previously used procedure.  However, the investigators warn against the presumption that predictive accuracy in the selection of long-term Clerks could be maintained at 97 percent or even 94 percent, as the results suggest.  Thus, as tenure increases so do reasons to quit that are beyond the control of the Benchmarks—employees move to new locations, they have babies, they go to school, they have to spend too much time getting to and from work, they don’t like their new supervisor, they have interpersonal conflict with another employee, changes occur in organizational policies, the economy changes, and the list goes on.  Also, conditions are right for the law of regression toward the mean to make a modest reduction in the currently high percentages.           

Nevertheless, the results of the study justified the following conclusions:   

1. Benchmarks 1 and 2 are effective tools for reducing early terminations significantly below their current levels among Clerks at the five participating stores.  

2. Benchmarks 1 and 2 are significantly more effective for reducing early terminations among Clerks at the five participating stores than is the Reid Assessments instrument when conditions are relatively the same at the time of both applications.       

3. Benchmarks 1 and 2 are sufficiently effective that their use in the selection of Clerks at the five participating stores would yield a highly cost-effective return on investment. 

Return on Investment

Precision Human Development is not privy to accurate replacement costs associated with short-term Clerks at ABC Grocery.  However, estimates by industry standards suggest direct and indirect costs could amount to CDN $4,600 per employee (source: Coca Cola Retailing Research Council Grocery Turnover Study; IGA Coca Cola Institute for Learning).

For a Benchmark to be cost-effective, it would need only to reduce turnover by four percentage points, from 63 to 59 percent.  On the other hand, to illustrate potential benefits, 300 new hires at a 35 percent rate of turnover, which would be realistic, would yield a net savings of $340,446 and an ROI of $1,135 per employee.2 

Guarantee

Precision Human Development is so confident of the benefits of its services that it will guarantee the cost-effectiveness of its benchmarks, whereby, any benchmark that is less than cost-effective will be adjusted at no cost to ABC Grocery. 

Recommendations

The study provided evidence that supports recommendations whereby ABC Grocery Co. would maximize benefits and increase savings further by expanding the scope and application of PHD Benchmarks, as follows: 

1. Utilize the current Tenure Benchmarks, according to instructions, for selecting Store Clerks throughout the network of stores.

2. Obtain additional ROI benefits by benchmarking key department or store-level management positions for improved performance and to establish efficient Succession Planning.

3. Consider creating other types of benchmarks to improve sales performance or other risk factors such as, workplace accidents or abuses of Workman’s Compensation.

4. By expanding utilization of PHD services, ABC Grocery would qualify for additional discounts in services.

Chart 5

Note:  “TO” refers to Turnover; “No. Terms” refers to Number of Terminations; “BM” stands for Benchmark; “SC” means Service Counts, where one SC is used for each Survey administered.