Survey Misstep #1 -Using Averages

The purpose of this “Survey Missteps” series is to discuss some of the more common issues we see with organizational surveys, whether employee or customer related. In many of our new engagements, our clients are already conducting surveys, so our primary service is to work with the existing design to improve validity, reliability, and analysis. Because of this, we see a lot of different situations and techniques and hope this series can add value to your survey design and data collection.

One of the most common “missteps” we see is the use of averages in data reporting. It is quite common, even in sophisticated organizations, to see reports providing the “average” or mean of the respondent answers. For example, a question might require the respondent to answer on a scale of 1 to 7, indicating their level of agreement or disagreement with the question. The results are then combined and reported as an average score. At first glance, this appears to be valuable information, as it allows the organization to compare averages across time, identifying seeming trends or problem areas. The challenge with this method is that it ignores the real value of the survey data…the distribution of the responses.

To illustrate, let us use the following simple example:

Company ABC conducts an employee survey with one question: “I love my job.” The employees are required to answer on a scale of 1 to 7, with 1 being “Completely Disagree,” 7 being “Completely Agree,” and 4 being “Neutral.” All 100 of ABC’s employees answer the survey, given annually over a period of three years, the randomly generated results of which (using averages) are as follows:

I love my job (n=100):

Year	2021	2022	2023
Average Response	4.13	3.90	3.85

Reviewing this data leads one to believe that there is a negative trend indicating fewer employees who love their jobs. The problem in this case; however, is that the lack of a distribution of the responses makes it impossible to see any meaningful trend outside of an arithmetic mean.

Using the same example (and the same randomly generated data), but reported using frequency distributions (answers of 6 or 7 on the scale as “high,” 3, 4, or 5 as “medium,” and 1 or 2 as “low”) provides the following:

I love my job (n=100):

Year	2021	2022	2023
Frequency:
High (6 or 7)	31 (31%)	31 (31%)	37 (37%)
Medium (3, 4, or 5)	38 (38%)	49 (49%)	40 (40%)
Low (1 or 2)	31 (31%)	20 (20%)	23 (23%)

A review of this data paints a much different picture than the previous example. Not only does it provide valuable insight into how groups of individuals answered (frequency distributions), but it also shows a much different trend. In this case, we can see that the number of employees who really love their job (“high”) increases over time, from 31 to 37. In addition, the number of employees who really do not love their job (“low”) decreases from 31 to 23. This indicates a positive trend in the data, even though the average scores decrease over time. This is the inherent problem with averages…they distort the potential interpretation of the data by ignoring the distribution. An average of 4 on a seven-point scale could consist of 100 4’s, or 50 1’s and 50 7’s, two very different scenarios.

Although this example comes from randomly generated data, we see the same issue in organizational survey results quite often, where companies use material resources trying to correct a perceived issue that does not exist. This is a common issue that can be solved quite easily through the simple reporting of responses as frequency distributions as opposed to averages.

The change from reporting survey results as averages to frequency distributions is a major step in effective analysis and one that can make a significant difference to interpretation accuracy. If you currently use averages for survey reporting, we encourage you to explore the difference frequency distributions can make in your overall survey analysis.

Please join us in our next discussion as we continue to examine some of the ways you can improve your employee and customer surveying techniques!