(Editor’s Note: Today’s post is written and sponsored by the National Business Research Institute (NBRI). They have over 30 years of experience in conducting scientific, psychological research for businesses. NBRI clients include Walt Disney World, Marriott Hotels, Oracle and Waste Management.)
Surveys are often touted as a perfect remedy for any of your business’ ailments. Whether you survey employees or customers, a survey company might claim to be able to identify your most glaring business problems. But, as an established businessperson, you might easily identify most of these problems yourself. The important differentiator is to identify not only the problems, but also the root causes of the problems and their solutions. To achieve this accurate look inside your business, it’s vital to employ a company that collects valid data, employs powerful analytics, and turns the combination into a powerful plan for improvement for your company.
Do-it-yourself ‘monkey’ type surveys provide about as much accurate and useful insight as, well, a monkey could. If you ask your employees how they feel about their pay, they will almost always tell you they want more money. The goal is to collect broad and deep information. Composing questions that achieve that goal isn’t easy. In fact, organizational psychologists are the best at doing it – and they’ve earned doctoral degrees to equip themselves.
While you may lament that this gives you more data to slog through, it is absolutely essential to determining the root cause of any issue. If you only have top-layer data, you will only find top-layer problems that anyone could have identified – for free. If you collect that hard (and sometimes tough-to-swallow) data, you will be able to fix the problems at their core, instead of simply placing Band-Aids. It’s most effective and efficient to make decisions that will reap long-term rewards, and that’s no different with surveys.
Collecting accurate data about your business is great, but it has limited value all by itself. The real power of this data is realized when it’s compared against similar businesses within your industry. This process is called benchmarking. Part of the benchmarking process involves normative data, which is the average score of a given survey question. As a decision maker within your company, it’s important to know how your scores stack up against your competition. Without this key data, you are left in the dark as to how you are actually doing within your industry. While do-it-yourself surveys or cheap survey companies might accidentally gather good data, you will only get quality benchmarking data through top of the line research companies.
Longitudinal Research
Collecting data, conducting the research, and assessing sentiment gives you powerful information. However, posting a sign on your wall that says, “We conducted a survey!” isn’t going to convert more customers or improve your employee engagement. The only useful outcome of all this work is creating new procedures and making changes based on the information. And, that’s not even the end of the survey line! You need to know if your changes produced any noticeable improvements. Longitudinal research assesses the impact that your changes produce by re-evaluating your subjects after a pre-determined time.
A quick Internet search will reveal how many survey firms exist today. Some are capable of collecting data, benchmarking, and conducting analyses. Some are simply survey companies while others are research firms. Ultimately, surveys are a vehicle to collect data. You need to combine that data with further research to draw accurate conclusions and implement action plans to improve your company. Don’t fall prey to the quick and easy survey options – you need research that will drive your business past the competition.
Many thanks to NBRI for sharing their expertise. NBRI offers comprehensive full-service research solutions when it’s time to deploy employee surveys or customer surveys. If you’d like to learn more about their services, please visit their website, watch an online survey demo or download their free ebook on “How to Conduct a Survey”.
0
John@PGISelfDirected says
This is a very insightful post. The inputs on root cause analysis deserves 5 stars!
David Bowles, Ph.D. says
Sharlyn two articles of yours which caught my eye in one day…you are hitting all the hot buttons! OK as someone who has done hundreds of surveys over 25 years you make very good points. Like all data they can be: badly collected and/or badly analyzed and/or badly used! But a good survey is a powerful tool indeed and our clients just could not wait to get their hands on the results…
One thing I want to counter here though, and its very important: I would urge anyone considering it to avoid benchmarking their data against other companies. There are many reasons, which I covered here:
http://wp.me/pEDK3-4n
Mainly, there is evidence that these benchmarks are wildly off the mark from each other, so who can we trust? If you look at the article, I gathered benchmarks for ONE industry, during ONE period and on the SAME questions; there were up to 20 percentage point differences on these benchmarks for individual questions, even under these conditions. I don’t know anyone else who has done the research on this that I have, and I used to be a purveyor and fan of benchmark-backed employee opinion data! I so lost faith in this I stopped using external benchmarks and went to purely internal, which is a very very powerful tool, again done right. Your own organization has its unique culture, and is therefore its best benchmark of all. I have helped clients find their own internal pools of excellence, they didn’t need to see what Google and Apple, or whoever, were doing. This is not what the big research houses want to hear: they make money from their benchmarks. But its what increasing numbers of people are realizing: keep the benchmarking in-house, you will have far far better data to work with.
best to you,
DB
Co-author, “The High Engagement Work Culture: Balancing ME and WE” (July 3rd 2012, Macmillan)
Ken West says
Hello all.
The Benchmarking spoken about in this post is at the survey question level and is based upon the NAICS Code(s) of the specific business under examination. NAICS stands for the North American Industry Classification System.
Agreed that the external benchmarking referred to in the above comment can be inaccurate, but that is different and is based upon a collection of questions that are meant to indicate employee engagement or customer satisfaction or some other metric. Unfortunately, too many (almost all) of these types of measurements use different question sets to measure the same metric(s) and then are not based upon the NAICS Code(s) of a specific business under examination, but rather a large overall population.
NAICS Code based benchmarking is very accurate in comparing the performance of a specific business in a specific industry to other businesses in that same industry.
I hope this helps clarify the Benchmarking referred to in this post..
Best,
Ken West
Chief Operating Officer
National Business Research Institute
David Bowles, Ph.D. says
Ken, perhaps you didn’t read the link I put in my comment, or the details of my comment, which cover what you are talking about. If I may, I’d like to reiterate that:
–I examined questions from different benchmarks, all for the same industry, the US electric utilities (probably all one NAICS Code)
–I laid out side by side the exact same questions from multiple benchmarks belonging to different consulting firms
–the differences were huge, on the same questions, in the same time frame and the same narrow industry.
The problem is that each company has a different culture; it then has different levels of morale, engagement etc. No one consultant has a lock on all company opinion data in an industry, hence data sets from one group of companies are quite different from those of another group. Nothing you are saying can account for, or mitigate, this. My data are from a very specific, narrow industry. Both clients and consultants fiercely guard such data and it was only by chance that I was able to gather all these benchmarks together. I have never seen it before or since.
For this reason, I’m willing to bet that if we got together all the employee opinion benchmarks for one specific NAICS Code group of companies, including yours, they would be quite different, for the reasons I have stated. As I said in my first comment, I sold such data, but gave up when I saw these results.
DB
Co-author, “Employee Morale: Driving Performance in Challenging Times” (Macmillan, 2009), which details many of these problems and the alternative solutions (internal benchmarking).
Ken West says
Hello David.
Yes, I read the article that you authored that you referred to in your previous comment. It referred to Trend Data and percentages of engagement, etc.
What I refer to are mean scores of particular survey questions that map to normative percentiles for particular industry segments (NAICS Codes).
There are actually 21 different NAICS Codes related to Power Generation and Transmission, which was my point. Most data of the type you speak of is of a large overall population, rather than pinpointed at a particular NAICS Code. We are unaware of any other research firm who had the foresight to segment their benchmarking database by NAICS Code, certainly not for the last 30 years as we have done. This makes us unique and uniquely qualified to provide meaningful benchmarking comparisons.
Example: assume that we analyze the mean scores of two different survey questions and BOTH score 3.99. One survey question is: overall, I like my job. The other survey question: I am fairly compensated. We might find that a mean score in a particular NAICS Code of 3.99 for the first question represents an “average” score at the 50th percentile, while a mean score of 3.99 for the second question represents a Best in Class score. Without external benchmarking of the type I am referring to, this particular fictitious company is left to wonder about the goodness or badness of their scores as the only thing that they can be compared to is the rating scale that was used on the survey.
David Bowles, Ph.D. says
Ken perhaps we don’t want to go on and on here and bore visitors to this interesting website with the esoterics of this issue; but sorry I remain deeply skeptical and unconvinced. My link was not just about trends, it was the fact that I found the benchmarks of several major consulting firms in this field for one industry to be extremely different from each other, i.e. basically useless. Your argument is that you break it down to very small units, but unless you have a monopoly of survey data for each NAICS code in the US, and unless all in each code have been polled, you will have only a small sample of responses out of the whole population of all people working in that code. How could you convince me, as a client, that your small sample for any given code represents the whole? If you have solved this problem I would be very interested, and very, very surprised.
For example, I surveyed a lot of (electric) linemen; I had tons of utility clients down the west coast and into the mountain west….but even then I had just a tiny fraction of all the linemen in the US. When I saw the data from other consultants for the same types of companies I realized the differences…and stopped doing this.
So here is the question for you: for which NAICS code do you have the most data? Secondly, as a percentage of all those working in the US in that code, what percentage did you poll, for what percent do you have current data… 5%? 10%? (those would be incredibly high numbers for one consultant to control). Third, how do I know that the other 90-95% of those you have not polled in that code would be represented by your data?
Honestly once my clients gave up on benchmarking they never looked back; our powerful internal benchmarking software gave them so much information on internal champions etc., not under one external business code but in the same exact business and culture. Do you think Apple wants to compare to Dell? I don’t think so….they want to be “insanely great”. Google, which does tons of worker surveys, I bet they don’t look at Yahoo….they have their own values and their own goals, which is why they beat the c**p out of Yahoo! The issue which you raise was never a concern with my clients, they just set their own standards based on what the best in their own company could achieve, and what their vision said they wanted to achieve, and went from there. They became their own benchmark.
best to you
DB
Co-author, “The High Engagement Work Culture: Balancing ME and WE” (July 3rd 2012, Macmillan)
Ken West says
Hi again.
Prepare to be surprised! There are 1175 6 digit NAICS Codes in existence and NBRI has a minimum of 250,000 survey responses on all but 8 of those. Remember, we’ve been at this for over 30 years and have collected more than 3.5 Billion survey responses.
Obtaining a representative sample of a business population is a mathematical formula dependent upon the population size and several other factors. 250,000 responses is way within the limit of valid data. In fact, it exceeds a 99.99999% Confidence Interval with less than a 1% Sampling Error. As you may be aware, a 95% Confidence Interval with a 5% Sampling Error is perfectly acceptable in business research all day long.
Your question: “Do you think Apple wants to compare to Dell?” YES, absolutely they do! They would love to know exactly how they stack up against their competition with an objective measurement tool. That’s External Benchmarking!
I do not disagree with Internal Benchmarking. It is one of many tools used to understand the performance of an organization. External Benchmarking is the same, a tool to be used to assist the Client organization. Internal Benchmarking tells you how you did versus the last measurement. Did you improve or not? External Benchmarking is the same, but with a view towards one’s competition. How do we stack up against our competition? That is a very important question and one that many organizations want answered.
But, yes, lets not take up a lot more time. Also, Benchmarking is only one of the items mentioned in the original post. There is Valid Data Collection, Deep Analytics, Benchmarking, and Longitudal Research to name a few of the tools that are critical to being a data-driven organization and reaping the benefits available with today’s technology.
David, thank you for the very interesting discussion!
Thanks all!!!
David Bowles, Ph.D. says
Ken I appreciate your tone and all the work you have done. I know what its like, I maintained databases for a long time. I think discussions like this are useful, especially when friendly! This is how things move forward…..
I understand your statistics; the only question I would have, if I were your client (or the client of anyone with a benchmark database) is this: since the Crash affected everything, and we don’t want data from pre- or mid- Crash, we need something from at least 2010-2012. We also want data from our main competitors otherwise we are comparing ourselves to insurance companies, software companies, etc….even if they have some of the same jobs we do…..and we are a construction company with a unique culture! Apples to oranges. So can you give us very recent data from our competitors? Data which is statistically representative and valid across each of your job codes?
Most, if not all, consultants would have to answer that with a “no”…..with or without the job codes. I certainly had to. The industry is so incredibly fragmented, most competitors’ data are divided up one here, one there, etc….between consulting firms.
That’s all I am saying, and thanks for the great discussion!
David
Ken West says
David,
Yes, if a client wishes to benchmark themselves across a specific timeframe that is available. Also, yes, NBRI offers valid benchmarking data (at least 95%CI & 5%SE) segmented by almost all NAICS Codes from 2010 to present.
It’s obvious that you are very familiar with benchmarking from the questions you have asked. And I hear you about some of the potential pitfalls of benchmarking. It’s a serious topic and a very serious undertaking. Our clients make multi-million dollar decisions based upon the information and analysis we provide and we are extremely careful and very diligent in making sure that we only provide valid benchmarking data.
Thanks again! Good luck to you and your business!
David Bowles, Ph.D. says
Ken, its obvious you have done your homework, as you say a lot depends on it. Thanks for the discourse and the agreeable way we can disagree on some things! As you say there are many tools to use; mainly you and I support the use of surveys as a powerful diagnostic tool, when done right. There is simply no other way to find out where an organization stands, something I have written a lot about. When I see some blogs slam surveys I wonder how on earth a large company, perhaps with people around the world, can find out confidentially and honestly, what is going on on every nook and cranny….
Good luck to you and your business; I will look into following you on Twitter.
David
Co-author, “The High Engagement Work Culture: Balancing ME and WE” (July 3rd 2012, worldwide release, Macmillan)
Sharlyn Lauby says
Wow! What a terrific discussion. Thank you for taking the time to share your expertise on the subject. By doing so, you’ve given us all a lot to contemplate when considering the use of benchmarks.
As Ken indicated (and we can see from the discussion), it’s a very serious undertaking given everything at stake. The survey approach is critical – unless we’re just trying to determine the right color to paint the employee break room. Ultimately, businesses may need professional guidance to draw accurate conclusions and implement the best action plans.