Don`t Bother With Surveys That Aren't Credible

To paraphrase an old adage, statistics don’t lie, but liars can prove almost anything with statistics. Much the same could be said about many of the dozens of surveys that land on credit union CEOs’ desks throughout the year seeking their valuable input. Take credit union publication readership surveys as a prime example of the dozens of different surveys that pop up with great regularity. It seems that whatever publication is responsible for any given survey miraculously always seems to emerge as number one, Obviously, like stats, surveys can easily be manipulated. Over the years, I have seen dozens of credit union publication readership surveys (including of course those done by Credit Union Times) as well as many other types of surveys seeking CU opinions. This exposure causes me to ask some pointed questions and make some candid observations. Like these: What types of questions are asked? How many? In what order? How are they worded? These are all things that can greatly influence responses. The use of incentives can also have a bearing on how CEOs respond. Does a crisp new dollar bill encourage CEOs to stop what they are doing and immediately fill out a survey? Or does it anger them because of what some perceive as an implied bribe? Sample size is another factor that can influence readership survey results. Is it representative? Surveying a miniscule number of readers brings the legitimacy of results into serious question. So does keeping that number a secret. The length and complexity of surveys also play an important role in what response rate is generated. Most CU CEOs are simply too busy to bother with long and complicated survey instruments. How hard potential respondents have to think before making a choice often means a survey goes into the trashcan rather than a mailbox. Putting convoluted or essay questions in a survey is a sure-fire way to guarantee a smaller response rate. Multiple-choice questions that require little if any deep thought and no research always bring in bigger numbers. CU publication readership surveys that are handled entirely in house can almost always guarantee a favorable result for those behind them. If a publication writes the questions, sends it out, and tabulates the results internally it should come as no surprise that they will finish number one every time. If a readership survey fails to disclose its methodology, results should also be taken with a grain of salt. There are some pretty basic procedures to follow to assure a survey’s credibility. For example, one very familiar to credit unions is the annual American Banker newspaper consumer survey. It always discloses who is being surveyed, how many, and by what means. The publication uses a well-known and respected independent survey firm to handle this project. As a result, when it shows credit unions number one in consumer satisfaction ratings year after year, it is impressive because the survey has solid credibility. Of course what I am leading up to is the recent readership survey conducted for Credit Union Times. It, too, follows all the rules that separate a credible survey from those seeking pre-determined results. For example, the methodology was disclosed prominently up front. Its very name, “Credit Union CEO Readership and Editorial Preference Survey,” made it clear from the outset that it was a survey of 500-plus credit union CEO subscribers. It was designed, printed, mailed out, and tabulated by an independent North Carolina-based research firm. There were no incentives or reminders to meet a deadline because there was none. The survey was sent out from North Carolina and returned to North Carolina. Results were tabulated after four weeks. The response rate was an “amazing” (the survey firm’s term not ours) 37 percent. The Credit Union Times readership survey was brief and to the point. All questions were straightforward and all multiple choice. Because all credible readership survey procedures were followed, the results can also be considered credible according to the independent survey firm used to handle the entire readership survey process from A to Z. Frankly, every time one of these readership surveys goes out, Credit Union Times crosses its collective fingers. There is never a guarantee that the final results will be what is always hoped for. So how did Credit Union Times do? Very well. Although every question in the most recent Credit Union Times readership survey is considered important, the results of one question more than any others would seem to sum up the high regard in which readers hold this publication. That question is: “If you were limited to receiving only one credit union publication, please check which one of the following you would choose.” The five choices were Credit Union Magazine, Credit Union Management, Federal Credit Union, the Credit Union Journal, and of course Credit Union Times. Credit Union Times garnered the most votes earning an 82 percent rating. The next closest was Credit Union Management with seven percent followed by Credit Union Magazine and the Credit Union Journal each with a three percent total. Federal Credit Union scored a goose egg on this question. This publication also finished way out front in comparing usefulness in choosing vendors, pass along rate, and average time spent reading each issue. The point of mentioning the Credit Union Times readership survey in some detail is not to say our results were impressive, but rather to point out to all those handling surveys that results mean nothing if credible survey procedures are ignored. Going by the book and getting great results is the best of both worlds. Trying to fool those being surveyed only hurts those behind a survey. Next time a survey lands on your desk whether from a credit union group, a vendor, a regulator, a consumer organization, or a credit union publication, consider what makes a survey credible before spending time filling it out. Comments? Call 1-800-345-9936, Ext. 15, or Fax 561-683-8514, or E-mail [email protected]