Don’t Buy that Pitch on Survey Mode as Key to Higher Scores

The world of survey research is filled with “research professionals” who tout their methodology for collecting data as the elixir to all of your HCAHPS or other survey ailments. Like the snake-oil salesmen of the 19th century American West or Aunt Polly from The Adventures of Tom Sawyer, the remedies these folks are selling are grounded in nothing more than the salesperson’s own words. In many cases the effects of the snake oil can be worse than the ailment itself (which of course requires … more snake oil).

So when you hear the following pitch, remember the comments that follow.

“Step right up and gather around to hear about the wondrous health-giving effects of the super survey data collection method. It can instantly transform any set of data into super data to make you look years younger and better compared to your peers. It can magically move scores and change the fortunes of your organization for years to come.”

The mode you choose to survey your patients will not have any lasting effect on your performance. Anyone who says otherwise is selling you snake oil.

In fact, mail, phone and Internet are all equally appropriate modes of data collection. In some cases a combination of them is most appropriate. That is not to say that each one does not present a set of unique challenges or that one is without bias. What it does mean is that they all have certain characteristics that make them attractive to certain types of respondents. If care is taken to adjust for the nuances of each mode, they can provide strikingly similar results.

The truth is that over the past 20 years survey research has shifted focus away from single-mode survey design to more multi-mode methods of data collection. The implication is that no one mode holds the secret to a good sample. Recent research in survey design and implementation has made one thing clear: With proper adjustment, the use of phone, Internet or mail is perfectly valid separately or together.

The academic study of survey design has produced similar trends. The proceedings for this year’s American Association of Public Opinion Research conference were packed with researchers presenting on multi-mode data collection and the use of mail, phone and esurvey together. This is because organizations such as Press Ganey, the Centers for Medicare and Medicaid Services (CMS) and the Census Bureau understand that respondents react differently to different modes of survey administration. Differences in patient responses that are a function of the method of survey administration (“mode effects”) can be accounted for, by taking steps to adjust for differences in response. These adjustments make samples from different modes virtually identical.

The mode of data collection can have an impact on scores but only if there are no steps taken to adjust for these effects. CMS knows this and makes adjustments to the data before they are publicly available to ensure hospitals can be fairly compared with one another. In its HCAHPS Quality Assurance Guidelines, the agency notes that it provides phone mode adjustments for each of the six survey domains as well as the two global questions (“likelihood to recommend” and “rate this hospital from 0-10”). The amount of adjustment is different for each mode and for each domain or question being reported. These adjustments cancel out any mode effects.

Nevertheless, some “survey scientists” tout one methodology as being superior and that switching to a “better” mode is a quick route to increased scores. This is no truer than switching from a black Prius to a red Prius will get you better gas millage.

Every mode has strengths and weaknesses, and each mode presents unique challenges to capturing a good sample, but there is no evidence that one mode is the “best” for soliciting responses from patients in any medical setting. And no mode of survey collection will replace a commitment to improving the patient experience.

Facilities have every reason to look for an edge in the highly competitive world of value-based purchasing. When you hear someone say survey mode will provide that edge, that odor you smell is snake oil.