Industry sponsored research? Leave it on the shelf!

Over at Marketing Pilgrim, an excellent resource from Andy Beal and friends, I’ve been giving Jordan a hard time about citing a radio industry study that (surprise!) shows that radio is awesomely effective. I see a ton of this in the travel sector and these bogus studies that “prove” economic or advertising effectiveness are really starting to piss me off, because this is an abuse of the correct notion that research is a great way to measure the effectiveness of things. Ironically you don’t even need any “cheating” on these industry sponsored studies to get bad results for the reasons I discuss with Jordan below:

Joe Duck Says:
Supported with fundingprovided from Radio industry companies
Studies by agencies like this generally *will not* publish anything but favorable things about radio. All such industry sponsored research is therefore suspect.

Jordan McCollum Says:
Well, yes and no. They might not publish their studies that don’t have favorable results, but (I hope) they’re not screwing with their methodologies to produce results skewed in their favor in the experiments that they do publish.

And while increasing unaided recall 450% is a pretty nice stat, it’s the only concrete, conclusive, across-the-board improvement found in the study. The other positives were significant for some brands studied and not others, with aggregate totals of almost no change. That’s why I said that it can influence them, but “well-established brands” might not be as effective.

Thank you for the comment, though. You’re right–you gotta follow the money.

Joe Duck Says:
Jordan it’s a very slippery slope to use industry studies due to the selectivity, though it is really common. Interestingly studies like this don’t have to screw with anything at all to create problems for people who want unvarnished truth. Assume for example that they did 3 excellent, methodologically sound studies on this topic and 2 of them indicated “zero increase in unaided recall”. The logical research conclusion is to be skeptical of the recall claim, but if we only see the positive study we’ll draw wrong conclusions. It’s rarely this cut and dried and you rarely see industry studies with good sets of assumptions, so all I’m also suggesting that studies like this are better left on the shelf if you are building a quality marketing strategy. One should stick to research done by people or groups who will still be around regardless of the outcome of the research.

4 thoughts on “Industry sponsored research? Leave it on the shelf!

  1. Its the same way with medical studies. If the results look disappointing, the study is cancelled or at best the data is consigned to the bottom drawer of some filing cabinet. Even if a researcher does want to write something up its hard to get anything published that is unimpressive. Journals want ‘pizzazz’ even in scholarly works and few publishers really want to offend advertisers or indirect funders. A lot of medical research is intentionally truncated rather early so as to avoid unpleasant discoveries.

    Most ‘research’ is advocacy anyway and not true ‘research’. Sure those radio gimmicks of ’12th caller’ gives them data, the phone company equipment they lease counts the busy signals and they can show advertisers some impressive charts. Same with TV and ratings, though often its overly “massaged” data.

  2. Most ‘research’ is advocacy

    Somewhat ironically, the closer you get to research with high practical value the more problems you have with bias since the outcome affects the chance the researcher will get funded again or directly suits the agenda of the research group as with the radio study above.

    Of course the responsibility for “fixing” this best rests with the users of the data, and until they become more discriminating about math we’ll continue to see worthless data held up as “interesting”.

  3. Often there is a dearth of truly neutral, scholarly research.

    Environmental Impact Statements are advocacy documents. There may be two sides but each side has advocates, not researchers.

    A stadium’s value to a city? Rent Control? Growth Managment? A Public Referendum? … There won’t be any neutral scholarly researchers instead there will be advocacy groups with misleading names and distinct views to advocate under the guise of research. There may be no middle ground at all. The quality of the data is so poor that anyone basing later decisions on it is on slippery ground.

  4. The quality of the data is so poor that anyone basing later decisions on it is on slippery ground

    Yes, but they do it nonetheless. If there was *one reform* I’d like to see with public spending it would be an unbiased and *BINDING* ROI analysis for every major spending decision that would force low ROI projects out of the loop. Savings from this? 50+% of all Govt spending.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s