A recent story in the San Francisco Chronicle recounts the adventures of a PPM panelist as she tried to fulfill her responsibilities:
“I wore the meter all the time and followed the instructions. I didn’t find it that intrusive. But I wouldn’t take it to some occasions, like out to dinner, and they want you to wear it all day, from the time you wake up until you go to bed, and to wear it on your person. You can’t just leave it in your purse.”
After joining her current company, Sprauve felt less at ease wearing the meter. “Someone asked if I was wearing a pager,” she said. “Pagers are like ’80s things. And it was bulky.”
It sounds like Arbitron made an attempt to have the panelist perform her duties, and it sounds like the panelist made a conscientious effort to fulfill her duties. However, we suspect she was less than candid about how often she didn’t do as Arbitron asked. The meter probably spent a little more time in her purse than she admits.
While PPM is described as passive, the story shows that panelists are very conscious of the meter. In this respect, the story reinforces a point that we made last year that the meter is far from passive.
The goal of PPM is to capture all exposure to encoded radio, yet clearly Ms. Sprauve had more opportunities to hear radio than the meter had. Whether the meter was in her purse, left at home, or left on her desk at work, the meter didn’t capture all the radio she heard.
How much listening does PPM lose?
It’s a good question, because lost listening lowers TSL which hurts ratings. Arbitron understands this because they have elaborate editing procedures to recapture as much lost listening as possible.
They can’t do anything when a panelist leaves the meter at home, except bug her not to forget. They can and do credit radio stations under some circumstances when the meter can’t figure out which station was on, say, when it’s playing just at the threshold of audibility.
But despite their best efforts, there will be times when Arbitron can’t even guess what the meter was hearing.
She might be listening while driving down the road in her convertible. She might be listening on her Zune without the special PPM headphone adapter. She might be out with friends at a noisy bar.
In any of these situations, it is likely that the meter will fail to some degree to identify the radio stations she heard, and there is nothing Arbitron can do.
Despite PPM’s sophistication, the accuracy of PPM ratings ultimately rests on the meter’s ability to identify a radio station. The meter needs to hear a station clearly enough and long enough to decode the station’s unique identification.
So how often does the meter miss stations?
Arbitron is mum on the subject. We’ve asked, but it’s classified.
They’ve never published specifications, they’ve never released the results of any tests.
However, using the little information available, it is not an insignificant amount. We think perhaps three or four hours of a market’s TSL goes missing. Those might be your quarterhours.
The evidence
The first clue can be found in British tests of PPM. The radio ratings cooperative in the UK (RAJAR) put PPM through a series of tests, including its ability to identify the radio station that was playing.
As reported by the RAB PPM Task Force:
In November of 2004 the PPM participated in a test where 33 radio stations were played in various environments and levels of “noise.” The PPM correctly identified 59% of the total radio sessions, in an environment where the goal was 50-70%.
Dr. Joe Pilotta, a professor at Ohio State University had some unkind words to say about PPM and the apparent inability of PPM to identify radio stations 41% of the time.
Arbitron responded by declaring:
The RAJAR test showed that the Arbitron PPM does what is designed to do — reliably and objectively register a respondent’s exposure to encoded media — and shows that the PPM does its task very well.
So Arbitron does not deny the 41% error rate. PPM does what it is designed to do, which apparently is to identify at least 59% of radio stations it is exposed to.
Strictly speaking, this test didn’t measure TSL, but given that PPM estimates for reach are higher than diary-based estimates, the missed stations in this test apparently translate into lost TSL in the real word.
Were the 41% error rate applied to average TSL, it would work out to nearly seven hours a week.
It should be noted that after a thorough review of PPM, UK Broadcasters opted to continue using diaries.
More evidence comes from Arbitron.
We recently reported on Arbitron’s efforts to convince media buyers to pay higher PPM costs per point. In the Diary to PPM conversion chart (PDF) Arbitron publishes, they say media buyers should pay on average 16% more per point.
We can use their recommendation as a means to calculate how much listening PPM misses.
In an earlier post, we showed that TSL is just a product of AQH and cume. If AQH as measured by PPM is 16% lower than when measured by diary, we can work back and calculate how much listening disappeared.
Working through the numbers, it turns out that Arbitron is essentially saying that we should use a TSL conversion of 0.81. In other words, the average market measured by PPM has 81% of the TSL that it has when measured by diary.
Based on average TSL, this works out to be a loss of about three hours per week. Of course, this is an average. Based on Arbitron’s conversion chart, some markets could be losing as much as four and a half hours a week.
What’s the real number? Is it seven hours? Is it three hours? Someplace in between? Arbitron isn’t saying, so take your pick.
Should you feel good or bad that PPM misses only (say) 20% of the listening panelists are exposed to?
Anything more than zero and somebody is losing quarter-hours.
Whatever it is, just hope it’s your competition’s listening that gets lost, not your’s.
–Glenda Shrader Bos & Richard Harker of Harker Research
