We asked a well-known industry research guru about XM's "Custom Listening Study" from Arbitron, sent to agencies for a year or so now. A radio buyer had sent the report to the researcher, who looked it over for both of us. This was the latest report, Fall 2004 (mid Nov.-mid-Dec.). The researcher had a few questions and concerns on the methodologies.

"They're rating XM listening based on interviewing what they call the 'primary user.' They have some good methodologies approaches-they're using random selection of the sample. Someone has to be a subscriber for at least 60 days. It's telephone interview, which is good. But what they're doing is asking the household to define the primary user of XM. Which raises the interesting question of how can a household determine this? In other words do they ask everyone in the household to vote on the primary user, do they simply ask the person who answers the phone, or do they ask the person who's listed as the subscriber? The trouble with that is they're asking people for self-definitions and how you define primary user might be different that how I do. I have some concerns about that, as well as the fact they ask the survey participant to recall who else was present while listening. Will the industry accept that?

They also ask Over the last seven days what are the different channels that they listened to? And my concern there is, try to name all of the television channels you watched over the last seven days. It's recall over a seven day period. Was it this week, last week?

They also determine AQH listening by asking them about previous day listening. And the question I'd have there is what is the exact question Arbitron uses, because that can influence a respondent. I mean what probes, if any, did they give? Did they ask if the listener was riding in the car yesterday, were they listening to a News channel? Researchers do that, and it's not really disclosed in their methodology. So is this unaided or aided recall?"

How do they arrive at a currency?

"They have a sample of 2,000 subscribers as their sample base. And they project that 2,000 reflects the universe estimate, which is a very shaky proposition. You're assuming it is representative of the population. When you do that, you really can't do any statistical analysis, although though they do talk about it.

In the report itself, they only report ratings. They don't report based on the population that they're defining on. They do not report the actual AQH number itself because, if you look at the report, you see a 2.4 or a 1.7. In the buyers' mind, it could be confused with what's out in the real marketplace in terms of what Arbitron produces in its standard reports. So you have to figure out what the actual audience is yourself by doing it by hand.

The other thing is they sort of package these channels into groups. If I was a buyer, I would ask them how do you traffic my commercials? In reality, an advertiser or agency doesn't really care about how many people are subscribing to XM, nor how many primary user they have, but how many people are going to hear the commercial. So if they buy morning drive on the east coast, it's 4AM on the West Coast. Also, do they post? When do these commercials actually run and what affidavit do I have that shows me it did for a particular channel. Therefore, do they take that audience from that channel and add to it when it ran at another time on another channel and give me a full number? Or is it an AQH for the full week or a daypart? Those are the questions I would be asking if I were a buyer."

We tried putting in a call to Dr. Ed Cohen, Arbitron VP/Domestic Radio Research, but the offices were closed Friday. We will update on this during the week, if Dr. Cohen or XM can comment.