As promised, a point-by-point response from Arbitron on concerns we aired from a buyer and industry researcher on XM's "Custom Listening Study." Actually, we got that, and a complete education on satellite radio ratings - - compliments of Dr. Ed Cohen, Arbitron VP/Domestic Radio Research, who was happy to respond to our Tuesday story (5/31 RBR #106):

"Arbitron has conducted custom ratings studies for XM Satellite Radio for over three years. To date, five studies have been released and the Spring 2005 study is in the field now. These studies are proprietary to XM, although Arbitron has publicly discussed the methodology involved, and answered questions at presentations a number of times over this period.

Over time, the methodology has evolved somewhat, just as our syndicated diary service methodology has been tweaked over time. These changes are designed to improve the quality of the data and we hold this service to the same standards as the diary service. It is extremely important to Arbitron that we not compromise the quality of our services, whether it is our syndicated radio audience measurement service or our custom studies.

---

The Primary Listener

To answer this issue requires an understanding of what we are trying to accomplish with this service. Unlike our syndicated services, we have only one universe number, which is the number of XM subscribers at the end of any quarter. This number is the subscriber base, which is reported by XM as part of the quarterly earnings announcement. It's comparable to a 12+ population estimate for a market, except that in this case, it is much closer to truth as XM knows how many subscribers it has at that time. The report is subject to regulatory scrutiny (XM is a public company) and so, we accept that this number is accurate.

At that point, we make an assumption of one subscription per household. However, the name on the subscription is not necessarily the person who uses the service. And while we have seen an increase in usage of XM outside of the vehicle, car and truck listening is still the predominant location for listening. These facts drive the need to determine a primary listener as against the billing name.

Self-Determination of the Primary User

This has not been an issue. The primary user is defined in the questionnaire and is not open-ended. There are two criteria. If the XM radio is installed in the car or truck, the primary user is the person that drives that vehicle most often. If the XM radio is in the home, the primary user is the person who uses the radio most often.

We started asking about additional listeners beginning with our second study for XM in 2003. Using only one person in the household, the primary user, understates XM's audience as this assumes no one else (additional users) listened to that radio. XM charged us with developing a methodology to capture this listening. At the same time, whatever method we used would have to meet Arbitron's standards. The result was the additional listener system.

The enumeration of additional listeners is done to set a base for projection. We are not certain when all of these people may have listened during the seven days, so we do not project additional listener cumes, only AQH. To answer the question about recall, based on the Fall 2004 study, over two thirds of the additional listeners were household members. Less than one third of the listeners were non-household members. We're very comfortable with a respondent's ability to remember who else may have heard the XM radio (usually family members who rode in the vehicle). This information is clearly disclosed in the printed report. This system is somewhat akin to print measurement with respect to magazine and newspaper measurement with respect to circulation figures compared to readership estimates.

This survey does derive cume from a seven-day recall and Arbitron has always made this very clear in our presentations. We ask for all XM channels that were heard for at least five minutes or more in the past seven days and the interviewer probes until the respondent cannot remember any more channels that were heard. I agree with the comment that a respondent would be unable to remember all the television channels they viewed in the past seven days, but we're talking about radio in this case. Television is program-based while XM channels are generally genre-specific and in many cases, highly targeted, far beyond what terrestrial radio offers. Someone who cumes Hank's Place, Boneyard, or Open Road knows what that channel offers and can readily tell you that they listened.

Collection of Previous Day Listening

This points to the problem with much of the research we see today. So much is accepted point blank when questions are often twisted a bit to elicit responses that favor the client. From the beginning, this product was designed to be used as currency and under no circumstances would we slant our questions. In every presentation we have made about how we measure XM, we clearly state that the AQH questions are unaided, in other words, the only way we could put our name on the report. Our commitment to using unaided recall (just as in the diary) continues as we move into sports play by play measurement. Not every potential client is excited about using unaided recall for play by play sports, but we see no other way to measure this audience while still putting the Arbitron name on the cover.

AQH and Sample Size

The AQH questions cover three dayparts of ten hours each. We ask about any listening to XM, collect start time, stop time, and the channels heard. We probe for more listening, but use only "lifestyle probes" keyed to the time of day. Under no circumstances do we ask a question around programming, nor do we try to "help out" the respondent if they cannot remember the channel or channels they heard. We also ask whether any of the additional listeners that were enumerated earlier were listening with the primary user at that time and if so, which ones and for how long. The additional listener AQH is derived from this set of questions.

The sample size of 2,000 is acceptable although XM had used a sample target of 3,500 in the previous (Spring 2004) report. You'll be pleased to know that the Spring 2005 report will be based on 3,500 minimum intab.

A random sample of 2,000 persons (and the sample is randomly selected from the XM subscriber base) can be projected to the universe complete with a calculation of sampling error. As I noted earlier, we have only one good universe estimate, which is total subscribers. XM does not know the demographic makeup of subscribers, let alone primary users. We do make the assumption that the intab sample is representative of this group and do not weight the sample. Based on a "sniff test", the face validity appears good. Each study has shown the sample to be heavily male (nearly 70%) with the additional listener group looking very similar to the population as a whole by demo. These percentages have moved across time as XM's subscriber base has increased, which should not be a surprise.

We do report ratings based on the populations that we have defined which are the XM primary and additional listener universes. The reason for using only ratings, at XM's request, is that their subscriber base is growing at a fast pace and the studies need to be used for multiple quarters. Think about it this way: the fastest growing market in the country over the past two decades or so has been Las Vegas. Now, imagine Las Vegas on steroids. Our first XM study in 2002 was based on 100,000 subscribers. The last study has a base of over 3.7 million. And then consider if Claritas (Arbitron's provider of population estimates in the syndicated service) were to change the population estimates every quarter rather than once a year. Under these circumstances, the use of ratings rather than persons estimates makes good sense.

We do not project to the entire US population, but if buyers wish to do so, they can make this comparison. This would assume that no others in the population listen to XM radio at any time, which is probably unlikely. However, Arbitron has not measured that listening and we do not report on that basis.

The printed report is based on the channel groupings requested by XM and in a custom report, they have this right. The channel group estimates are based on adding the AQH for all of these channels together, with the primary listener cumes de-duped. In addition, each page is footnoted noting that all estimates have been normalized to Eastern time.

Throughout our relationship with XM, we have been very open with the buying community about our methodology and have answered any and all questions, just as we do with our syndicated service. We take very seriously placing the Arbitron name on any piece of research, just as we have for over a half century. That policy did not change in any way for XM."

Arbitron also offered this PDF explaining more of the XM ratings methodology. See it here.