The Ultimate Survey: A "How-To" Approach and Exploration of Stations'
Surveys
By B. R. Forbes
Originally published in the Development Exchange journal i.e. development, October 1987.
"To manage a business is to manage its future; and to manage the future is
to manage information."Marion Harder, Jr., "A New Profession to Aid Management," Journal
of Marketing, January 1961, page 1
I. Introduction: Purpose of Report
Market
Information: helping to make informed decisions
We make fund-raising decisions based on experience,
intuition and information. Experience takes years to
accumulate, usually through trial and error. Intuition
is fickle -- and not easily defensible before a board
of directors. Information can be actively acquired,
analyzed and applied to the fund-raising decisions we
must make every day. The kind of information we need
is knowledge concerning our primary market, our listeners:
their program preferences, listening habits, demographics,
consumer habits, perceptions) media habits and how they
think. This kind of market information helps us make
more informed decisions -- and (sometimes more importantly)
help justify our decisions to general managers or boards
of directors.
Sources
of market information
We already have access to some good market information
through Arbitron ratings, the National Public Radio
Audience report and (for classical music) WFMT's "The
Classical Advantage." However, the most useful and accurate
information is from our own local audience.
Conducting
our own market research
The purpose of this report is to give a basic understanding
of research approaches, to provide an overview of how
other public radio stations are gathering market information
and to provide a practical guide to creating a survey.
The most useful analysis includes market information
from three different groups: members, listeners and
the general population. However, for the purposes of
this report, we’ll concentrate on the most important
markets: listeners and current members/subscribers.
II.
Public Radio Overview
Local
market research: What's everyone else doing?
Generally, development managers agree on the need for
local market research. However, based on observations
and brief telephone surveys, public radio stations can
be divided Into three categories:
- Those
who have not conducted research either because of
lack of time or lack of resources.
- Those
who have conducted some research but, through poor
sampling or misinterpretation, have misread the
market.
- Those
who conducted research competently and who have
improved their audience numbers and their financial
support through a better understanding of their
market.
Category
#1: "I just don't have the time! I can't get the money!"
The time it takes to read this report is the time needed
to make a few telephone calls to line up a professional
researcher (whose services may either be paid or donated)
and volunteers to conduct the research. Stations have
used paid professional market research companies, the
donated services of marketing companies, professional
marketing volunteers, university marketing departments,
or have just done it themselves. If money is the problem,
write a grant proposal or offer to trade underwriting
credits for market research services. If the problem
is getting the general manager to support local market
research, show him/her how KHPR, Honolulu, has increased
their underwriting support or how WGUC, Cincinnati,
has increased their audience through more targeted advertising.
Development Exchange research shows that some of the
smallest stations in the system have successfully utilized
market research for audience-building and fund-raising.
This should be an incentive for all stations!
Category
#2: "It just didn't work..."
Here are some of the most common problems or pitfalls
in conducting local market research;
- Member
surveys are treated as listener surveys. Wrong!
Although members are listeners, research has indicated
that members are quite different than general listeners
in many ways.
- Poor
sampling is conducted. Whether
through lack of time, or money
or just ignorance of sampling
techniques, many stations have
used convenience samples rather
than high probability samples.
(These terms will he explained
later.) Poor samples invalidated
those stations' findings and
led them to wrong conclusions.
- The
job was not completed. Some stations have conducted
fine surveys -- and still have the survey forms
sitting in their office in a cardboard box! They
spent more time working out the questionnaires than
they did on planning the collation and analysis
of the data.
- Findings
of the research are not acted upon. The major reasons
were: the information may have been irrelevant to
the decisions actually being made; or staff members
may have felt the information was inaccurate since
it did not reflect their own perceptions.
Category
#3: "Local market research has made a big difference!"
Some stations have conducted excellent local research
-- and are reaping tremendous rewards.
- For
example, KHPR, Honolulu, has used the NPR national
listener profile and WFMT's "The Classical Advantage"
to encourage underwriting support. But the most
effective information is its detailed local listener
profile -- which no other Hawaiian station has been
able to offer to prospective underwriters.
- A
KUSC, Los Angeles, survey indicated that listeners
thought of the station as a "classical music service,"
rather than an "arts and information service" which
the station used as its ID. KUSC changed its positioning
statement -- and increased the cumulative audience
reported by Arbitron (which uses the recall of stations'
slogans as a measurement tool).
- WGUC,
Cincinnati, was getting bogged down with the same
old premiums. Then a survey indicated that many
of their listeners were gardening enthusiasts. So
they began offering WGUC gardening gloves, seeds
and tools -- and reaped a substantial harvest of
their own.
Every
station could become a "success story" -- with the proper
understanding of research approaches and a practical
approach to conducting its own research.
III. Establishing Objectives And Strategies
Develop the plan for conducting research by taking these
steps:
1. Determine objectives
2. Apply objectives to actual decision-making
3. Determine the information needed to make the decision(s)
4. Select the source of information
Step
#1: Determine your objectives
Before beginning market research, determine the objectives
and how the information will be used. The sampling,
methodology and questions will be determined by the
objectives. Some of the most frequently-mentioned objectives
in conducting local market research are:
- To
increase local underwriting by gathering demographic,
consumer, media and psychographic information concerning
the audiences of various types of programming.
- To
assist programming staff in making programming decisions
and to predict membership and audience reactions
to programming decisions.
- To
develop new fund-raising strategies and activities
which would encourage new members and upgrade the
support of current members.
- To
understand the difference in perception of the station
between members and non-members and to facilitate
perceptions among non-members which would encourage
their support.
Step
#2: Apply objectives to decision-making
Once objectives have been determined, be more specific
by listing the actual decisions that will be made. For
example, if the objective is to increase underwriting
through a detailed listener profile, the decisions may
be:
- To
which businesses should I devote most of my attention:
banking, insurance, retail, manufacturing, leisure,
etc.?
- Which
aspects of my audience profile are competitive with
the profiles of other radio stations?
If
your objective is to develop new fund-raising strategies
and activities which would increase the support of current
members, the decisions may be:
- What
giving levels should we suggest?
- What
member benefits should we offer?
- In
what neighborhoods should I concentrate my fund-raising?
- What
kind of on-air fundraising copy should I write which
would encourage increased support?
Step
#3: Determine the information needed to make decisions
Once the actual decisions to be made are listed, determine
the information which will make decision-making easier.
For example, if the objective is to increase the support
of current members, the information needed may be:
- Giving
levels: household income, net worth, support of
other non-profits, investments, level of current
support, number of children, and marital status.
- Benefits:
home entertainment equipment,
credit card usage, travel, sources
of entertainment (movies, plays,
etc.), restaurant patronage,
general consumer information.
- Neighborhoods:
value of home, zip code, level of current support,
household income, place of work.
- On-air
copy: level of education, programming interests,
perception of station, reason for giving, media
habits.
Step
#4: Select the source of information
The objectives, decisions and specific information will
determine the source of your information. For example,
to increase the support of currant members, question
members rather than the general audience. In the same
way, if a general listener profile is needed, an accurate
sample of your listeners -- not members --is essential.
Comparative information -- which relates one set of
data to another -- will be helpful in analyzing the
information gathered. For example, the average total
household income of members may be $50,000 -- but this
information means one thing if the average total household
income of market area is also $50,000 and quite another
if it is $20,000. If the station can't survey the general
market, call the local chamber of commerce or the Department
of Commerce for demographic analyses of the area.
IV. Choosing Methodologies
Sampling
unit
Of course, the ideal way to analyze members or listeners
is to interview every single person. The lack of time
and money dictate choosing a much smaller "sample" or
"sampling unit." A sampling unit is that group of individuals
selected to represent the responses and behavior of
the total population to be analyzed. Choosing the sampling
unit is very important -- a poor sample will invalidate
the entire survey.
Sampling
methods
A good sample should resemble the total population as
closely as possible. Statisticians measure the resemblance
through "confidence factors" which measure the probability
that a sample will react like the population from which
it was drawn. Choose a sampling method which increases
the probability that the sample will closely resemble
listeners or members. High probability procedures include:
- Simple
random sampling in which every member of the population
has an equal chance of selection.
- Stratified
random sampling in which samples are randomly drawn
from mutually exclusive groups (such as age),
- Cluster
or area sampling in which samples are drawn from
mutually exclusive groups which are defined by one
or more similar characteristics (such as zip code
and sex).
Non-probability
samplings (which should be avoided)
Include:
- Convenience
sampling in which the group easiest to reach is
selected (such as those who respond to a survey
in the program guide).
- Judgment
sampling in which individuals are selected whom
are thought either to give accurate information
or to support pre-determined expectations.
- Quota
sampling in which a determined number of individuals
in each selected cluster are surveyed.
Sample
Size
The size of the total sample should
depend on two factors: the sampling
method and the smallest "cells" or "clusters"
into which respondents are categorized.
If a high probability sample is chosen,
a minimum of 100 respondents are adequate
for most purposes, regardless of the
size of the population. Remember that "high
probability" means that the sample
will most closely resemble the group
from which it was selected --so more
respondents would be a waste of time
and money.
However, if a non-probability sample is selected, a
much larger sample will be needed. Obviously, the higher
the percentage of the respondents to the population,
the higher the probability that the sample will resemble
the population. And the higher the coats and time involved.
A "cell" is simply one sub-set of the
sample which may be compared with
another sub-set. The size of the over-all
sample should allow the smallest cell
to contain at least 100 respondents.
For example, we'll compare the responses
of news, classical music and jazz members.
Based on on-air fund raising for the
last year, it's guessed that 40% of
the total membership support news,
40% support classical music and 20%
give for jazz. If a minimum of 100
jazz supporters is needed, the estimated
minimum sample would be 500 (100 jazz,
200 classical and 200 news). If listeners
are sampled, use this same approach
with Arbitron numbers.
When selecting a sample size, be sure
to include an additional 25% to
50% to account for incomplete surveys.
Some people may not return the
questionnaires even after repeated
reminders or may not be home to
answer telephone calls. Therefore,
if the estimated minimum sample
is 500, select between 625 and
750 members to be contacted.
Contact
methods
As in all fund raising, there are three
choices of contact methods: mail,
telephone and personal contact
(either individual or group interviews).
For our purposes (and resources),
only mail and telephone approaches
are feasible.
Telephone Interviewing has two advantages:
information can be gathered quickly
and the interviewer can explain
the questions. On the negative
side: only people with telephones
can be contacted (although this
represents the vast majority of
public radio listeners), only those
with listed numbers can be called
(professional services use random
digit dialing and change one digit
of phone book numbers to reach
these people) and the interviews
have to be short and not too personal.
Mailing questionnaires has two advantages: it’s the
best way to reach your entire sample (if addresses are
accurate) and it lacks the bias an interview might bring
to a telephone survey. On the negative side: the questions
must be clearly and simply worded and the response is
usually slow and low.
Survey
formats
Whether the survey is given by telephone,
through a direct mail piece or
in a program guide, include the
following elements:
- Introduction
which explains the purpose and importance of the
survey.
- Warm-up
questions, which are easily asked and provoke interest
in the survey, should be presented first,
- The
main body of the questionnaire.
- Demographic
questions which may be more personal and which may
"turn off" respondents in the beginning of the survey.
- A
warm "thank you" for respondents' participation,
- Easy
mechanism for responses such
a postage-paid envelope, which
may be necessary for getting
a good sample size.
A
prepared script is essential for the volunteers or professional
telemarketers who are conducting a telephone survey.
This script is most important in establishing the initial
contact with the respondent. A telephone survey should
be designed to be completed in 10 to 15 minutes.
In designing a direct mail survey, ask either that the
entire survey to be returned or just a response card.
To get the entire survey back, print a "self-mailing”
four-page questionnaire. With a response card, a traditional
direct mail package is advisable: carrier envelope,
survey with introduction or cover letter, response card
and pro-paid reply envelope, if a cluster sampling has
been chosen, be sure to pre-code your surveys or response
cards. Also consider the importance of anonymity in
mailed surveys -- obviously, the respondents' names
will appear on a "self-mailing" survey.
A program guide survey is least advisable
since it is a "self-selecting" sample.
That is, responses are received
primarily from those individuals
with the time and inclination
to respond, such as retired people.
However, a program guide survey may be used for promotional
rather than statistical purposes. There are many options
for including this kind of survey with a program guide:
- survey
with questions and answers on
same page which would be ripped
out and mailed in the respondents
envelopes;
- saddle-stitched
four-page "Self-mailed" survey
which would be folded, stapled
and mailed; and
- saddle-stitched
pre-paid envelope or survey
response card In the middle
with the questionnaire In the« body.
Recommendations
Let's say the objective is to increase
renewals and improve upgrading
of current members. The strategy
Is to develop a major donor program
and a basic member benefits package.
Let's further assume that there
are two types of listeners: those
who tune In for news and those
who enjoy classical music. Basic
member and major donor benefits
must be attractive to both audiences.
Analysis of the member file Indicates that about 80%
of the members give between $25 (basic membership level)
and $200 (the floor for "major donors"). An analysis
of last year's on-air pledge drives shows that about
60% of listeners support news programming and 40% classical
music. Therefore, there are an estimated 48% of “news”
basic members (60% of 80%), 12% “news” major donors
(60% of 20%), 32% “classical” basic members (40% of
80%) and 8% “classical” major donors (40% of 20%). These
are the four "cells" which will be analyzed:
| Total
Audience: |
100% |
News:
|
60% |
Classical:
|
40% |
| Basic
Membership ($25-$199): |
80% |
Basic/News:
|
48% |
Basic/Classical: |
32% |
| Major
Donors ($200+): |
20% |
Major/News:
|
12% |
Major/Classical:
|
8% |
If the member file has recorded program
interest and level of giving for each
member, the cluster sample approach
can be used. Divide the file into the
four cells then choose 150 members
from each cell (the minimum of 100
plus 50% for the unreachables). If
the file does not record either of
these characteristics, the random
selection approach must be used. In
this method, choose at least 1,875
members in order to get a minimum of
100 classical major donors.
Members can be randomly selected by their ID numbers
in two ways:
- Use
a table of random numbers (as
printed in the National Association
of Broadcasters' booklet "Why
Do Research?”
or in any statistics text); or
- Have
the station's computer whiz
write a random number generating
program using the internal clock,
the digits in pi (π), various
logarithms, a table of random
numbers, or a combination of
these approaches.
A
convenient method of selecting members
if by choosing every 10th or 20th
member from the file. If a sample
of 600 members is required from a file
of 8,000, select every 13th member
(8,000 divided by 600 This is not,
strictly speaking, a random selection
since not all members have an equal
chance of being selected. However,
it's the easiest method and acceptable
if the selection is started with a
randomly generated number.
Once a sample is selected, decide on the format and
length of the survey. As an example, let's choose an
eight-page, "self-mailer" questionnaire with return
postage guaranteed. This will allow an introductory
letter, out-going address panel, return address panel
and between 60 and 80 questions. (Many other alternatives
are available; this example was chosen for discussion
purposes.)
In the introduction, stress the fact that the member
was carefully chosen to respond to the survey and that
the member's response is critical. If a minimum number
of respondents haven't returned their surveys at the
end of two weeks, follow up with another copy of the
questionnaire. If enough surveys still aren't received,
try calling the recalcitrant members. Another sample
may be selected to get the required response.
V. Developing Questions
Relevant
to research objectives
Make sure each question actually contributes to the
objective of the survey. Ask "What decision will be
made based on the information received from this question?".
Avoid questions which would simply provide "interesting"
information or that are just Irrelevant to the purpose
of the research.
Relevant
to respondents
Be sure that the questions are relevant to the sphere
of the respondents’ knowledge, understanding and interest,
for example, avoid questions that are outside the respondent's
experience (such as some political issues) or those
that rely too heavily on memory (such as specific news
stories over the last month).
Type of information Questions may be classified by the
kind of information requested:
- Attitudes,
or what people say they want
- Perceptions,
or what people think is true
- Behavior,
or what people actually do
- Attributes,
or what people are
Be
careful to differentiate between these
types of questions. Otherwise, some
incorrect assumptions will be drawn.
For example, if first-time members
were asked what was important in prompting
them to support the station, perhaps
5% will mention a premium. Yet, if
asked whether they actually received
a premium when they became members,
40% may say "yes." Clearly, there
is a difference between their attitude in becoming members and their actual
behavior In requesting a premium.
Form
of questions
Some of the forms of questions are:
- Dichotonous,
in which the respondent chooses
one of two answers, such as "yes" or "no": "Have
you listened to WXYZ at least
once In the past week? A) YES
b) NO."
- Multiple
choice, in which a number of
answers are offered (either
a single or multiple answers
may be acceptable);
"To what type of programming on
WXYZ do you listen: a) News b)
Classical c) Jazz d) Folk".
- Likert
scale, which indicates the amount of agreement with
a statement: "How strongly do you agree or disagree
with the following statement: WXYZ plays only 'top-forty”
classical music: a) strongly agree b) agree c) neither
agree nor disagree d) disagree e) strongly disagree".
- Semantic
differential. In which the respondent selects a
point on a scale between two opposite words to correspond
with their perceptions: "The program hosts of WXYZ
are generally: Professional |__|__| X |__|__|__|
Amateur".
- Importance
scale. In which a respondent rates the importance
of some attribute: "Receiving the station program
guide to me is: a) Extremely important b) Very important
c) Somewhat important d) Not very important e) Not
important at all".
- Rating
scale, in which an attribute is rated excellent,
very good, good, fair, or poor: "The program information
in the program guide is: a) excellent b) very good
c) good d) fair e) poor".
- Open-ended,
which allows the respondents
to select their own, unaided
answers. Note that these questions
provide the most accurate answers
but are the most difficult to
tabulate.
Wording
of questions
When writing the actual question, keep these guidelines
in mind:
| -
Use simple words |
| |
NO:
Numerate the co-habitants of your domicile |
| |
YES:
How many people are in your household? |
| -
Make question clear and specific |
| |
NO:
Do you listen to public radio? |
| |
YES:
How many hours last week did you listened to WXYZ? |
-
Keep your questions short |
| |
NO:
Please indicate the public radio programming which
was the primary influencing factor in precipitating
your financial support of WXYZ |
| |
YES:
What type of program prompted your support of
WXYZ? |
-
Avoid bias |
| |
NO:
Do you spend more time listening
to mass-market commercial
radio than to intelligent commercial-free
public radio? |
| |
YES:
To what other radio stations have you listened
in the past week? |
-
Avoid embarrassing or very personal questions |
| |
NO:
What is your exact salary? |
| |
YES:
Into which range would your
salary fall? |
-
Don't use double questions |
| |
NO: Do you own a TV and a VCR? |
| |
YES: Do you own a TV? Do you own a VCR? |
-
Avoid negative questions since they may confuse
your respondents |
| |
NO: You don’t support other non-profit groups,
do you? |
| |
YES: Do you support other non-profit groups? |
Sequence of questions
Lead questions should create interest
in the survey and should be more
general in nature. More difficult,
special, or personal questions
such as income range should be
asked near the end so that the
respondents do not become defensive.
Sequence should also not direct
the respondents. An example of
a leading sequence of questions
is "Are you aware or any station
that programs primarily news and
information" followed by "What
type or programming does WXYZ
have?" Obviously, the respondent
is being 'led" to reply that WXYZ
has news programming.
Number
of questions
Accumulating as much information as
possible is always wanted but remember
that every question costs time
and money. The more questions,
the more time that is required
for the telephone interviewe, for
data entry, and for analysis. However,
a long direct mail survey will
not necessarily discourage people
from responding. In fact, WMFE
In Orlando conducted a psychographic
survey with over 200 questions
and still received a relatively
high response rate.
Testing
Make sure the questions will mean the
same for all the respondents by
testing them with a small sample
of members or listeners. Explain
why their help is needed and encourage
them to find flaws in wording
and to make suggestions for improving
the survey. A little time spent
on testing now may prevent Invalidating
some questions later.
Sample
questions
if time Isn't available to create a
questionnaire or some ideas are
needed, write or call the Development
Exchange. Sample market research
questions were culled from over
twenty public radio stations. The
questions are grouped by the following
categories:
- Consumer
habits
- Demographic
Information
- Fund-raising
information
- Listener
habits
- General
media habits
- Perceptions
- Programming
interests
- Psychographic
information
- Specialized
Information
VI. Collating Information
The way Information is collated and analyzed will depend
on the computer resources available.
Manual
analysis
First, divide the returned samples
Into the clusters to be analyzed,
such as the listeners who tune
to news, classical music and jazz.
Second, develop a form which has
the number of questions down the
side and the number of the answers
across the top. Then, go through
each of the forms and make a mark
in the proper column for the answer
Indicated for each question. At
the end, count the marks for each
answer and calculate the percentage
by dividing the number by the
total number of responses for
the question.
For open-ended questions, list all
the responses, or types of responses,
then indicate how often that answer
was given. This will give only
a simple analysis of the results.
Cross-tabulation, or analyzing
smaller cells of the results,
requires either more tedious manual
analysis or some kind of database
computer software.
Database
computer software
With access to a personal computer,
any database software can be used
to aid analysis. Either hire a
computer whiz, get a computer-savvy
volunteer or recruit a staff member
to create a file structure, input
the data and write programs for
different analyses and report formats.
By computerizing Information,
cross-tabulation of the data is
possible. For example, the number
of classical music listeners who
both own homes valued at $100,000
and give less than $100 per year
can be determined.
Specialized
marketing research software
In the best of all possible worlds, a marketing research
company would donate their time and software to analyzing
your data. For example WAMU, Washington, received computer
analysis services from one of their underwriters. Professional
marketing companies can provide detailed cross-tabulation
analyses, graphics and measurements of statistical significance
and confidence limits.
VII. Developing a Budget
A budget is needed whether in-house resources, donated
services, or a foundation grant is used to fund a market
research project. A detailed budget will not only outline
the time and money needed but will indicate to a potential
grantor the amount of resources devoted to the project
and will indicate the market value of any donated services.
Here is a sample budget (pricey for some but cheap for
others) for a member survey with some donated services
and a grant request.
| DESCRIPTION
OF EXPENDITURE |
TOTAL
EXPENSE |
DONATED/
IN-KIND
|
REQUEST |
| PLANNING |
|
|
|
| Station:
determining objectives, preparing questions (40
hours @ $17/hour) |
$680 |
$
680 |
|
| Research
Services: designing questionnaire (usual fee:$500) |
500 |
500 |
|
| DISTRIBUTION |
|
|
|
| Printing
2,000 survey packets: questinnaires, letter, envelope,
reply envelope (2,000 @ $0.30) |
600 |
|
600 |
| Postage
(2,000 @ $0.071) |
142 |
|
142 |
| Mailing
Services (volunteers) |
100
|
100 |
|
| Return
postage (1,300 @ $0.30) |
390 |
|
390 |
| DATA
PROCESSING & ANALYSIS |
|
|
|
| Research
services: data entry (1,300 @ $0.50) |
650 |
|
650 |
| Research
services:computer tabulation and report writing
(20 hours @ $150) |
3,000
|
1,500 |
1,500 |
| TOTAL
EXPENSE |
$6,062 |
$2,780 |
$3,282 |
| SOURCE
OF INCOME |
AMOUNT |
|
|
| Station
Resources |
$ 780 |
|
|
| Research
Services |
2,000 |
|
|
| Requested
from Philanthropic Foundation |
3,282 |
|
|
| TOTAL
INCOME |
$6,062 |
|
|
VIII. Conclusion
Local market research is a valuable
tool in facilitating programming,
promotion and fundraising decisions — and
providing necessary information
for potential funders. However,
the information gathered is only
as good as the approach taken.
Once again the steps in conducting
and Implementing market research
are:
A.
Determine objectives and strategies
1) Determine objectives.
2) Apply objectives to decision-making.
3) Determine the Information needed.
4) Select the source of information.
B.
Determine methodologies:
1) Choose sampling methods and sample size
2) Determine contact methods and survey format
3) Write questions and/or scripts
4) Test questionnaire
C.
Administer survey
D.
Collate information and analyze results
E.
Make marketing decisions based on information
Good luck! And share your successes with the Development
Exchange!
|
Writing