Transcript 10.OANP Presentation Evaluating Survey Evidencex

Evaluating Survey Evidence:
A Tale of Two Surveys
By:
Brian Blake, Ph.D.
Jennifer Vedder
Erin Sonenstein
Consumer-Industrial Research Program
Cleveland State University
Vanguard Conference
National College of Natural Medicine
Portland, Oregon
January 11, 2011
Too often we hear…
“A survey usually raises more questions than it answers!”
“Be careful that a survey doesn’t lead you in a wrong
direction!”
Too little do we listen…
“A goal without a plan is just a wish.”
- A. Saint-Exupery, 1900-1944
“You designed this research backwards!!!”
Five Key Questions
1.
2.
3.
4.
5.
Are we listening to the right people?
Are we listening to enough of the right people?
Can we trust what they say?
Do we understand what they tell us?
Does the information provide guidance for action?
Two Study Designs
 “Gen-Pop”
 ND Survey
“Gen-Pop”
 Descriptive and Prescriptive
Online survey of Oregon residents to:
 Identify the public’s familiarity with NDs
 Gauge the public’s image of NDs relative to other health care providers




(HCPs), broadly defined
Unearth the distinct segments that compose the Oregon population, the
segments being sectors that differ in regard to their orientations toward
medical care
Estimate the demand for naturopathic medicine (NM) and how it changes
from one segment to another
Calculate the “targetability” of each segment, i.e., the relative
contribution of each segment to the spread of NM in Oregon
Devise a marketing/communication/service distribution strategy based
on the above to increase usage of NDs and NM care in Oregon.
“Gen-Pop”
 Discover parameters that can be effectively brought to the
attention of Oregon legislators and health insurance carriers
that can provide a rough gauge of:
 Unmet health care needs in Oregon
 Public satisfaction and particular difficulties experienced with health care
coverage
 Readiness to turn to NDs as a Primary Care Physician
ND Survey
Mainly Descriptive
Online survey of the nation’s ND community to:
 Profile the ND community in America in regard to a variety of
parameters:
o
o
o
o
o
o
o
o
o
Engagement in particular professional activities
Professional goals and interests
Provision of alternative treatment modalities
Form of reimbursement
Involvement in a primary care role
Reliance upon various forms of medications in prescriptions
Income from NM and other income streams
Demographic characteristics
Licensures and certifications beyond the ND
ND Survey
 Chart the diversity of NDs in the above regards and identify
sectors or “types” of NDs that compose the contemporary ND
community in America.
 Assess the needs and priorities of each ND type/sector in order to
learn how to serve their distinct interests.
 Identify graduates of ND programs who are not presently involved
in NM. Who are they? Why are they no longer active?
 In conjunction with information from the Oregon General
Population survey explore the viability of mutually beneficial
referral networks with non-ND healthcare providers.
Five Key Questions
1.
Are we listening to the right people?
2.
Are we listening to enough of the right people?
Can we trust what they say?
Do we understand what they tell us?
Does the information provide guidance for action?
3.
4.
5.
Q1: Are we listening to the right people?
 “Right people” = Representative sample of population
 So, how do we obtain a representative sample?
 Six steps to a representative sample…
For a representative sample…
1.
Specify population clearly.Who is in?Who is out?


Gen-Pop… adults (+18 years old) in Oregon who have received care
from a health care provider in the previous 2 years (from January 1,
2009)
ND Survey… persons who graduated in the past ten years (since 2001)
with an ND degree from an accredited US college and/or who are
current members of the American Association of Naturopathic
Physicians (AANP) or a state level ND organization (e.g., OANP).
Does not include current students working toward an ND degree.
Includes ND degree holders not presently active in NM
For a representative sample…
2.
Devise the “sampling frame,” a population listing with
reasonable coverage current and is error free
Gen-Pop - members of an online panel residing in Oregon previously
screened for age and medical care based on vendor records and
confirmed by questions in survey. Like e-Rewards
 Evaluation - practical, not perfect
- coverage OK, but suspect
- current and clean
• ND Survey - listing of alumni records of five major ND colleges in US,
membership list of AANP, membership listing of all state level ND
organizations, compiled-cleaned by HRI

For a representative sample…
3. Draw probability sample
Here a “stratified random sample”
• Gen-Pop - sample proportionate to population of Oregon
counties (most recent US Census estimate)
•
ND Survey - sample drawn proportionate to the ND
population in each of 50 states (% calculated from sampling
frame)
For a representative sample…
4. Design to obtain a high response rate
Dillman review
Finding in ancient crypt
Procedure
High rate essentials
Gen-Pop
ND Survey
1. Pre-notification

2. Personalized cover letter

3. Guaranteed anonymity


4. Incentive


5. “Noble cause”


6. Credible sponsor
?

7. Accessible contact person


8. Fairly easy to answer questionnaire


9. Multiple (3) waves


For a representative sample…
5. Design so that subsample weighting of obtained data is possible
 Gen-Pop - double weighting
 by county to make data set proportional to county’s population (Census
based)
 by those few demographics (e.g., age) empirically shown in study to be
predictive of readiness to see an ND (Census based)
 ND Survey - single weighting
 by state (based on percent of sample frame)
Five Key Questions
1.
Are we listening to the right people?
2. Are we listening to enough of the right people?
3.
4.
5.
Can we trust what they say?
Do we understand what they tell us?
Does the information provide guidance for action?
Q2: Are we listening to enough of the right
people?
A. Standard formulae based on:
•
•
•
•
•
desired precision, e.g., ±3%, ±2% band
desired level of confidence that population characteristic is within band,
often 95%
variability in characteristic of interest
• more variability, larger sample needed
• typically assume maximum
• population vs. sector
number/size of subpopulations to be analyzed “stand alone,” males vs.
females
projected response rate, e.g., if project 50% response, send out double
needed sample size
B. Gen-Pop
 Given: population =3.8 million (adjusted from 3,421,399)
qualified (had care) = 70%, response rate = 20%
variability = pop. max (50-50), seg mod (70-30)
For OR whole need 600
precision = +/- 4%
confidence = 95%
send out = 3,000
 For typical (of 5) segment with average 150 per segment
precision = =/- 7%
confidence = 95%
send out = 3750
C. ND Survey
 Given:
 For population (US) as a whole = 6,000,
response= 50%, 90% qualify
 Variability = pop max (50-50), seg mod (70-30)
precision = ±3%
confidence = 95%
 For typical (of 5) segment
precision = ±6%
confidence = 90%
D. Recommend
 Gen-Pop
need 750 (assuming 5 segments)
send out 3750 (assuming 20% response)
possible “booster” sample if familiarity with
NDs low
 ND Survey
needed - 750
send out - 1,500
Five Key Questions
1.
2.
Are we listening to the right people?
Are we listening to enough of the right people?
3. Can we trust what they say?
4.
5.
Do we understand what they tell us?
Does the information provide guidance for action?
Q3: Can we trust what they say?
 Eliminate “bad respondents,” for example
 overstatements
 suspicious patterns of answers
 Eliminate bad items for “illogical” answers based on
relationships (correlations) among items
 For Gen-Pop
 health care provider (HCP) familiarity with licensed phentologist
 What they say versus what they do. Stated Impact vs. Predicted Impact
for HCP characteristics
Q3: Can we trust what they say?
 Evaluation
 Minimal checks for ND
 Practical approach for Gen-Pop
Five Key Questions
1.
2.
3.
Are we listening to the right people?
Are we listening to enough of the right people?
Can we trust what they say?
4. Do we understand what they tell us?
5.
Does the information provide guidance for action?
Q4: Do we understand what they
tell us?
 How to interpret a given question
 e.g., suppose numerous NDs say they place high priority upon time to educating
the public about health care issues?
 Why do so? Good for business? Desire to advance NM care? Enjoy being in
limelight?
 Use of mathematically derived dimensions that “look underneath” overt
answers in both Gen-Pop and ND Surveys
 Example of dimensions in image of NDs and other HCPs in Gen-Pop
First, the HCPs considered
 Second, the characteristics
 Then, the dimensions

HCPs Included
 A conventional medical doctor (MD degree)
 Naturopathic doctor (ND degree)
 Licensed physicians assistant (PA) for primary care
 Licensed practical nurse/licensed nurse practitioner/registered nurse for primary care
 Licensed midwife/Dula
 Psychiatrist (MD)
 Psychologist (PhD/PsyD) or other licensed mental health professional
 Chiropractor (DC degree)
 Physical therapist (PT degree)
 Certified massage therapist
 Licensed acupuncturist
 Registered dietician
 Yoga, other mediation/controlled movement trainer
How well are particular HCPs described by each characteristic?
(Scale 1-5)
Characteristic
ND
MD
Rest
Prescribes principally natural healing methods (such as medicines from plant
products)
4.1
1.5
2.0
Emphasizes preventative medicine to avoid ailments before they occur
4.1
2.1
3.1
Spends time getting to know me as a patient
4.2
1.9
2.5
Uses conventional healing methods (for example: acupuncture, herbal
remedies)
4.3
1.2
3.1
Makes it simple to schedule appointments that fit into my schedule
3.4
1.2
3.9
Prescribes only medications that have had significant scientific research and
have been approved by the Food and Drug Administration
…
…
…