Transcript ab120908_part_2_ch03-ch07_slides_1

Health Informatics
Research Methods:
Principles and Practice
Part 2: Chapters 3-8
Survey Research
Chapter 3
Overview of Survey Research
– Choose a topic of study
– Formulate criteria to develop questions about that
topic
– Can explore a disease, community, organization,
culture, health information system or software
etc.
– Random sample of subjects chosen to answer the
questions in a standardized format
Overview of Survey Research
• Develop the questionnaire
• Develop the cover letter that explains the
study and directions for completing the
questionnaire
• Pretest the questionnaire for validity and
reliability
• Disseminate the survey via email, mail, Web,
fax or use the survey during interviews
Overview of Survey Research
• Researcher must choose the best medium to
disseminate survey
• Overall goal of survey research is to collect the
most appropriate and accurate data that will
answer the questions pertaining to the
research topic.
Survey Creation
• Use/Adapt Existing Surveys
– Health Information National Trends Survey
(HINTS)
• Created a population-based survey that tracked trends
in the use of communication technologies as a source
of cancer information
Survey Creation
• Use/adapt existing surveys cont’d
– National Center for Health Statistics
• National Health Interview Survey (NHIS)
• National Ambulatory Medical Care Survey (NAMCS)
• Health Assessment Questionnaire
• Adapt several different questionnaires into
one
New Survey Development
• Need to consider the following items:
– Content
– Audience
– Medium
– Sample or survey entire population
– Statistics to be generated
Advisory Committee
• Focus group of experts in survey design and
the topic under study
• Assist in phrasing the questions
• May also seek assistance from
organizations/departments which provide
assistance and guidance in survey design,
development, and analysis of results
• Inclusion criteria
Types of Questions
• Open-ended (unstructured or qualitative)
• Close-ended (structured or quantitative)
• Scales:
– Nominal
– Ordinal
– Interval
– Ratio
Pilot Test Survey
• Pilot test survey on a small group of
respondents
• The sample should reflect the true sample of
respondents
• Provide accurate simulation of administration
of survey questionnaire
• Review all comments and discuss with
advisory board
• Incorporate into final survey
Test Survey for Validity
• Face Validity
Examines how survey looks
• Criterion-related validity
Accuracy of intended
survey
• Construct validity
Agreement between
theoretical concept and
survey
• Content validity
Survey captures the
information intended to
measure
Test Survey for Reliability
• Reliability or consistency of
survey
• Reliability Coefficient :
– Cronbach’s Alpha
• Measures whether survey has
internal consistency—Do all variables
measure the same concept?
• Should be measured on an interval or
ratio scale
• Normally distributed
• Reliability Coefficients close to 1.00
have very high internal consistency
or reliability
– Test-Retest for Reliability
• Measures whether survey is
consistent over time or when given
multiple times
• Correlation coefficient between the
relationship between two total
scores given two different times
• Those coefficients close to 1.00 show
strong reliability.
Factor Analysis in Refining Survey
• Sometimes researchers need to refine the
number of questions used in a survey.
• Factor analysis is a statistical technique in
which a large number of variables are
summarized and reduced down to a smaller
number based on similar relationships among
those variables.
Audience
– Need to know your audience so that questions on
the survey can be built so that they are able to
answer them
– Use clear, unambiguous terms
– Do not use terms that are unclear, such as “not
sure”
– Pilot test the survey to capture terms that should
not be included in the original development of the
survey
Framing of Questions
• Order of questions is important
– Demographic data is usually first, followed by
more broad or general questions, followed by
more specific questions with ranked or ordinal
type responses or open-ended type questions
• Provide a checklist of possible responses
– HIPAA Privacy Rule Implementation Study by
Firouzan
Incentives
• Consider whether incentives should be
provided to individuals who complete the
survey
• Some incentives may influence responses so
be careful when choosing the type of
incentive
• Researcher should decide if it is necessary and
what type of incentive should be used if any
Confidential Reponses
• Discuss how the individual’s responses will be kept
confidential in the cover letter or instructions
• Institutional Review Board (IRB) will also need to see
how information collected will be kept confidential
• Examples:
– individual non-identifying number and password can be
part of the URL in a Web-based survey,
– no identifying information linked with survey,
– separate database used to collect demographic
information but not linked to responses
– All information reported in presentations and published
material should be reported in aggregate form
Limitations
• Every study design has limitations
• Survey research design includes the following
limitations:
– Inaccurate responses due to:
• Not understanding the question or instructions
• Not having appropriate time to complete the survey
• Not able to recall past experiences to answer the
question
• Too long and respondents may tire
• May exaggerate responses to questions—i.e. salary
Reduce Limitations By
• Ask questions that:
– are unambiguous
– Do not require extensive recollection
– Jog the respondent’s memory with pictures,
graph, table etc.
– Provide ranges to choose from especially if the
question is more sensitive in nature
Type/Medium
•
•
•
•
•
Web-based
Email
Mail
Fax
Group
Web-based Surveys
• Advantages:
– Reduced cost when compared to paper
– Little or no data entry
– Ease of data analysis
– Use of pop-up instructions and drop-down boxes
– Ability to present questions in random order
• (Gunn, 2002)
Web-based Surveys
• Disadvantages
– Missing respondents who do not have a computer or
access to Internet
– Increased up front time in the development of the
questionnaire
– Hire a person with skills in Web-based survey design
and development
– Difficult making changes to survey once on the Web
– Respondents more reluctant to provide responses
over Internet due to lack of confidence in privacy and
security
Other Types of Surveys
• Mail—Still used effectively in health informatics
especially when surveying physicians, nurses and other
health care providers
• Email—very similar to Web-based surveys but time to
develop may not be as extensive, however, automatic
response to database is limited
• Fax---Similar to mail surveys and can be used so that
one captures all possible respondents
• Group---Paper survey made available to participants at
a conference, retirement community, physician
practice etc.
Examples of Survey Research in Health
Informatics
• EHR/ASTM study by Watzlaf et al, 2004 used
all types of surveys (Web-based, mail, fax,
group) with the major focus being Web-based.
Intent was to increase response rate using all
types of medium.
• Criswell et al, 2002 used mail surveys to
determine if physicians in family practice
residency programs used PDAs
Examples of Survey Research in Health
Informatics
• Murff et al, 2001 used mail surveys for physicians
to determine their satisfaction with using two
different CPOE systems. Used the Questionnaire
for User Interface Satisfaction (QUIS).
• Patel et al, 2005 used both survey and face-toface interviews to collect pre-and post-course
data on students enrolled in the Woods Hole
Course in Medical Informatics. Interviews were
used to supplement the survey questions.
Examples of Survey Research in Health
Informatics
• Couper et al, 2007 used both mail surveys and
phone interviews to collect data on weight
management and also were interested in
which method produced a better response
rate.
• Results: phone interview (59% response rate)
and mail (55% response rate).
Distribution of Survey
• Researcher should determine HOW the survey
will be distributed
– Email or phone each of the respondents selected to
participate and explain the purpose of the study and
their role
– Ask if they are willing to participate
– Determine the best way to send the survey and obtain
contact information for them
– Assure respondent that their answers will be kept
confidential and that only aggregate data will be used
when reporting results
Distribution of Survey
• Discuss the cover letter content
• Discuss incentives if they are used
• Once the respondent agrees to participate, contact
them via email, mail, fax, etc. and provide a copy of the
cover letter and survey.
• Make sure that the deadline date to return the survey
is bolded in the cover letter (See Appendix 3C for
sample of Cover Letter)
• After 2 weeks with no response, send Follow-up letter
and reiterate the importance of study and the need for
their participation. (See Appendix 3E for sample of
Follow-Up Letter)
Sample and Sample Size
• Most survey research is performed on a
sample
• Census survey includes the entire population
• If a sample is chosen, make sure that it
includes an accurate representation of the
population under study. In this way, the
characteristics of the sample participants are
similar to the population characteristics.
Sampling Methods
• Stratified Random Sampling—separate the
population by certain characteristics such as
physician specialty, nursing units, DRGs, and
then choose the sample
• Systematic Random Sampling---draw the
sample from a list of items such as diagnoses,
I9 codes, or discharges and select every nth
case
Sampling Methods
• Cluster Sampling—separate first into a city
block, randomly choose residences and then
sample everyone within that residence
• Convenience Sampling—not random, can
generate quick results but the results should
not be generalized to the population.
Example: When one surveys everyone in a
specific HIT department to determine their
knowledge of HL7
Sample Size Calculation
• Refer to Table 3.5 and pages 68 and 69 for an
Example of the Sample Size Calculation for the
EHR study by Watzlaf et al, 2004.
Response Rate
• Very important in survey research
• Even a low response rate may prove beneficial
if it is in an area or topic that has not been
researched in depth before
Response Rates
• Methods used to increase response rates include:
– Follow up letters, emails, phone calls, fax
– Do not be overly annoying
– Include the title of the research study, when the
survey questionnaire was sent, the importance of the
study, and how important the respondent’s reply is to
the research study
– Reiterate how data will be kept confidential
– Explain any incentives
– Attach survey again –do not make the respondent
look for previous survey sent
Statistical Analysis of Survey Study
Data
• Usually quite simple
– Frequencies and percentages
– Correlation coefficieints
– Tests of significance
– Confidence Intervals
– Open-ended questions should be analyzed using
content analysis
Examples of Statistical Analysis of
Survey Study Data
• See Tables 3.6 – 3.12 for a simple data display
for the EHR study by Watzlaf et al, 2004.
Summary
• Development of survey instrument
• New survey or adapt existing one
• Address the following areas:
– Content
– Audience
– How Administered (Type/Medium)
– Sample or Population
– Type of Statistics
Summary
• Choose advisory committee of experts to
review the survey
• Determine the different types of questions
and response types or scales
• Pilot Test the survey
• Feedback reviewed with Advisory Committee
• Make changes based on the pilot survey
Summary
• Validity and Reliability Testing
– Cronbach’s alpha
– Test-Retest
– Factor Analysis
• Other issues to consider
–
–
–
–
–
Incentives
How to maintain confidentiality
Minimize bias or error
Increase response rate through follow-up
Appropriate statistical analysis
Health Informatics
Research Methods:
Principles and Practice
Chapter 4
Outline of Discussion
• Observing
• Why do Observational
Research
• Non-Participant
• Naturalistic
• Simulation—Usability
• Case Study
• Focused Interview
•
•
•
•
•
•
•
Informal Conversational
Standardized Open-Ended
General Interview Guide
Focus Group
Participant Observation
Ethnography
Content Analysis
Observing
• Take a minute to observe
something near you. It can
be anything on your desk, in
your office or home.
• Look at its color, shape,
function, age, etc.
• Jot down some information
about it.
• Let’s discuss
Introduction
•
•
•
•
•
•
Qualitative research
Perceptions
Interactions
Feelings
Attitudes
Depth
Why choose Observational Research?
•
•
•
•
•
New topic
Transitional program
Background for larger study
Robust, rich data
Observation, field
interviews, medical record
reviews, ethnographic
methods, combination of all
Attention To
•
•
•
•
•
Observation site
Time period
What will be observed
How it will be recorded
Who will conduct the
observation
• How data will be analyzed
• How results will be
disseminated
Non-Participant Observation
• Non-Participant
Observation
• Observes actions of study
participants with limited
interference.
–Example: Observing how
employees react to a new
documentation software system
• Three types: Naturalistic
observation, simulation
observation, case study
Naturalistic Observation
• Observing behaviors or
actions that occur naturally
in the environment
• Useful to see if study
participants are following a
procedure, rule, law, policy
• Participants should not
know what the researcher is
observing or when in order
to simulate the “normal”
environment.
Naturalistic Observation Example
– We examined the reasons for
underutilization of the cancer registry
• Informal discussions with
physicians
• Naturalistic observations of cancer
registry
• Semi-structured interviews with
physicians and nurses
• Compile data to enhance the new
cancer registry software system
•
Can anyone think of another example of
naturalistic observation in HIM?
Simulation Observation
• Naturalistic observation
conducted in environment
created for participants
– Hackett, Parmanto et al, 2007
dissertation research
observed individuals with
visual disabilities using
different websites chosen for
the participants. Goal is to
see if the transcoder is
effective in improving
websites for individuals with
visual disabilities.
– Usability study---different
tasks developed
Usability Studies
•
•
•
•
•
How fast can the learner use the PHR system (Ease of
learning)?
How fast can the learner accomplish the tasks? (Efficiency
of use)?
How effectively can the user learn the PHR system upon
several different uses? (Memorability)
How often do errors occur, how serious and what do users
do to manage the error? (Error frequency and severity)
How much does the user like using the system? (Subjective
Satisfaction)
(Hackett et al, 2007, Dissertation Research, An Exploration into Two Solutions to Propagating Web Accessibility for Blind Computer Users)
Case Study
• Non-participant observation
• Individual, group,
institutional case study
• Collects demographic,
disease, religious, social,
cultural, technological
system and software,
community etc.
• Field notes, tape recordings,
video, images, etc.
Examples Case Study
• Who can provide an
example of an
individual, group or
institutional case study?
• How does the case
study differ from the
ethnographic analysis?
Focused Interview
• Informal conversational
• Standardized openended
• General Interview guide
Informal Conversational
• No set questions developed,
moves forward based on
what the study participant
would like to discuss
– Example: What is the
meaning of living with
dementia while in a nursing
facility?
– Other examples in HIM?
Standardized Open-Ended
• Specific questions are
used to interview the
study participants
– Example: Used to
study automated
coding software and
the potential to
decrease fraud and
abuse---See
examples
Examples of Standardized Open-Ended
•
Automated Coding Study:
–
Government:
•
•
What problems do you
foresee in relation to fraud
and abuse when the
Electronic Health Record
(EHR) is used?
Are you aware of incorrect
coding or abuse detected
with Natural Language
Processing (NLP)? If you are
familiar with the approach of
the NLP, was it a rules-based
approach or data-driven
approach? Please describe.
Examples of Standardized Open-Ended
•
Vendor
–
–
–
What type of automated
coding system do you
provide?
When was your first
installation of the
automated coding system?
How many installations
(users/clients) do you have
and in what settings?
What is the average
installation and training
time?
Examples of Standardized Open-Ended
•
User
–
–
–
–
What is the level of
accuracy on coding and
billing?
How is the automated
coding system used with
the EHR?
How is the automated
coding system used within
the coding and billing
process?
What are the anti-fraud
features available and how
do they link to the
automated coding system?
Other Examples—Standardized OpenEnded
– Other examples of
how the
standardized openended interview
could be used in
HIM?
General Interview Guide
• Outline of issues is used
to conduct the informal
interview
– Examples: Medical
schools used the
general interview
guide to reassess
team based learning.
– Other examples in
HIM?
Focus Group
• A focus group is a group of
subjects, usually experts in
the particular area of study,
who are brought together
to discuss a specific topic.
– Example: Focus group of
experts used in evaluating
ICD-10-CM and its
effectiveness in capturing
public health related
diseases—See example
Example of Website Tables – Alzheimer’s progression
Step 1
Step 2
Step 3
Questions
1. After review of the public health diagnoses reportable list,
are there any diagnoses that should be added, deleted or
changed? If so, please explain.
2. Do the hypotheses and explanations that relate to the
coding of the reportable diagnoses provide enough
information so that changes to the coding system can be
made? If not, please specify which sections need further
detail.
3. Do the ranked scale data and explanations related to
differences in the ICD-10-CM and ICD-9-CM coding systems
make sense? Do you need additional information to clarify
any cases? If so, which ones.
4. Based on the information provided to you, what
recommendations do you have to improve the ICD-10-CM
for public health reporting?
Focus Group
• Other ways focus group
can be used in HIM
Participant Observation
• Observer part of
environment being
observed
• Will true feelings etc. be
observed when
participant is also
observer?
Example of Participant Observation
• Hofler et al, 2005 HIPAA
compliance
Ethnography
• Delving into a particular
culture or organization
in great detail in order
to learn everything
there is to know about
them and to develop
new hypotheses.
Ethnography
• Not objective
• Includes opinions of researcher
• No two ethnographers will examine a specific
culture or organization the same way
• Focuses on people, culture, life
Field Notes
• Brief notes written at field
site
• Description (everything
researcher can remember
about the event such as a
meeting, encounter etc.)
• Analysis (linking step 1 & 2
to research questions)
• Personal viewpoints
(Hall, 2007)
Ethnography
• Can include both qualitative and quantitative
approaches
• Participative and non-participative methods
– With participative: extensive field notes
– Open-ended and unstructured interviews
– Documents pertinent to setting
Ethnography--Cyclical
Create
new
hypothesis
Review
Literature
Revise
tentative
hypotheses
Develop
Tentative
Hypotheses
Analyze
Data
Collect
Data
Day in the Life
• Let’s discuss our
ethnographic examples
Ethnography Questions
•
•
•
What parts of the ethnography did you find the
most difficult to collect and why?
Do you think the ethnographic method is a good
way to collect research data? Why or why not?
Do you think the ethnographic method is the best
method to use when collecting observational
research data related to health care and health
informatics? Why or why not?
Analysis of Data
• Content Analysis
– Examine textual data
to detect recurrent
terms and emerging
themes reflective of
culture or facility
– Examples
• Automated coding
study
Constant Comparative Method
• Grounded Theory
• Comparing incidents applicable to each
category
• Integrating categories
• Define the theory
• Writing the theory
Example--CCM
• Tang, 2007—Patients with Colorectal Cancer
– Open analysis of interviews
– Transcripts read and words reflected respondents’
ideas and thoughts labeled
– Patterns derived from actual wording
– Categorizations developed from interpretation
and grouping of codes
– Words with similar meaning grouped into
categories for further analysis
– Grouping categories developed
– All information reflected back to theory or
objectives of study
Software
Content Analysis
• Count word frequencies
• Category frequencies
• Cluster analysis—groups together words used
in similar contexts
• Co-word citation—examines occurrence of
pairs of words
• May be expensive and require training
Summary
• Non-Participant Observation: Observes actions of
study participants with limited interference.
Example: Observing how employees react to a new
documentation software system
• Participant Observation: Researcher is a part of the
environment he or she is observing Example: Used
to assess how well employees in health care facilities
abide by HIPAA
Summary
• Naturalistic: One example of non-participant observation in
which behaviors and events are recorded as they occur
naturally in the normal environment
– Example: naturalistic observations conducted to determine whether
the cancer registry is underutilized
• Simulation: Observing participants in an environment that
has been created for them rather than their normal
environment
– Example: observing individuals with visual disabilities use different
websites to determine if the websites are accessible.
Summary
 Case Study: Non-participant observation when the researcher wants to thoroughly
assess an individual, group, or institution.
 Individual: Record and collect as much information as possible about a particular
individual as they progress through a certain disease, procedure, treatment, cultural
or system change.
 Example: individual patient is followed and evaluated to determine how they use
an assistive technology device.
 Group: Very similar to individual case study except that interviews or observations are
conducted on a group of individuals
 Example: Assessing the moral reasoning skills of medical students using ethical
issues.
 Institutional: Observing a particular health care institution or facility to determine
how it conducts a particular process, system or procedure.
 Example: Department of Veterans Affairs health care system used the institutional
case study method to describe how their electronic health record system is used
in home-based primary care programs
Summary
•
•
•
•
•
Focused Interview: Interview used in observational or qualitative research in to collect
in-depth, rich, robust information.
Focus group: Group of subjects, usually experts in the particular area of study, who are
brought together to discuss a specific topic.
– Example: Focus group of experts used in evaluating ICD-10-CM and its effectiveness
in capturing public health related diseases.
Informal conversational: No set questions developed, moves forward based on what
the study participant would like to discuss
– Example: What is the meaning of living with dementia while in a nursing facility?
Standardized open-ended: Specific questions used to interview the study participants.
– Example: Used to study automated coding software and the potential to decrease
fraud and abuse
General interview guide: Outline of issues used to conduct the informal interview
– Example: Medical schools used the general interview guide to reassess team based
learning.
Summary
• Ethnography: Delving into a particular culture or
organization in great detail in order to learn
everything there is to know about them and to
develop new hypotheses.
– Example: Used to assess interactions between
physicians and patients when using the EHR.
Questions
Experimental and
Quasi-Experimental Research
Chapter 5
Experiment
•
•
•
•
•
•
Begin with a hypothesis
Test it
Refine hypothesis
Test again
Reach conclusions
Try to establish cause and effect
Experimental Research
• Most powerful when trying to establish cause
and effect
• Expose participants to different interventions
• In order to compare the result of these
interventions with the outcome
Independent Variable
• The intervention or factor you wish to
measure in order to determine if it will have
an effect on the outcome or disease under
study.
• Examples: medications, diet, exercise,
education, health information system
Dependent Variable
• The outcome, end point or disease under
study
• Examples include: survival time for patients
with cancer, reduction of pressure sores in
patients using specific type of wheelchair,
decrease in the number of adverse events in
health care facilities using a CPOE
Dose-Response Relationship
Experimental research also
tries to determine a doseresponse relationship.
If a new medication has
slowed the progression of
cancer, will a higher dose
slow the progression even
faster?
Or if a specific factor is
removed from the
environment it may also
decrease the progression
of a certain disease.
Use Experimental Research
• Consider the following:
1. Eligibility of appropriate participants
2. Randomization
3. Ethical Issues
Quasi-Experimental Research
• Similar to experimental research but does not
include randomization of participants.
• Independent variable may not be manipulated
by the researcher, and there may be no
control group
• It may be used over time with something
other than individual participants
Example of Quasi-Experimental
Research
• Study the effects of automated coding system
to determine if there is an increase in hospital
reimbursement before and after the system is
implemented
• Study the cost/benefits of using an EHR before
and after its implementation. Cost/benefits of
a paper-based system is compared to the cost
of an EHR in all HIM functions.
Overview of Experimental Research Designs
Study Design
Pretest-posttest control group
method
Solomon four group method
Posttest only control group
method
R = randomization
O = observation
X = intervention
Characteristics
Randomly assigned to
intervention or nonintervention (control) group
Pretests given to both groups
Posttests given to both groups
after intervention
Two intervention groups
Two control groups
Randomization used to assign
to all four groups
Pretest for one pair of
intervention and control groups
Same intervention used in both
groups
Posttest used in all four groups
Randomization used for
assignment into intervention
and control groups
No pretest given
Intervention given to one group
only
Posttest given to both groups
Diagram
R---O---X---O
R---O--- ---O
R---O---X---O
R---O--- ---O
R--- ---X---O
R--- --- ---O
R--- ---X---O
R--- --- ---O
Overview of Quasi-Experimental Study Designs
Study Design
Characteristics
One-shot case study
Simple design
One group
Intervention
Posttest
One group pretestposttest method
One group
Pretest
Intervention
Posttest
Static group
comparison method
Two groups
Intervention
No intervention
Posttest for both groups
Diagram
---X---O
O---X---O
---X---O
--- ---O
Elements
• Randomization
– When study participants are randomly chosen to
be in the experimental, control, or comparison
group using a random method, such as probability
sampling, so that each participant has an equal
chance of being selected for one of the groups.
– Intervention = experimental group
– No intervention = control group
– Different intervention = comparison group
Example Randomization
• Researcher may be interested in determining whether
individuals retain more if they do higher levels of exercise
before they learn how to use the PHR.
• Three different groups of participants will be established.
• First group will run for thirty minutes before sitting down in
front of the computer to learn to use their PHR,
• Second group will walk for thirty minutes before learning to
use the PHR,
• Third group will not do any type of exercise before learning
to use the PHR.
• Therefore, the first group is called the experimental group,
the second is called the comparison group and the third is
called the control group and randomization will be used.
Example Randomization cont’d
• Develop a list of all the study participants and number
them
• Pick out each number and allocate to a particular group.
For example, the first number drawn will go into the
experimental group, the second number to the control
group and the third number to the comparison group and
so forth until all the numbers are drawn and participants
allocated.
• The goal is to have the experimental, control, or
comparison groups as similar as possible except for the
intervention under study.
• Randomization techniques can be performed using
statistical software programs
Comparison Group
• May be unethical to withhold a certain intervention from one
group of participants,
• Some experimental research studies do not contain a control
group but instead use two comparison groups.
• The intervention under study is still used but all members of
the study are receiving some type of intervention.
• For example, if researchers are assessing the effect of using
RHIOs to decrease the incidence of hospital-acquired
infections in four nursing facilities in a particular region, then
two of the nursing facilities will use the RHIO based data and
two of the other nursing facilities will need to utilize some
other type of database in order to minimize the unethical
consequences of not providing any type of data.
Cross-over Design
• A cross-over design can also be used to minimize the
unethical effects of not providing certain types of
interventions.
• It includes using one group of participants as both the
experimental group and the control group. A group of
participants start out by being assigned to the
experimental group and receive this intervention for a
certain period of time, such as 6 months or a year.
After they receive the intervention, they cross-over to
receiving no intervention or another comparison
intervention for another 6 months to a year.
Example Cross-over Design
• May be used when studying whether certain
types of telerehabilitation will improve the
outcomes of patients with multiple sclerosis.
• Patients may start out using sensors and body
monitoring and then cross-over to using a PDA or
the traditional in-house therapy monitoring in
order to monitor their functional levels after
treatment.
• The same group is used as the control (or
comparison group) and the experimental group.
Observation
• Pretest—observing the experimental and
control or comparison groups before the
intervention
• Posttest—observing the experimental and
control or comparison groups after the
intervention
Examples Observation
•
•
•
•
Blood pressure taken before and after the administration of medication, diet, or
exercise
Questionnaire given to determine levels of depression before and after a medication
intervention.
Observing a group of individuals before the administration of a policy or procedure
change and then observing them again after the change has been in place for one
month.
Other types of observations are not administered before or after the intervention but
during the middle of the particular study. These observations are called midtests.
•
Other observations may be conducted several months or years after the intervention
ends to determine its long term impact.
•
Other observations may be conducted throughout the study period as a new policy or
law is implemented. These are called time-series tests. For example, a researcher
might use the time-series test to examine the number of breaches of confidentiality
after the implementation of HIPAA. These rates could be compared to rates before
HIPAA was implemented.
Control Group
• Use of the control group allows the researcher
to determine if the effect seen is really due to
the intervention and not other extraneous
factors or confounding variables.
• In clinical trials when medication is being
tested as the intervention, the control group is
given a placebo so that they are as similar as
possible to the intervention group but not
receiving the medication under study.
Treatment
• Treatments or interventions are also the independent variable.
• Use of experimental medications, changes in an individuals’ behavior such
as smoking or alcohol cessation, or changes in a particular assistive device,
technology, software or system.
• Should be administered in the same way for all participants in the
experimental group.
• For example, if physical therapists are going to be educated and trained
on-line in using a new rehabilitation EHR system, the level (hours of
training), quality (content of the on-line education and training), and
hands-on application (amount of time using the EHR system) should be
the same for all physical therapists in the experimental group.
• The control group may consist of those physical therapists that will receive
the traditional in-class education and training.
• The hypothesis is that those physical therapists trained on-line or with
distance education will be the same or better than those trained using the
in-class method.
Experimental Studies
Pretest/Posttest Control Group Method
• Similar to the randomized controlled trial (RCT) or clinical trial
• Pretest-posttest control group method provides an
intervention that may include a specific program or system
change than a medication or treatment.
• Participants are randomly assigned to either the intervention
(experimental) or a non-intervention (control) group.
• Control/Comparison group may receive a different
intervention other than the one under study.
• Pretests are given to both groups at the same time to assess
their similarities and differences.
• Posttests are given to both groups to determine the effect of
the intervention.
Example in Health Informatics
• Shegog et al, 2001---Assessing the impact of a
computer-assisted instruction (CAI) program on
factors related to asthma self-management
behavior.
• Baseline data collected on asthma selfmanagement skills
• Children in experimental group used CAI program
• One week later post-test data (assessment of
asthma self-management skills and attitudes
toward CAI collected
Solomon Four-Group Method
• Two experimental groups which both receive the
intervention
• One group receives a pretest and posttest while the
other experimental group receives a posttest only.
• Two control groups are also used in this design
• One control group receives a pre and posttest while the
other group receives the posttest only.
• All participants are randomly assigned to the groups.
• This method controls for pretest exposure but also
requires more time, effort and cost due to the
additional groups.
Example of Solomon Four Group
• Adaptation of the Solomon four-group design was used
by researchers evaluating the effectiveness of a multimedia tutorial in the preparation of dental students to
recognize and respond to domestic violence (Danley et
al, 2004).
• First experimental group of dental students was
randomly assigned to take the pretest, the tutorial
(intervention), and then a posttest.
• The second experimental group first took the tutorial
and then the posttest.
• The third group (control group) took the pretest and
then the posttest.
Posttest-Only Control Group Method
• Participants are randomly assigned to an
experimental group or a control group and
posttests are the only means of observation.
• No pretests are used.
• This is done to reduce the effect of familiarity
with exposure to a pretest.
• Not using a pretest eliminates the ability to
assess an improvement in scores from before
the intervention to after the intervention.
Example of Posttest-Only Control
Group Method
• The experimental posttest only control group method
was used by researchers assessing the effect of
community nursing support on clients with
schizophrenia (Beebe, 2001).
• 24 participants randomly assigned to control group
(routine follow-up care and informational telephone
contact at 6 and 12 weeks) and
• Experimental group(weekly telephone intervention
plus routine follow-up care for 3 months).
• All were followed for 3 months after hospital discharge
to determine the length of survival as well as frequency
and length of stay for re-hospitalizations.
Quasi-Experimental Studies
One-Shot Case Study
• This study is a simple design in which an
intervention is provided to one group which is
followed forward in time after intervention to
assess the outcome (posttest).
• No randomization, no control group, and no
pretest is included
• No baseline measurement to provide a
comparison to the intervention outcome.
Example of One-Shot Case Study
• Researchers conducted a quasi-experimental one shot case
study to determine if an automated two-way messaging
system will help HIV-positive patients comply with complex
medication treatments (Dunbar et al, 2003).
• 19 HIV-positive patients enrolled and received two-way
pagers that included reminders to take all medication doses
and follow any dietary requirements.
• No control group
• Outcome measures consisted of the number of times
participants reported missing one or more medication
doses, medication side effects, and participant’s
satisfaction level in using the messaging system.
One-Group Pretest-Posttest Method
• Similar to one-shot case study except that the
pretest is used before the intervention.
• No control group and no randomization.
• Used when it is unethical or inappropriate to
withhold the intervention from a group of
participants.
Example of One-Group PretestPosttest Method
• Researchers assessed the timeliness and access to
healthcare services using telemedicine in individuals aged
18 and younger in state correctional facilities (Fox, et al,
2007).
• Data were collected one year before implementation of the
telemedicine program and two years after implementation.
• The telemedicine intervention consisted primarily of
remote delivery of behavioral health care services.
• Timeliness of care and use of healthcare services before
and after telemedicine implementation was examined.
• The data was collected primarily from medical records and
other claims and information assessment logs.
Static Group Comparison
•
•
•
•
Two groups are examined;
One with the intervention
One without the intervention
Posttest is given to assess the result of the
intervention.
• There are no pretests and no randomization but a
control group is used.
Example Static Group Comparison
• Researchers assessed the use of alcohol in patients
after a traumatic brain injury (TBI) based on patients’
and relatives’ reports (Sander et al, 1997).
• This study examined the validity of patients’ reports by
comparing them to relatives’ descriptions of postinjury alcohol use.
• In this design, researchers use the brain injury as the
intervention and then assess via a post-injury
questionnaire whether drinking habits as perceived by
the patient with the TBI and the close relative are
similar or different.
Internal and External Validity
• Internally validity demonstrates that the
dependent variable (outcome measure) is only
caused by the independent variable
(intervention) rather than other confounding
variables.
• External validity is concerned with being able
to generalize the results to other populations
(Campbell and Stanley, 1963).
Factors Affecting Internal Validity
History
• History or the events happening in the course of
the experiment that could impact the results.
• Researcher collects level of functioning data on
hip replacement patients before and after the use
of a new physical therapy device. During the
time that this device is being used, the developer
becomes ill and unable to fully train all physical
therapists in its proper use. Therefore, the study
may be affected by inadequate time in training
rather than the device itself.
Factors Affecting Internal Validity
Maturation
• Maturation and refers to the natural changes
of research subjects over time due to the
length of time that they are in the study.
• For example, older individuals may become
very fatigued after completing a training
session on using a computer to manage their
finances. Their fatigue could then affect their
responses on the posttest.
Factors Affecting Internal Validity
Testing
• Testing is the effect created once exposed to questions that may be
on the posttest.
• For example, participants of a study that is assessing whether a
course module on the use of privacy and security within the
electronic health record (EHR) improves their knowledge content of
this subject, use a pretest and posttest to assess whether there is
improvement due to the course module.
• However, since the students are already exposed to the pretest and
are able to think of some of the test questions, they may change
their answers on the posttest and do better by learning from the
pretest.
• Therefore, the use of the pretest is what may be causing the
improvement in test scores more so than the course module on
privacy and security of the EHR.
Factors Affecting Internal Validity
Instrumentation
• Instrumentation--Changes in instruments,
interviewers, or observers may all cause
changes in the results.
• For example, interviewers may probe for
answers more from one individual they are
interviewing more so than others, if training is
not performed consistently across all
interviewers.
Factors Affecting Internal Validity
Statistical Regression
• Statistical regression or regression toward the
mean is when extreme scores of measurement
tend to move toward the mean because they
have extreme scores, not because of the
intervention under study.
• For example, coders who performed poorly on
the ICD-10-CM coding exam are selected to
receive training. The mean of their posttest
scores will be higher than their pretest scores
because of statistical regression not necessarily
because of the ICD-10-CM training session.
Factors Affecting Internal Validity
Selection
• Selection is when there are systematic
differences in the selection and composition
of subjects in the experimental and control
groups based on knowledge or ability.
• For example, one group of subjects who have
viewed an instructional video on how to give
themselves insulin injections is compared to
another group which has not watched this
video. No randomization is used.
Factors Affecting Internal Validity
Attrition
• Attrition is the withdrawal of subjects from the study.
Those individuals who leave a study can be very different
than those who remain in the study and the characteristics
of these individuals can affect the results.
• For example, a study which focuses on trying to reduce the
number of incomplete medical records due to incomplete
nursing documentation have 15 nurses leave the
experimental group and 2 nurses leave the control group.
The 15 nurses who leave the group may be very different
than those who remain in the experimental group.
• Also, the difference in the numbers of nurses who leave
each group may be a problem.
Factors Affecting Internal Validity
Interaction
• An interaction of factors or a combination of
the factors discussed above may also lead to
bias in the final results.
• Therefore, the researcher needs to be aware
of the effect of a combination of some of the
factors discussed above and their impact on
internal validity
(Shadish and Cook, 1998, Key, 1997, Shi, 1997).
Factors that Affect External Validity
• Testing
• Selection bias
– Participants are chosen who are frequently under
medical care
– Volunteers
– Participants who receive compensation
• All may be different than the general population
Control for Internal and External
Validity
• Randomization—most powerful to control for
selection, regression to the mean, interaction
of factors, improves external validity because
subjects are not pre-selected but uses random
assignment
• Use of control or comparison groups---help
control for effects of history, maturation,
instrumentation, interaction of factors
(Key, 1997, Shi, 1997)
Poor Experimental Procedures
• Control group exposed to part of the
intervention
• Multiple treatment interference
• Length of time of treatment intervention
• Loss of participants
Summary
• Experimental study designs are one of the most powerful designs to
use when trying to prove cause and effect.
• Quasi-experimental study designs are also very effective but tend to
have many more problems with external validity since most do not
include randomization of subjects
• Researchers in health informatics choose to use the quasiexperimental design for many reasons such as ethical
considerations, the difficulty in randomization of subjects and small
sample size (Harris et al, 2006).
• Several examples of experimental and quasi-experimental studies
and the methodology used in the health informatics and healthcare
setting demonstrate that this study design is a viable option for
health informatics research.
Epidemiological Research
Chapter 6
Introduction
• Epidemiology examines patterns of disease
occurrence in human populations, and the factors
that influence these patterns in relation to time,
place, and persons.
• Essential tool when developing specific
research methodologies in health informatics.
• This chapter provides examples of
epidemiological principles to study disease
and health informatics.
Types of Epidemiology
• Epidemics -- what caused them and how they
could be controlled and prevented.
• Expanded rapidly beyond the study of
infectious diseases into the study of all types
of illnesses.
• Cancer epidemiology; pharmacoepidemiology; environmental epidemiology,
nutritional epidemiology; chronic disease
epidemiology, health services epidemiology,
Epidemiology and Health Informatics
• Epidemiological principles can be used to
study any type of behavior, outcome,
occurrence, community, or healthcare system.
The key is to know which epidemiological
study design to use to inspect a particular
problem.
• Epidemiological principles and study designs
are used to examine many of the health
informatics systems and structures that
sustain the healthcare system today.
Example
Researchers (Bell et al. 2003) used a crosssectional study to determine whether
physician offices located in high-minority and
low-income neighborhoods in southern
California have different levels of access to
information technology than offices located in
lower-minority and higher-income areas.
Example cont’d
•Use epidemiological
principles similar to those
that Snow developed.
•Researched physician
offices in targeted
geographic areas and
neighborhoods to determine
the use of different types of
health information
technology.
•Even though they did not
establish the cause of any
particular disease, they
determined whether or not
socioeconomic
demographics play a part in
the use of information
technology.
Infectious Disease Model
Infectious Disease
Host
Age, gender, race religion,
Marital status, ethnicity, genomics,
Social behaviors, anatomy & physiology
Prior illness or disease
Physical Environment
Nutritional, chemical,
Physical, infectious
Agent
Tornado, flood, hurricane, war
Occupational Environment
Environment
Gordis 2004, 16; Lilienfeld 1994, 37-38).
Chronic Disease Model
Component I
Component IV
Component II
Component III
Chronic disease model:
Example of lung disease
Infant
respiratory
infection
Smoking or
Air Pollution
Family History
or Genetics
Poor Nutrition
Using the Epidemiological Models
of Causation in Health Informatics
Epidemiological model: Health informatics example—
Computer-assisted coding (CAC)
Host Experience
Training
Understanding of CAC system
Computer problems
Coding errors in system
User friendly
Encoder issues
Agent
Documentation in EHR incomplete
Structured text
Free text
Artificial Intelligence
Environment
Chronic disease model:
Example of reluctance to use PHR
Unfamiliar
with
computers
and PHR
systems
Privacy and
Security Issues
Inaccessible
systems
Additional Time to
Develop and Use
Epidemiological Study Designs
• Descriptive Study
– Cross-sectional or prevalence study
• Analytic Studies
– Retrospective (Case-Control) Study
– Prospective Study
• Experimental Study
– Clinical and community trial
Progression of Epidemiological Study
Designs
Descriptive Study
Design:
Analytic Study Design:
Retrospective
Case-Control
Prospective Study
Historical-Prospective
Study
CrossSectional
Prevalence
Experimental Study:
Clinical
Trial
Community Trial
Components of the cross-sectional or
prevalence study
• Describes health characteristic at one point or
period in time
• Generates hypotheses
• Determines whether the disease or health
characteristic exists now
• Generates new ideas
• Performed when very little is known about a
topic
• Excellent design when studying new concepts
in health informatics
• Leads to analytic studies
Prevalence Rate
Below is an example of how a prevalence rate is
determined:
Number of U.S. ambulatory healthcare facilities that use digital radiology systems
Number of ambulatory care facilities in the US
X N
where N = 1000 if expressing the rate per 1,000 facilities, 10,000 if expressing
the rate per 10,000 facilities, and so forth.
Sensitivity and Specificity
• Sensitivity and specificity rates can be used in
prevalence studies when assessing correct
measurement or correct labeling.
• True Positives (TP): Correctly categorize true
cases as cases (cases are individuals with the
disease or outcome) = VALID labeling
• False Negatives (FN): Incorrectly label true
cases as non-cases (non-cases are those
individuals without the disease or outcome =
INVALID labeling
Sensitivity and Specificity
• True Negatives (TN): Correctly label non-cases
as non-cases = VALID labeling
• False Positives (FP): Incorrectly label non-cases
as cases = INVALID labeling
• Sensitivity = Percentage of all true cases
correctly identified where TP/(TP+FN)
• Specificity = Percentage of all true non-cases
correctly identified where TN/(TN+FP)
(Lilienfeld and Stolley 1994)
Example of Prevalence Study
The American Hospital Association (AHA)
(2007) conducted a prevalence study by
surveying AHA member hospitals to
determine their use of health information
technology. Survey instruments were sent to
hospital chief executive officers (CEOs) from
all types of hospitals and from different
geographic areas across the country.
Analytic Study Designs: Case-Control
(Retrospective)
Step 1
Determine the hypothesis and
decide whether to use prevalence
(existing cases of disease) or
incidence cases (new cases of
disease)
Step 2
If prevalence cases, seek out
cases from the state or hospitalbased cancer registry. If
incidence cases, have health care
facilities provide new cases as
they are treated.
Step 3
Decide who will be part of the
study by using inclusion criteria
such as ICD-9-CM codes, laboratory
reports, radiology reports,
medical records, and so forth,
which all validate the disease
under study.
Step 4
Randomly select the cases by
obtaining a list of possible cases
(either from the state or
hospital-based cancer registry or
from a list of ICD-9-CM codes and
so forth) and use a systematic
sample by choosing every 5th case.
Analytic Study Designs: Case-Control
(Retrospective) cont’d
Step 5
Choose controls from siblings or friends, who are
of similar age, gender, socioeconomic status, or
from the same hospital. Controls should be similar
to the cases for all characteristics except the
disease under study. For example, if studying
melanoma, choose controls from the same hospitalaffiliated cancer registry as the case but who has
another type of cancer such as colon cancer or lung
cancer. Select these controls from a list of
cancer cases identified by their ICD-9-CM code and
validate the diagnosis through pathology reports
and medical records. Also, choose controls from
this list that are similar in age by at least five
years.
Step 6
Decide whether matching of the cases and controls
will be used on certain variables. Matching on
variables such as age, gender, race and so forth
should only be used when the researcher is certain
that there is a relationship between that variable
and the dependent variable. For example, age is
always related to cancer because as we age, our
chance of developing cancer increases. Therefore,
age becomes what is called a confounding variable
because it may be the underlying factor that is
leading to the development of the cancer instead of
the specific risk factor that one is trying to
prove is related. Therefore, when studying cancers,
matching should be done for age.
Analytic Study Designs: Case-Control
(Retrospective) cont’d
Step 7
Design the instrument used to
collect the exposure or risk
factor data. Collect it through
phone or in-person interviews,
self-report questionnaires,
abstracts from existing sources
such as the EHR, cancer registry,
birth certificates, death
certificates, financial records
and so forth.
Step 8
Analyze the data to include the
appropriate statistics.
Step 9
Summarize the results and
determine if they support or
refute the hypothesis
Step 10
Publish the results
Example—Odds Ratio
The odds ratio for this example is:
AD
=
(200) x (100) =
BC
(500) x (10)
20,000 =
5,000
4
The value of 4 means that those individuals that use tanning lamps are 4
times more likely to develop melanoma than those individuals who do not
use tanning lamps.
If the odds ratio for this particular example equaled 1, then it means that
the risk for melanoma is actually equal for the cases and controls and that
use of tanning lamps is not a risk factor for melanoma. If an odds ratio is
less than 1.0, this means that the factor (the use of tanning lamps)
actually decreases the risk of disease and provides a protective effect.
Example: Case-Control Study in Health
Informatics
Hippisley-Cox et al. (2005) used the case-control design to examine the
relationship between myocardial infarction (MI), and use of non-steroidal
anti-inflammatory drugs (NSAID). The authors of this study used a
research database called QRESEARCH to examine this relationship.
Cases were patients aged 25 to 100 identified as having an acute
myocardial infarction (MI) for the first time recorded from Read Codes
(similar to SNOMED codes) during a four-year study period.
Controls were those individuals with a diagnosis of coronary heart disease,
but without an MI, matched to each case by age, year, gender, and
physician practice.
Odds ratios were computed.
Confounding variables also were collected and controlled for, and include
smoking and comorbitities such as diabetes, hypertension, coronary heart
disease, osteoarthritis, rheumatoid arthritis, and obesity.
Cohort (Prospective) Study Design
• This study design has two groups of study
participants:
– One with the exposure (independent variable)
– One without the exposure (dependent variable).
• Both groups are then followed forward in time
to determine if and when they develop the
disease or outcome variable under study.
Calculating the Relative Risk
• The calculation for the relative risk is:
Incidence rate of the exposed group
Incidence rate of the unexposed group
• The incidence rate is:
Number of new cases of a disease over a period of time
Population at risk
where N = 1000 if expressing the rate per 1,000 people, or N = 100,000 if
expressing the rate per 100,000 people and so forth.
Population at risk refers to those free of the disease at the start of the study.
XN
Calculating the Relative Risk cont’d
• The calculation for the relative risk is:
• Incidence Rate of Exposed = A
•
(A+B)
• Incidence Rate of Unexposed =
C
•
(C+D)
• Relative Risk: [A / (A + B )]
[C / (C + D)]
Calculating the Relative Risk cont’d
•
•
•
•
•
•
The relative risk for this example is:
Incidence Rate of Exposed = 200
275
Incidence Rate of Unexposed = 25
325
Relative Risk: [0.727]
[0.077] = 9.4
•
In this hypothetical example, the relative risk of 9.4 is very high for the association between
use of video games and migraine headaches, and those children who play video games are
almost nine times more likely to develop migraine headaches than those who do not play
video games.
Prospective Study Example in Health
Informatics
Baxt et al. (1996) conducted a prospective study that
compared the accuracy of physicians diagnosing
patients with acute myocardial infarction (MI) to an
artificial neural network.
Compared data collected by physicians when
evaluating 1,070 patients who entered the
emergency department of a teaching hospital in
California with anterior chest pain to the neural
network diagnosis of the same patients.
Patients were o followed over time by review of their
medical records in the outpatient department or via
telephone to determine their final diagnoses, which
were validated by serum creatine kinase levels and
EKG evidence.
Experimental Study Designs in
Epidemiology
Experimental research studies expose
participants to different interventions
(independent variables) to compare the result
of these interventions with the outcome
(dependent variables).
Two examples of experimental research
studies in epidemiology include the clinical
and community trial.
Clinical Trials
Clinical trials are designed to help healthcare
professionals test new approaches to the
diagnosis, treatment, or prevention of
different diseases.
Patients who are at high risk for developing
these diseases are often the ones who
participate in the clinical trial.
The clinical trial is designed to test new
medications (most common) and surgical
procedures, as well as new treatments or
combinations of treatments to prevent
disease.
Community trials
Very similar to clinical trials but take place in a particular
community and have less control over the intervention than
one would have with the clinical trial.
The community trial’s goal is to produce changes in a specific
population within a community, organization, or association.
Participation includes all members of the community and the
intervention tends to be provided throughout the population
(Friis et al. 2004, 322-323; UPMC 2008).
Clinical and Community Trial Protocol
Rationale and background
Specific aims
Randomization
Blinding or masking
Types and duration of treatment
Number of subjects
Criteria for including and excluding participants
Outline of treatment procedures
Clinical and Community Trial Protocol
Procedures for observing and recording side
effects
Informed consent
Analysis of data
Dissemination of results
Types of Clinical Trials
Treatment trials test experimental
treatments, new combinations of medicines,
different types of surgery, radiation or
chemotherapy
Prevention trials aim to prevent disease in a
person who has never had the disease or to
prevent it from advancing or reoccurring.
Diagnostic trials are conducted to find better
tests, procedures, or screenings to detect a
disease or condition.
Types of Clinical Trials cont’d
Screening trials examine the best method to
detect diseases or health conditions.
Quality of life trials explore methods used to
improve comfort and the quality of life for
individuals with a chronic disease
Clinicaltrials.gov 2007
Phases of Clinical Trials
Phase I, II, III, or IV based on the size of the population and
the intervention being tested. The FDA provides guidelines for
the different types of clinical trials.
Phase I clinical trials usually test a new drug or treatment in a
small group of people (20-80)
Phase II clinical trials study the intervention in a larger group
of people (100-300)
Phase III the study drug or treatment is given to even larger
groups of people (1,000-3,000)
Phase IV clinical trials include studies that collect additional
information after the drug has been marketed, such as the
drug’s risks, benefits, and optimal use.
Clinical Trials in Health Informatics
Shea (1996) examined several different experimental studies comparing
computer-based clinical reminder systems to manual reminder card-type
systems in ambulatory preventive care settings.
They found 16 different randomized controlled trials from 1975-1994 in
which computer-based clinical reminders were used for several different
preventive services such as hypertension follow-up, influenza vaccine,
pneumococcal vaccine, mammography, fecal occult blood test, pap smear,
tetanus vaccine, dental screening, smoking assessment, dietary
assessment, and so forth.
They found that computer-generated reminders when compared to
manual reminders increased preventive practices by 77 percent.
The use of the experimental study to examine the effectiveness of the
computer-based clinical reminder system provided thorough evaluation of
this system.
Rules of Evidence for Causality
• Strength of association: The strength of the association is
measured by the relative risk.(RR) A strong RR is important,
and those >2 are effective to show causality. However,
repeated findings of weak RRs may be of equal importance if
it is found in studies with reliable methodology.
• Consistency of the observed association: Confirmation of
results in many different types of epidemiological studies in
different populations and different settings. This can be seen
in the study by Shea (1996) in which upon review of 16 RCTs,
found that computer-generated reminder systems improved
preventive practices over time.
Rules of Evidence for Causality
• Specificity: A one-to-one relationship between an
independent variable and a dependent variable, or between
the exposure and the disease is necessary to add weight to
causality. However, because some exposures may lead to
many different adverse outcomes, if specificity is not found
this does not mean an association is not causal.
• Temporality: The independent variable must precede the
dependent variable, not follow it. For example, in order to
state that decision support systems decrease medical errors,
the use of the decision support system must precede the
development of the medical error. Sometimes this is not easy
to determine. A prospective study design can help support
this rule.
Rules of Evidence for Causality
• Dose-response relationship: As the dose of the independent
variable is increased, it strengthens the relationship with the
dependent variable.
In epidemiology, this can be demonstrated for smoking, in
which dose and duration increase risk of disease. In health
informatics, if clinical reminder systems for colonoscopy
reduce the likelihood of developing colon cancer, increasing
the use of the clinical reminder systems for other types of
cancer screening can be assumed to also reduce the
development of cancer.
Rules of Evidence for Causality
• Biological plausibility: The relationship must
make sense in relation to what is known about
it in the sciences, animal experiments, and so
forth.
• Experimental evidence: A well-conducted RCT
may confirm the causal relationship between
an independent variable and a dependent
variable.
Rules of Evidence for Causality
• Coherence: Association should be in
accordance with other factors known about
the disease.
• Analogy: If similar associations have
demonstrated causality, then the more likely
this association is probably causal.
Summary
• Epidemiology and its principles can be used
effectively when studying health informatics.
• Researchers can use infectious disease or chronic
disease models of causation to do this.
• Many different types of epidemiological study
designs can also be used to examine health
informatics.
Summary cont’d
• These include the descriptive (prevalence or
cross-sectional), case-control (retrospective),
prospective, and the experimental (clinical
and community trial) study designs.
• Most epidemiologists conduct research by
beginning with the cross-sectional or
prevalence study, and then move forward to
the case-control, prospective, and
experimental study designs.
Informatics Evaluation and
Outcomes:
Research
Related to Core HIM Functions
Chapter 7
Objectives
• Provide a rationale for conducting evaluation
research
• Show the relationship between core health
information management functions and research
• Outline important research methods pertaining to
evaluation research
• Explain terms related to evaluation research and
theory
Introduction: What Is Evaluation
Research?
• Evaluation research consists of a set of methods to explore
the impact of new or existing processes
• Two categories of evaluation are formative and summative
– Formative evaluations are conducted during the course of
development or implementation of a new program or method to
provide iterative feedback on the process.
– Summative evaluations are conducted after implementation and are
designed to examine areas such as the effects or outcomes of a
program.
Formative Evaluation
• Needs assessment
– the process of determining, analyzing, and prioritizing
needs, and in turn, identifying and implementing solution
strategies to resolve high- priority needs
• Implementation evaluation
– Monitoring how well the planned events are actually
occurring
– Whether the implementation is meeting the expected
timeframes.
• Process evaluation
– measures the effectiveness of the program.
Examples of Formative Evaluation
• Improvement in delivery methods with regard to
technology used
• The quality of implementation of a new process or
technology
• Information about the organizational placement of a
given process
• The type of personnel involved in a program
• Other important factors such as the procedures,
source and type of inputs
Summative Evaluation
•
•
•
•
Impact evaluation
Outcome evaluation
Cost-effectiveness analysis
Cost-benefit analysis
Summative Evaluation
• Impact evaluation:
– to assess the intended or
unintended net effects
of the program or
technology
• Outcome evaluation:
– to determine if the
program or technology
has caused
demonstrable effects as
defined in the project
goals
Summative Evaluation (cont.)
• Cost-effectiveness analysis and Cost-benefit
analysis
– Determination of the financial impact of a given
program
• Cost-effectiveness evaluates the degree to which
quality of life is improved as a result of investment in a
program
• Cost-benefit compares a list of financial benefits with a
list of costs so that an informed decision can be made
about beginning or continuing a program can be made
What Are Areas of Research?
• Formulating models
• Developing innovative computer-based systems
• Installing innovative systems and making them work
reliably in environments
• Studying effects
Summative Evaluation of
Programs or Projects
• Conducted after implementation
• Designed to examine areas such as:
• The effects or outcomes of a program
• Determine whether the program had the intended
effect it was designed to have
• Determine the overall impact of a project or program
• Evaluate the costs, revenues, and benefits
Needs Assessment
• Research by Garvin and Watzlaf (2004)
established a need for further preparation of
coders for future professional competencies:
– Evaluated the degree of alignment of current HIM skills
with the projected coding competencies
– Evaluated the readiness of the coding workforce in
comparison to future work requirements
Statistics Review
• A review of the statistics used in the research
studies follows:
– Probability
– Odds
– Sensitivity
– Specificity
– Positive Predictive Value
– Negative Predictive Value
Odds versus Probabilities
• Probability of an event is the fraction of the total
possibilities in which the event is expected to occur.
– Probability of rolling a ‘6’ on one roll of a fair die
• There is one ‘6’ on a die and there are 6 total
possibilities
• Therefore the probability of rolling a ‘6’ on one roll of
the die = 1/6
• Probabilities can ONLY vary between 0 and 1
Odds versus Probabilities (cont.)
• Odds is a ratio of the number of times an
event may occur versus not occur.
– Odds of rolling a ‘6’ on one roll of a fair die –
• There is one ‘6’ on a die and there are 5 “wrong”
possibilities
• Therefore the odds of rolling a ‘6’ on one role of on die
= 1/5
• Odds can vary between 0 and infinity!
Relationship Between Probability and
Odds
• Probability = odds/(odds + 1)
• Odds = probability / (1 – probability)
• For the dice example
– Probability = 1/6 (1 correct answer out of 6 possibilities) =
0.167
– Odds = 1/5 (1 correct answer, 5 wrong answers) = 0.2
– Based on the above formulas
• Probability = 0.2/(0.2 + 1) = 0.167
• Odds = 0.167(1-0.167) = 0.2
The Good and Bad of Probabilities
• The good
– The probability of two independent events occurring is
simply the product of the two component probabilities
• Probability of rolling a 6 on the first try and again on
the second try is (1/6)x(1/6) = 1/36
• The bad
– If the probability of an event is 20% and the likelihood
doubles it is plausible (but incorrect!) to conceptualize
the probability doubling to 40%
– If the probability of an event is 90% and the likelihood
doubles, it is implausible that the new probability
would be twice as much (180%)
Odds Behave Differently
• Unlike with probabilities, you CAN always
arithmetically multiply odds by a factor.
– If an event has odds of 2:1, the odds can double to
4:1, even though both of these odds are greater
than one
How Does this Relate to Sensitivity
and Specificity?
• Given some underlying truth about the presence
or absence of an outcome, and the ability of a
test to correctly identify the outcome, there are 4
possibilities
Disease
Present
Disease
Absent
Test
Positive
True Positive
False
Positive
Test
Negative
False
Negative
True
Negative
Total
population
What is Sensitivity?
• Sensitivity is a PROBABILITY of testing
positive given that you ALREADY know the
outcome exists (True positive/(True positive
+ False Negative)
Disease
Present
Disease
Absent
Test
Positive
True Positive
False
Positive
Test
Negative
False
Negative
True
Negative
Total
population
What is Specificity?
• Specificity is a PROBABILITY of testing negative
given that you ALREADY know the outcome
does not exist (True negative/(True negative +
False Positive)
Disease
Present
Disease
Absent
Test
Positive
True Positive
False
Positive
Test
Negative
False
Negative
True
Negative
Total
population
Higher Sensitivity and Specificity is Better, but…
• The definition of the terms require that you are
already CERTAIN of the presence or absence of the
outcome
• In reality you do not know if the outcome is truly
present or absent, that is why you are performing
the test
• The positive and negative predictive values tell you
the probability of having an outcome given a positive
test and the probability of not having an outcome
given a negative test.
What is Positive Predictive Value?
• Positive predictive value (PPV) is a
PROBABILITY of having an outcome positive
given a positive test result (True
positive/(True positive + False Positive)
Disease
Present
Disease
Absent
Test
Positive
True Positive
False
Positive
Test
Negative
False
Negative
True
Negative
Total
population
What is Negative Predictive Value?
• Negative Predictive Value (NPV) is a PROBABILITY
of not having disease given that you have just
tested negative (True Negative/(True negative +
False Negative)
Disease
Present
Disease
Absent
Test
Positive
True Positive
False
Positive
Test
Negative
False
Negative
True
Negative
Total
population
Example of Sensitivity/Specificity
for Coding
Code +
Code -
Expert
Expert
Review + Review 180
10
20
790
200
800
190
810
1000
Sensitivity/Specificity
• If Expert Review is a gold standard for the presence
of disease, then the disease has a prevalence of 20%
(200/1000)
• The coders found that the records of 190 cases had
documentation substantiated as the disease and
these were coded and 810 did not.
• The sensitivity of the coders for picking up true
disease is 180/200= 90%
• The specificity of the coders for not assigning codes
in charts that did not have disease is 790/800=98.8%
PPV/NPV
• While sensitivity and specificity are important
measures of coding accuracy, the true correctness of
the coders conclusions is assessed by the PPV/NPV
calculations.
• Given a disease prevalence of 80%, if a coder
believes a disease is present, that is correct 180/190
times (95% PPV)
• Given the same disease prevalence if a coder
believes a disease is absent, that is correct 790/810
times (97.5% NPV)
,
PPV/NPV
• Note: If the coders were equally as accurate
(same sensitivity and specificity), but the
prevalence of the disease was different the
PPV and NPV will change.
• If the prevalence decreases, the PPV will
decrease and the NPV will increase
• If the prevalence increase, the PPV will
increase and the NPV will decrease.
What is an Odds Ratio?
• An Odds Ratio is a comparison of the odds of
an outcome given a positive test against the
odds of an outcome given a negative test
• Remember that “odds” themselves are really a
ratio of correct vs incorrect possibilities (as
opposed to probabilities that are ratios of
correct possibilities out of the total number of
possibilities
What is an Odds Ratio?
• Therefore the odds of having an outcome given a positive test
are True positives/False positives
• The odds of having an outcome given a negative test are False
negative/True Negative
• There for the odds ratio (OR) is (True positives/False
positives)/(False negative/True Negative) which simplifies to
just the cross product of the 2x2 table.
• The value “OR” is interpreted to mean that a person who tests
positive has an odds of disease that is OR times the odds of
someone who tests negative
Odds Ratios Interpretation
• This can be extended to variables divided into strata
or continuous variables
• A common use of the odds ratio is to compare the
odds of an outcome given the presence of a given
characteristic with the odds of the outcome given an
alternate characteristic such as the odds of death for
different ages groups compared with the oldest age
group.
Odds Ratios Interpretation
• Since the analytic data is usually a sample of a
larger potential population, if the experiment
were repeated with a different sample, the odds
ratio may vary from the original value obtained.
• To account for this uncertainty, a confidence
interval is calculated around the odds ratio to
reflect an expected range over which the odds
ratio is expected to occur on repeated samples of
data from the population..
Odds Ratios Interpretation
• If the upper and lower bounds of the confidence interval
are both greater than, or both less than 1, then the
association is considered statistically significant.
• The confidence interval is lower when the sample is larger
or has less overall variability
• For example if the AOR of death given a treatment is 0.8
with a 95% CI of 0.5 – 1.3, the point estimate for death
would appear protective, but the Confidence Interval that
spans 1 suggests that no conclusion regarding death can
be made.
Adjusted Odds of Outcomes in the
Women’s Health Initiative (WHI)
INTACT UTERUS
Treated with combined estrogen/progesterone
Adjusted Odds Ratio
95% CI
PRIOR HYSTERECTOMY
Treated with estrogen only
Adjusted Odds Ratio
95% CI
BREAST CANCER
Treated vs untreated
1.26
1.00-1.59
0.77
0.59-1.01
COLON CANCER
Treated vs untreated
0.63
0.43-0.92
1.08
0.75-1.55
HIP FRACTURE
Treated vs untreated
0.66
0.45-0.98
0.61
0.41-0.91
2.07
1.49-2.87
1.47
1.04-2.08
DVT
Treated vs untreated
WHI Explained
• The AOR for breast cancer in the intact uterus group for
people treated with combined HRT compared with untreated
individuals is 1.26 which means HRT is associated with a 26%
increase in the rate of breast cancer.
• The left border of the 95% confidence interval (CI) (1.00-1.59)
touches 1 which defines a borderline statistical significance
(p=0.05)
• Statistical significance is defined by the whether or not the CI
crosses 1. If the confidence interval includes 1, the odds ratio
is not statistically significant. If the CI does not include 1, it is
considered statistically significant.
Examples of How to Interpret Odds
Ratios
• Review Figure 7.2 from the chapter
• What does it mean that the odds ratios in the
1st quartile are all less than 1 across all
diseases?
• What is the difference between HQA(a) and
HQA(b)?
Examples of How to Interpret Odds
Ratios (cont.)
• Based on Figure 7.2 from the chapter:
– In comparison with the 4th quartile (4th quartile is not listed because it
is the reference group) the adjusted odds ratios for mortality in the 1st
quartile is less than 1 across all diseases and denotes that better HQA
scores are associated with lower mortality across all conditions.
Furthermore there is a dose response such that the odds ratio
gradually increases in the 2nd and 3rd quartiles. Note: that any odds
ratio below 1 denotes that the event is less likely than the occurrence
of the event in comparison group.
– Hospitals that score well are better than other hospitals and there is a
statistically significant association with lower mortality in important
conditions.
Differences in Odds Ratios
• What is the difference between HQA(a) and
HQA(b)?
– (a) The odds ratios are adjusted for factors such as
patient age, sex, race, and the presence or
absence of each of thirty comorbidities.
– (b) Adjusted as in note a, as well as for teaching
status, bed size, for-profit status, and region of the
country.
Process Assessment
• One extremely significant research study by
O’Malley et al. (2005) examined key areas
related to the use of coded data. In this study,
the process of inpatient coding was examined
via:
–
–
–
–
A review of the literature
Flow charting the process
Interviews
Discussion with coders and the users of the coded data
Process Assessment
• O’Malley et al. also provide a useful guide to
how accuracy measures for coding can be
calculated. The most common statistics are:
– Sensitivity
– Specificity
– Positive predictive value
– Negative predictive value
– The kappa (k) coefficient
Assessing Implementation
• Shah et al. (2005) redesigned drug alerts to
improve clinician acceptance
– A computerized medication alert system was
designed to decrease patient safety events
– However, it was found that the alerts often were
overridden because of poor specificity and alert
overload
Assessing Implementation (cont.)
• The key factor in the redesign was the modification of the
revised design, which used a selective set of drug alerts that
denoted the potential of a high- severity event.
• The system was designed this way in order to minimize
interruptions to clinician workflow.
• The new design was tested with presentation of alerts to
clinicians using a commercial knowledge base modified to
include a subset of only the most clinically relevant
contraindications.
Assessing Implementation (cont.)
• The revised design was used in 31 Bostonarea practices.
• During the use of this modified system, there
were 18,115 drug alerts during the six- month
study period.
• Of these, 5,182 (29 percent) were highseverity, and 67 percent of these alerts were
accepted by clinicians.
Evaluating Outcomes
• Jha et al. (2007) undertook research to evaluate the
outcomes associated with the Hospital Quality
Alliance (HQA) program. The study was designed to
gauge the importance of the HQA indicators and how
an outcome such as mortality relates to the HQA
indicators
• The data were risk- adjusted using the Agency for
Healthcare Research and Quality (AHRQ) Elixhauser
comorbidity scheme (Elixhauser et al. 1998)
Evaluating Outcomes (cont.)
• The researchers evaluated the relationship
between hospital performance as expressed
by the HQA quality indicators and mortality
data for Medicare enrollees admitted for:
– acute myocardial infarction
– congestive heart failure
– pneumonia
Evaluating Outcomes (cont.)
• The researchers found that higher condition
specific performance in the national quality
reporting program was associated with lower
risk- adjusted mortality for each of the three
conditions.
Evaluating Impact
• Smith et al. (2005) studied the impact of
missing clinical information in primary care
practice. Clinicians reported missing clinical
information in 3.6 percent of visits
• Missing information included:
–
–
–
–
–
laboratory results (6.1 percent of all visits)
letters/dictation(5.4 percent)
radiology results (3.8 percent)
history and physical examination (3.7 percent)
Medications(3.2 percent)
Cost-benefit Analysis
• Miller et al. (2005) conducted a study of solo
or small-group primary care practices to
determine the return on investment of
installing an EHR
– The average practice paid for its EHR costs in 2.5 years and profited
after that
– Some practices could not cover costs quickly
– Most providers spent more time at work initially
– Some practices experienced substantial financial risks
Research Methods
• Formative
– Brainstorming
– Focus groups
– Nominal group
techniques
– Delphi methods
– Concept mapping
– Surveys
– Interviews of
stakeholders
• Summative
–
–
–
–
–
–
Surveys
Simulations
Focus groups
Flowcharting
Work analysis
Evaluation of existing
data
Summary
• Evaluation research makes an important contribution to
science and to HIM/informatics functions
• There are two types of evaluation research: formative and
summative
• Formative: Needs assessment, Implementation evaluation, Process
evaluation
• Summative: Impact evaluation, Outcome evaluation, Costeffectiveness analysis, Cost-benefit analysis
• A Wide variety of research methods can be used: Brainstorming,
Focus groups, Nominal group techniques, Delphi methods, Concept
mapping, Surveys, Interviews of stakeholders, Surveys, Simulations,
Focus groups, Flowcharting, Work analysis, Evaluation of existing data