Transcript Evaluating Research Articles - Florida Institute of Technology

Evaluating Research Articles
Approach With Skepticism
Rebecca L. Fiedler
January 16, 2002
Preview of Article
•
•
•
•
•
Does the abstract intrigue me to read this article?
Will it be useful to me?
Who are the researchers?
Are they credible and unbiased?
What sources did the researchers use?
Major Components
•
•
•
•
•
•
Title
Abstract
Introduction
Literature Review
Research Question
Hypothesis
•
•
•
•
Methodology
Summary of Results
Discussion & Conclusion
References
Qualitative Research
• The research is conducted in
the natural setting
• Is descriptive
• Researcher observes a specific
situation
• Sometimes variables are
manipulated, but not always
• Goal is to gain insight or identify
key variables or ask new
questions for further research
Quantitative Research
• Attempts to quantify key variables and relate
them
• Variables are manipulated in some way
(Called a treatment)
• Results are measured and analyzed
statistically
• Goal is to identify cause-and-effect
relationships
Quantitative Research
Hypothesis testing
Title
• Is it specific?
• Is the nature of the research
clear?
• Does it reflect the content of
the article?
• Are the results accurately
indicated?
• Are the main variables clear?
• Is the population clear?
Abstract
• Was the purpose clear?
• Was the methodology indicated?
• Were the populations and
samples clearly identified?
• Did the the abstract highlight the
findings?
Introduction
• Is the purpose made clear?
• Did the authors explain the
significance of the study?
(Do you agree?)
• Clearly written and wellorganized?
Literature Review
• Does the review establish
significance of the study?
• Does the review address the
problem area?
• Is the review easy to read and
understand?
• Is it balanced?
• Are the reviewed articles
relevant and current?
• Is the review comprehensive?
• How credible are the cited
sources? Most should be
primary sources.
Types of Sources
• General references
• Primary sources
• Secondary sources
Research Question/Hypothesis
• Is it clearly stated?
• Is there a hypothesis?
• How well is it related to the other
components?
• Is it ethical to ask?
Methodology
•
•
•
•
•
•
•
Experimental
Survey
Correlation
Factorial study
Causal-Comparative
Regression analysis
Ethnographic study
Methodology - Research Design
• Do the authors justify the design decisions?
• Did they discuss the limitations?
• Are variables identified?
Dependent & independent.
• Are any external variables identified?
• Are those external variables controlled?
• Was the design appropriate?
Methodology - Samples
• Is the population identified?
• Are the samples representative of that population?
• How were the samples selected and will the techniques
compromise the results?
• Can the information be generalized to the proposed
population?
Methodology - Instruments
• What instruments were used to
collect the data?
• Was the choice of instrument
justified?
• Is evidence of reliability and
validity provided?
• Were any limitations
addressed?
Common Threats to Validity
•
•
•
•
•
•
•
•
•
Subject Characteristics
Mortality
Location
Instrument
Maturation
Regression
Hawthorne Effect
History
Implementation
Methodology - Procedures
• Is the description of procedures robust?
• Are there any threats to validity due to the procedures
discussed?
• Are you able to identify additional threats to validity?
• Are there any ethical issues in the procedures?
Summary of Results
• Are the results reported without any interpretation first?
• Are the results directly tied to the question, hypothesis or
problem?
• Did the author provide enough detail for you to independently
check the results?
• Is there enough description for you to interpret the results in
context?
Discussion & Conclusion
• What are are the findings? Are they clearly stated?
• Are the findings related to the results of the study and the
literature review?
• Any weaknesses or limitations?
• Did the authors make any statements about generalizability?
• Recommendations for future study?
References
• Are most references primary
sources?
• How many citations are
offered?
• Are the cited references recent?
• Based on the given info, can
you find them for your own
review?
Qualitative
Very descriptive
Similarities to Quantitative
•
•
•
•
Title
Abstract
Introduction
Review of Literature
Research Problem
• Did the focus of the research shift?
(That’s not bad)
• Was the shift justified?
• Did the researchers avoid a hypothesis at the start?
• What (if any) hypotheses were formed based on the data?
Samples
• Often purposive. The purpose should be identified.
• Accessibility is sometimes an important consideration. It
should be named if it is/was a factor.
• Is there a detailed description of the sample?
Setting
• Is there a full rich description
of the setting?
Researchers
• Are there biases that may
interfere with the study?
• Was there interaction between
the researcher and the
participants? Or was the
researcher only an observer?
• How were observers trained?
Data Collection
• Did the researchers use more
than one way to observe the
same phenomenon?
• Was there evidence of validity
by triangulation?
• Was there any quantitative
data? Frequency counts are
common.
Procedures
•
•
•
•
•
Should be the strongest section.
Are procedures fully described?
Are they appropriate?
Are there any ethical concerns?
Are there any threats to
validity?
Data Analysis
• Is analysis in descriptive form?
• If so, is the description
supported by the evidence?
• Is there any quantitative data?
• Is data provided for reader to
review?
Results
• What were the reported results?
• Did the researcher form a
hypothesis?
Discussion and Conclusions
• What conclusions did the researchers
reach?
• What implications can be drawn from the
research?
• Are there suggestions for further
research?
• What limitations were mentioned?
Now what?
You might want to….
• Use one of their suggestions for further research for your own
topic
• Replicate a research study
• Cite the article in your own research
• Use the research design in your own work
• Pick up other ideas for your own research
• Read the cited articles
My Bibliography
–Girden, E. (2001). Evaluating Research Articles: From Start to Finish
(2nd edition ed.). Thousand Oaks, London, New Delhi: Sage
Publications.
–Goubil-Gambrell, P. (1992). A Practitioner's Guide to Research
Methods. Technical Communication: Journal of the Society for
Technical Communication, 39(4), pp. 582-591.
–GraphicsLand. (2002). Squares PowerPoint template [template file].
GraphicsLand. Retrieved 1/12/02, 2002, from the World Wide Web:
www.graphicsland.com
–Lunsford MSE CO, T. R., & Lunsford MS MAPT, B. R. (1996). How to
Critically Read a Journal Research Article. Journal of Prosthetics and
Orthotics, 8(1), pp. 24-31.
My Bibliography (continued)
–Spyridakis, J. (1992). Conducting Research in Technical
Communication: The Application of True Experimental Designs.
Technical Communication: Journal of the Society for Technical
Communication, 39(4), pp. 607-624.
–Sullivan, P., & Spilka, R. (1992). Qualitative Research in Technical
Communication: Issues of Value, Identity, and Use. Technical
Communication: Journal of the Society for Technical Communication,
39(4), pp. 592-606.
–Yaw, M. (2001). Notes from Fundamentals of Graduate Research in
Education.
Class Exercise
• Divide into two groups
• Prepare 10-minute presentation (45 minutes allowed)
–Presentation should critique one of the following articles
• Illustrations in User Manuals
• Learning How to Use a Cellular Phone
• Group presentations
Bibliography Management
•
•
•
•
•
File cabinet
Sticky notes
Index card file
Database
Bibliographic software
Free downloads on web
–EndNote
–Reference Manager
–ProCite