Transcript PPT
Testing to Evaluate Learning New Faculty Orientation 2007 Dr. Diane Wink Professor College of Nursing ©Diane Wink 2007 Welcome To UCF Why test?? Achievement of objectives of learning measured with evaluation Why test?? Achievement of objectives of learning measured with evaluation Why test?? Achievement of objectives of learning measured with evaluation Class activities must reflect objectives and what will be tested Class Experience and Activities Objectives • All objectives should lead directly to evaluation through the learning experience itself. • Evaluation should be of the objectives. Class Experience and Activities Questions to ask • What is the best evaluation method to determine student mastery and application of this knowledge? Questions to ask - Test Alternatives • Term paper • Class activity – Debate and bib development • Problem set completion • Service Learning activity • Course project – Case Study – Development of tool • Non written evaluation – Performance of the competency being developed For Testing: Questions to ask • How will this test help me know if learning objectives are being met? • Is the test addressing essential information? • Does the test reflect the readings, class activities and/or web discussion? Questions to ask • How many questions will it take to determine if student has met objectives/ has the knowledge? Do a test blueprint! Test Blueprint Written Tests • • • • Multiple Choice True-False Matching Essay and Short answer essay For all types of exams • Have a colleague review your items • Consider the possibility of cheating – – – – – – – Alternative forms of tests Random seating 3X5 key points card New exams No cell phones, PDAs Open book if on-line In person for on-line Multiple Choice Items • The stem should: – Contain a self-contained question or problem – Present as much of the item's content as possible – Be precise - no extraneous information Multiple Choice Items • The stem should: – Be independent of all other stems. If part of a situation, say this clearly Do not make answer dependent on getting a prior item correct – Be a positive rather than a negative statement or question Should not be a reading test! Multiple Choice Items • The answer choices should: – Contain only one best or correct answer – Be grammatically consistent with stem – Be similar in complexity, detail, length and plausibility – Avoid "all of the above" or "none of the above" Multiple Choice Items • The answer choices should: – Be plausible – Be arranged logically Alphabetical, numerical, length – Contain action verbs from Bloom's Taxonomy – Randomly use each alternative position for correct answers (not all A or B or C or D) Multiple Choice Items • The answer choices should: – Be on the same page as the stem – Not have obscure or esoteric language Sample Item- Poor If all of the following heart valves are defective but only one can be repaired, which should it be? a. b. c. d. Aortic Mitral Pulmonic Tricuspid Sample Item Narrowing and tightness of which heart valve is most likely to result in left ventricular enlargement? a. b. c. d. Aortic Mitral Pulmonic Tricuspid True False Items • State the item as simply and clearly as possible • Keep statements equal in length • Write each item so that it is clearly true or false True False Items • Avoid qualifications such as "often" or "usually" • Test important information and facts • Randomize the true and false items Sample T/F Item (Hard to score) 15. For each of the following, choose A if true and B if false. a. The major pump chamber of the heart is the ventricle b. Valves control blood flow into the atria c. The pulmonary artery carries fully oxygenated blood to the lungs Sample T/F Item (Easy to score) For each of the following, choose A if true and B if false. 15. The major pump chamber of the heart is the left ventricle 16. Valves control blood flow into the atria 17. The pulmonary artery carries fully oxygenated blood Sample T/F Item (Hard to score) For each of the following, choose A if true and B if false. 15. The major pump chamber of the heart is the ventricle What is problem with the above item?? Matching Items • Put the basis of relationship for matching in the directions • Put the entire set of matching items on a single page • Arrange items in a systematic order • Number premises on the left and letter the responses on the right a-e Allows computer grading Matching Items • Use a larger or smaller set of responses and allow responses to be used more than once or not at all • Use homogeneous lists that concern the same content • Number each item separately to allow computer grading Sample Matching Item - Poor 31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once. Hoover a. Implemented emancipation proclamation Roosevelt b. Established the CCC Lincoln c. Major war ends Clinton d. Stock market crash f. Was impeached Sample Matching Item - Poor 31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once. Hoover a. Implemented emancipation proclamation Roosevelt b. Established the CCC Lincoln c. Major war ends Clinton d. Stock market crash f. Was impeached All questions (there are actually four) have same item number! Confusion over which Roosevelt, which stock market crash? Sample Matching - Better Match each of the presidents listed in Column A with the historical event listed in Column B. Events in Column B can be used more than once. Column A 31. Hoover 32. 33. 34. F.D. Roosevelt Lincoln Clinton Column B a. Post civil war reconstruction in progress in US b. Establishment of the CCC c. WW II ends d. Stock market crashed starting the Great Depression f. Impeached Exam grading support at UCF University testing service support for multiple choice/true false/matching exams Web CT and Webcourses@UCF • Turn off printing • Make exam end at particular time Disability Services Grading issues What if the item is bad??!! Essay/Short Answer Items What do they test? Higher cognitive levels Application Analysis Synthesis Evaluation Not for simple recall of facts Essay/Short Answer Items • First ask – Is item clearly written? – If extended response needed, is it too broad? – If restricted response needed, would the content be assessed better with an objective item? Sample Item • Poor: List the components of a persuasive letter to a government official. • Better: Write a letter to a government official in support of or opposed to gun control. Sample Item • Poor: What are the themes in War and Peace? • Better: Relate a theme in War and Peace to world events in 2007 Grading essay/short essay items Write out the answer Does your answer match the intent of the item? What problems did you encounter in answering the item? Can the student answer in the time allowed? Grading essay/short answer items • Analytical method • Points for each essential element • Holistic method • Look for essential elements • Evaluate overall quality of response • Little feedback Analytical Grading Method Identify criteria for evaluating the content of the response and assign points for each: – – – – – – * Essential elements required Accuracy of content Logical organization Clarity Depth and complexity of analysis Use of references and specific citations * Usually only used in formal papers Analytical Grading Method Decide on the extent to which writing ability and use of style format will be included in the scoring – – – – – * Spelling Grammar Sentence structure Writing style APA citation and format * Usually only used in formal papers Holistic Grading Method Sample “A” answer: Covers all key points with citations to major resources. Clearly written with analysis of problem clear and persuasive. Minimal to no grammatical or spelling errors. Holistic Grading Method Sample “B” answer: Covers most key points with citations to most major resources. Clearly written with good analysis of problem. Few grammatical or spelling errors. Sample “C” answer: Covers few key points with few references to major resources. Written poorly with limited analysis of problem and a non persuasive argument Multiple grammatical or spelling errors. Holistic Grading Method Sample “D” Answer: Omits most key points with few/no citations to resources. Writing difficult to follow with consistent grammatical and spelling errors Sample “F” answer: Omits all key points, does not answer question Grading essay and short answer items Before scoring Read a random sample of papers Get a sense of how students answered Revise evaluation criteria if needed Review grading criteria with any other graders - practice!!! Do key for commonly made comments (give to students and all graders) 1. 2. 3. 4. Great job. Shows excellent analysis of your main points Great ideas but needed to be developed more Unclear what you mean Needs citation Grading essay and short answer items Ideal grading technique: – Score answers to one question at a time for all students – Read each answer twice First for omissions of essential elements Second reread and score Grading essay and short answer items • Ideal grading technique: – Cover up prior scores to items or record them on a separate answer sheet – Read papers anonymously; have students use a code number Grading essay and short answer items • If fatigued, stop and rest! Grading essay/short answer items • GTA • Help from peers Consider time for grading when designing exams Questions??? Bloom’s Taxonomy Cognitive Knowledge Identification and recall of information Comprehension Organization and selection of facts and ideas Application Use of facts, rules principles Bloom’s Taxonomy Cognitive (continued) Analysis Separation of a whole into component parts Synthesis Combination of ideas to form a new whole Evaluation Development of opinions, judgments, or decisions Bloom’s Taxonomy Affective Domain Feeling tone Emotion Degree of acceptance or rejection Psychomotor Domain Resources Used Oermann, M., & Gaberson, K. (2006). Evaluation and testing in nursing education. New York: Springer Oermann, M. (1999). Developing and scoring essay tests. Nurse Educator, 24(2), 29-32. Tomey, A. (1999). Preparing a testing blueprint from levels of course objectives. Nurse Educator, 24(4), 11-12. Tomey, A. (1999). Selected response test items. Nurse Educator, 24(5), 910. Tomey, A. (1999). Completion testing items. Nurse Educator, 24(6), 6-7.