Transcript Final Presentation PowerPoint

The Online Academy
(TOA)
John Baek
Michelle Didier
Dan Feinberg
Lisa Herbert
Danalyn Robinson
Heather White
Overview
The project
– An online high school
– A partnership between GMU and Northern VA
county school systems
Overview
Review of last semester
– Focus on performance analysis, design &
development
Mission Statement
• Develop high quality courses, correlated to the
Virginia SOL and COPLS model
• Support the expert mentors
Overview
• Redirection of focus
• Focus this semester
– Framework for continued development: code library,
site structure, consistent design using style guidelines
and templates
– Field testing
– Evaluation & usability testing
– Complete the Intro module/ Resource guides
Overview
• Our process
– Team meetings
– SME (teachers) meetings
– Project management
– Divide & conquer
Presentation Outline
• Evaluation
– Evaluation needs & goals
– Methods
– Findings & recommendations
• Future of TOA
– Framework for future development
– Project recommendations
Evaluating TOA
Need for Evaluation
Methods of Evaluation
Findings and Recommendations
Need for Evaluation
• Inform and direct current and future development
• Examine the usability of the checklist
– Organization and self-regulation
• Determine effectiveness of navigation
– Locating resources
– Intuitiveness—do the learners know where to go next?
• Review the course modules
–
–
–
–
Clarity of instruction
Engagement and motivation of instruction
Adequacy of the resources, skills, and content for challenge
Pedagogical re-engineering (approach and format)
Evaluation Goals
• Analyze instructional strengths
• Determine the effectiveness of the design
and transfer of learning
• Evaluate the usability and the benefits of
the checklist to the learner
• Locate ease of use problems (navigation)
Methods of Evaluation
•
•
•
•
•
Quality assurance
Field testing
One-to-One evaluation
Expert review
Case study
Meeting the Evaluation Goals
Instructional strengths
Effectiveness of the
design and transfer
•One-to-one evaluation
•Expert review
•Case study
•Quality assurance
•Case study
•Expert review
Checklist Usability
Locate ease of use
problems
•Expert review
•Field test
•Field test
•One-to-one evaluation
Quality Assurance
For each course, at least two
modules were field tested.
•Chemistry
•Mars Matters
•Sappy Solutions
•Earth Science
•Passing Gas…For Effect Only
•Like It…Weather or Not
•Algebra
•Graphing – Slopes
•Geometry
•Right Triangles/ Trigonometry
•Points, Lines, Planes, Segments,
and Angles
•English 11
•The Two Macks: A Harlem
Renaissance Club
•Regionalism, Realism, and
Nationalism
•World History I
•India and China
•African Civilizations
•World History II
•Imperialism
•Between the Wars
Before anything was field tested, the modules
were evaluated for Quality Assurance.
• Forms that don’t work
• Typing mistakes
• JavaScript errors
• Form elements not properly labeled
• Pages that don’t print correctly
• Consistent styles
• Missing page titles or code comments
The result is a much more polished looking product.
Field Testing
Field Testing
•
•
•
•
•
14 Modules
Technical problems
Student response
Self regulation
Synchronous discussion (Digichat)
Digichat
• Encourages peer
interaction
• Encourages
mentor-learner
interaction
• Mentor office hours
Intro Module
Intro Module
• Need for Intro Module
• Elements of most
modules
• Use basic TOA
functions
Mentor Resources
• Overview
• Objectives
• Answer keys
Challenge
• Problem based context for entire module
• Background building knowledge
• Culminating activity
The Checklist
•Self regulation
•Mentor-student
negotiation
•Links to activities
TOA Introductory Module
• Research for the module was gathered
from several sources
– Teachers’ feedback from their field testing
– Results from one-to-one evaluation
– Review of related literature
• Maintain theme of the story introduced in
the Challenge
Working with a Mentor
Working with a Mentor
Activity
Being an Online Learner
Being an Online Learner
Activity
Online Research
Online Research
Online Research
One-To-One Evaluations
A face to face meeting with members of
our target audience where we ask them
to use TOA and give us their opinion on
the site.
One-To-One Evaluations
•
Are the instructions clear?
•
Can the user locate
resources?
•
Can learners navigate?
•
Are the resources, skills,
and content adequate for
completing the challenge?
•Is the instruction engaging
and motivating?
•Will the student know the
references?
•Is the content at the
appropriate level?
•Does the checklist provide
self-regulatory benefits?
One-To-One Evaluation Process
1.
2.
3.
4.
5.
6.
7.
8.
Recruit students.
Prepare evaluation questions.
Design data collection tools.
Write script.
Set up testing area.
Conduct evaluation.
Analyze results.
Write evaluation report.
1. Recruit Students
• We were provided access to two students
at a Washington, DC private school
• “Ralph” – A 16 year old white male
currently enrolled in Chemistry
• “Kelly” – A 16 year old white female who
had taken Chinese History
2.Prepare Evaluation Questions
3. Design Data Collection Tools
• Tape recorded and
transcribed
• Field notes were
taken by interviewer
and observer
• Follow up
questionnaire was
given
4. Write Script
5. Set Up Testing Area
6. Conduct Evaluation
7. Analyze Results
8. Write Evaluation Report
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module
• Summary of students’ impressions:
– Kelly:
• Liked the story presented in the challenge and carried through
the activities
• Liked the PowerPoint activities
• Thought the activities were explained well
• Liked self paced aspects of course
– Ralph:
•
•
•
•
Navigated using pull-down menus
Liked self paced aspects of course
Unsure of being motivated enough to complete activities
Liked the layout and color scheme of the site
Expert Reviews
Expert Review Steps
1.
2.
3.
4.
5.
6.
7.
Recruit experts
Prepare questions
Design data collection tools
Set up testing area
Conduct evaluation
Analyze results
Write evaluation
Dr. Dabbagh’s Interview
• Earth Science Module on Gas
• Open questions based on likes and
dislikes
• How learning theories are applied
• Does this address multiple learning styles?
Earth Science
Passing Gas…
The Checklist
An Activity
Dr. Dabbagh’s Comments
• Thought the content was very situated in real life
scenarios
• Liked the sub activities under the challenge, how
main challenge is umbrella
• Liked the graphics, thought they were
appropriate to content
• Liked the embedded links to additional
information
Dr. Dabbagh’s
Recommendations
• Main module page needs to reflect flow of lessons (ex. If there is an
order to a lesson, then having all the links available makes users
think they can skip around.)
• Students need to have a better understanding of where they are in
the order of things.
• Unclear how the module works, ex. the challenge is actually done
last.
• Needs to closely match the checklist with the verbiage of the
lessons.
• Include the amount of time available to complete module and the
number of hours students are expected to devote to the module.
• Should provide opportunity for reflection.
TOA Recommendations
• We suggest that curriculum writers strive to write clear,
specific instructions for activities.
• The titles of lessons and activities should also be more
consistent and simple.
• Another recommendation is to use consistent wording in
the checklist, lesson, and activities so they match each
other.
• Design opportunity for reflection into The Online
Academy.
The Checklist
– A self-regulation tool
– List activities & due dates
– Negotiated with mentor
The Checklist
Evaluating the Checklist
• Expert review: Dr. Kitsantas &
Dr. Dabbagh
• One-to-one evaluation
The Checklist
The Findings
Areas for improvement:
• Have students set goals by adding columns for
planned, actual and revision dates
• Keep track on progress
• Self-reflect on how they performed
• Provide scaffolding and modeling
• Link all activities
The Checklist
Recommendations
• Support the checklist with a database to track:
• Dates: planned, actual and revision
• Record grades
• Link key activities
• Provide scaffolding for completing activities within
lesson
• Provide opportunity for reflection: by mentor or in module
The Checklist
Suggested
Checklist
Case Study
Background
• Research Methods course project
• Research partner, Kimberly McDuffie
• Quantitative & qualitative methods
requirement
Purpose
• To examine the experiences of students
with learning disabilities with TOA’s design
(not academic achievement)
Research Questions
• How does the students’ relationship with
the mentor impact the online learning
experience?
• Do certain design elements of The Online
Academy accommodate students with a
learning disability?
Literature Review
• Use of computers with students with
learning disabilities (LD)
• Use of hypertext with students with LD
• Multiple design elements in technologybased instruction
• Lack of research in specific study meant
exploratory approach
Methods
• Participants
– 2 high school Geometry students
– Recruited because there was field testing in
inclusive classroom
• Measures
– Survey
– Interview
Survey Results
• Attempt to measure the aspects of the
mentor-student relationship:
Time Spent
Quality of Communication
Amount of
Communication
Use of Feedback
2.88 (3-5/week)
4.20 (Often)
3.13 (3-5/week)
3.00 (Sometimes)
Interview Codes
•
•
•
•
•
•
•
Clarity of instruction
Chunking
Readability
Navigation
Asynchronous communication
Synchronous communication
Organization of their learning
Conclusions
• Overall – LD students would not do it
again, but recommended it for others
• Preferred physically present teacher
• Expressed difficulty communicating their
learning problems through text
• Nothing positive reported about design
elements
Limitations
• Field test was artificial, done in a
classroom
• Self-made survey instrument, not
validated.
• Only two students, no external validity
Geometry Module
Geometry Challenge Page
Geometry Assignment
TOA Evaluation Summary
Evaluation informs design
• QA plan
• Field tests
• One-to-one
• Expert review
• Case study
The Future of TOA
The TOA Code Library
The Print Version
• Printable version of Code Library to keep as a
reference
• Code is color coded to find editable areas
Online Version
• Online Code Library is available on the VHS
website
• User can copy and paste code into their html
document
What’s Left To Do
• Find a Partner
• Login for site and Digichat
• Retroactive changes based on our
evaluation findings
• Mentor resources
Thanks
•
•
•
•
•
•
•
Priscilla Norton
Brenda Bannan-Ritland
Bill Warrick
All of the Virtual High School teachers
Nada Dabbagh and Anastasia Kitsantas
Graduate School of Education
Stafford, Loudon & Frederick school divisions