Transcript Data-driven Decision-making for Quality Control:

Data-Driven Decision-Making
for Quality Control:
The Power of a
Relational Database
Vicki L. Cohen, Ed.D.
Marlene Rosenbaum, Ed.D.
Joshua Cohen
Fairleigh Dickinson University
Teaneck, NJ 07666
Presented at the Annual AACTE
Conference
New Orleans, February 2008
This session will:



Describe how using a relational database
becomes the driver of a quality control system;
Describe the development and utilization of a
relational database;
Show how data is leveraged to support student
learning, program revision, and outcomes
based assessment.
The School of Education (SOE)
at Fairleigh Dickinson University
Comprised of aprox. 1,000 students
 Multiple programs that apply for state
certification:

5-Year Accelerated QUEST program
 MAT
 LD
 Educational Leadership
 Reading Specialist

The SOE at a Glance






On two campuses (Teaneck and Madison)
Located at 3 Community Colleges throughout
the state of New Jersey
15 Full-time faculty members
Aprox 35 part-time faculty
Place approximately 120 candidates into
student teaching/year
Place a total of approximately 700 candidates
into clinical field experiences/year.
FDU at a Glance




FDU has aprox 12,000 students
Largest private university in State of NJ
SOE is part of University College on Teaneck
campus
QUEST program:



45 candidates at Teaneck
200 Madison
75 CC
SOE Is a Complex Program!
Went for TEAC accreditation
Preparation for TEAC





School of Education (SOE) needed to collect
accurate information on its claims
Started gathering data on programs and
student performance
Recognized need to access, organize and
analyze data in meaningful ways
Developed a relational database
This would become the “driver” of our Quality
Control System
The Quality Control System
(QCS)
Every institution and program has a set of
procedures and policies to ensure quality
in hiring, admissions, curriculum, program
design, and student learning.
 Together, these procedures and
structures function as a Quality Control
System (QCS). (TEAC)

Need for Valid and Reliable
Data
QCS must yield valid and reliable
evidence about the program’s
practices and results, which
influences its policies and decision
making.
What is Valid Evidence?
Are we measuring what we intended to
measure?
 Are we sure that our evidence is pointing
us in the right direction?
 How confident do we feel about the data
we collected?

“Am I measuring what I think I
am measuring?”
What is Reliable Evidence?
Yields results that are accurate and stable
 Collected in a consistent way
 Confident we we are making the right
decision.

SOE TEAC Process
1. Developed a QCS that ensured we are
collecting valid and reliable data on our
claims and cross-cutting themes


Instruments (rubrics, observation forms,
surveys) were validated through external
panel of “experts”
Inter-rater reliability is being established
2. Developed infrastructure and system to
collect the data
SOE TEAC Process
3. Analyzed data: aggregated and disaggregated
4. Determined strengths and weaknesses with
total faculty involvement
5. Analyzed what revisions needed
•
•
•
Programs
Curriculum
Processes and policies
6. Currently making revisions based upon
evidence.
SOE Assessment Philosophy
We use multiple sources of data that are
designed to assess the performance of
teaching candidates as they progress
through our program.
 Collect data in three areas.

1) Throughout the Program
Continuous assessment of teaching candidates
throughout the program from entrance,
midpoint, and exit





Grades
GPA
Praxis scores
Rubrics
Reflections
2) In Clinical Field Experiences
Assessment of pedagogical knowledge
and skills that occurs during clinical field
experiences;
Placement
 Observation forms

3) Completion of Program
Perceptions of teaching candidates and alumni
after they have completed their program, which
is used for program improvement



Alumni Surveys
Exit Surveys
Focus Groups
Traditionally….Data Deficient
Schools of Education have not been
collecting data systematically
 Infrastructure not set up
 Not able to access multiple
sources of information

Traditionally…..Data Dummies
What data do we want to collect?
 How can we manage it?
 What do we do with data?
 How do we organize it?
 Access it?
 Make sense?

Currently….Data-Driven
Systematically collecting data
 Infusing into our faculty culture
 Meeting regularly to assess evidence
 Making decisions
based upon evidence
 Beneficial process
due to TEAC

Data-Driven Decision-Making
Collect
Data
Example of Data Collection:
Praxis Results 2005/2006
Program
FDU SOE
Pass Rate
NJ Pass Rate
Elementary Ed Content Knowledge
99%
84%
English Lang Lit Content Knowledge
100%
71%
Social Studies Content Knowledge
100%
64%
Secondary: Math Content
100%
92%
Middle School Math
100%
65%
General Science Content Knowledge
100%
78%
Spanish Content Knowledge
100%
87%
Me an S core s of Te ach in gC andidate son S ele cted C C IIn dicators Relate dto Pe dagogyas Rated by
S u pervisors (Fal l 2005, Sprin g 2006)
2.2 Creates and implemen ts
lessons that are developmentally
appropriate.
3.2 Effec tively incorporates
multicultural information and
strategies when appropriate into
the lesson; presents issues from a
multicultural perspective.
57
Midpoin t
Me an S tan dard
N
De viation
4.47
.630
51
57
2.82
2.010
51
3.59
1.846
4.1 All essential components of a
well designed plan for coherent
instruction.
57
4.58
.533
51
4.71
.672
4.2 Appropriate instructional
objectives that are clearly stated,
measurable and aligned with the
NJCCCS.
57
4.42
.706
51
4.67
.476
4.3 Effec tive use of a wide range
of resources including technology
to enhance instruction and student
learning.
56
3.86
1.420
51
4.39
.918
4.4 Demonstrates understanding of
curriculum design and
implemen tation, including various
curricular approaches (i.e.
interdisciplinary, integrated,
thematic).
57
3.79
1.278
51
4.12
1.336
In di cator
N
Fin al
Me an S tan dard
De viation
4.75
.483
Wanted: Database Administrator
New Job Description: full-time
professional staff
 Resources
 Support from administrators
 Ensure candidate had appropriate
knowledge and skills to design database

What Is a Database System?

A collection of data organized in tables,
which can be accessed and manipulated,
without having to restructure the tables

Elements of a Database System
• A storage system
• Data structures
• Manipulation tools
Database Advantages




Analyze sophisticated correlations easier because
relationships are established between data sets
Make decisions based on information derived from
data
Streamline business operations
Organize data and eliminates:
• Inconsistent data
• Missing data
• Redundant data
Problem #1: Field Placement
Office
Staff was overwhelmed with managing
Clinical Placements
 In 2007 800+ letters were mailed to 424
schools asking for placements
 Previously, these letters were individually
prepared in MS Word documents

Problem #1: Field Placement
Office (cont’d)

Clinical Placements must be coordinated with
• School
• Student
• Supervisor

Difficult to aggregate data
• What districts have most confirmed/declined rates?
• What trends are we seeing?
• What kind of schools are we sending our candidates to?
Solution #1: Streamlined Field
Placement Office

Developed a data collection system for
Clinical Placements

Data is:
• Entered on 2 campuses
• Used to create personalized communications to
schools, students, and supervisors
• Used to manage Clinical Placements
• Confirmed/Pending/Declined
• Supervisor Assignments
Solution #1: Streamlined Field
Placement Office (cont’d)

Gives us the ability to aggregate data,
look at trends, and make decisions
Confirmed / Declined distribution by district
 Analyze demographics of cooperating school
districts

Solution #1: Infrastructure
We start with the person record from the
university system
Data on the Clinical Placement is entered
Solution #1: Generating letters
Generate standardized reports
for “master lists”
Solution #1: Payroll
Problem #2: What Type of Districts Do
We Place Our Candidates Into?
The District Factor Group (DFG) is a
socioeconomic indictor used for
comparative test reporting of school
districts for New Jersey’s statewide
programs.
Problem #2: What Type of Districts Do
We Place Our Candidates Into? (cont’d)

DFG Factors:







% of adult residents failed to complete high school
% of adult residents who attended college
Occupational status (laborers, service workers, farm
workers, professionals, etc.)
Population density
Income
Unemployment rate
Poverty rate
Problem #2: What Type of Districts Do
We Place Our Candidates Into? (cont’d)
 Eight
DFGs have been created
based on the 1990 United States
Census data
 Range from A (lowest
socioeconomic district) to J (highest)
 A, B, CD, DE, FG, GH, I, J
DFG: State Distribution
District Factor Group Distribution
For All Districts In NJ
120
100
80
60
40
20
0
A
B
CD
DE
FG
GH
I
J
DFG: Apprenticeship Teaching
Distribution
35
Distribution For Apprenticeship Teaching In Spring
2007
30
Field Placements
25
20
15
10
5
0
Dis trict Factor Groups
A
B
CD
DE
FG
GH
I
J
DFG: Alumni Distribution
Reported Distribution of Working Alumni
18
16
14
Alumni
12
10
8
6
4
2
0
Dis trict Factor Groups
A
B
CD
DE
FG
GH
I
J
Solution #2: Share Data,
Discuss, Revise



Evident discrepancy between where alumni get
jobs and where candidates are placed
We share evidence with faculty and key
stakeholders
They discuss and make appropriate decisions
DFG: In the future
Will have full record of where candidates
performed clinical experience
 Will have record of where they are
working
 Can correlate accordingly

Data Needed

A state database of teacher employment
Difficult to track alumni as they move from
school to school
 Unique teacher & school identifier


State database needs to integrate with
University and commercial marketing data
systems
Problem #3: Data Systems Not
Integrated

SOE’s recordkeeping is not integrated
with the University system
Student information is entered into SOE
system manually
 Limits power of reporting
 Duplicate person records may exist if Student
ID is not entered correctly

Problem #3: Data Systems Not
Integrated (cont’d)
 A “live”
data connection to University
system is not possible
 Technology
is not in place
 Support for integration is needed
More Efficient and Effective Use
of Resources
Relational database assists with:
 Streamlined SOE business operations




Generate mail merge letters
Provide reports
Automate payroll
Leverages existing data to create information
for program improvement
Started Slowly
Started with trying to code and track our
students properly
 Administrative assistant created
rudimentary Access Database
 When she left, we hired a consultant to
manage database
 He totally redesigned and reorganized it

Skill Set for Database
Administrator







Problem-solving
Relational database design skills
SQL proficiency
Knowledge of structured programming
language
Excellent communication skills
Work with faculty and technical staff
“People-skills”
Conducted Extensive Search
Advertised
 Set up search committee
 Interviewed many different applicants
 Required each applicant to take a test
 Presented problems to candidates to
assess problem-solving abilities
 Found many could enter data, but not
design or problem solve

Working with the University
Collaborating with the Arts and Sciences
 Establishing knowledge-base in content
areas and general education
 Addressing NJCCCS and Professional
Standards in discipline
 Aligning content courses and standards in
matrices

Working with the University(cont’d)





Meeting with individual departments
Establishing long-term relationships
Shifting to new paradigm--Learning Outcomes
Assessment
Working with A&S to collect data
Database Administrator playing key role in
collection of data across college
Leveraging TEAC Across the
University







Establishing the need for LOA: Middle States
Educating the A&S faculty: LOA process
Addressing resistance of A&S faculty
Establishing a relational database system for
university: program, college
Creating the infrastructure to collect data
Collecting multiple sources of data for A&S
Getting various groups to communicate and
plan.
Conclusion: The Power of a
Relational Database
Data-driven Decision-making requires
an integrated system of collecting data
from many different sources.
Systemic
 The
whole institution needs to be
vested in the collection of data
 Data needs to be collected on faculty,
students, courses, grades, scores,
rubrics, observations
 University and Colleges need to ensure
data collection systems are in place
early.
Integrated
All data sets need to be connected so
relationships can be established
 Queries made
 Reports generated
 Correlations and relationships analyzed

Benefits for Total University






Middle States Accreditation
Nursing
Engineering
College of Business
Program improvement
Student learning
School of Education Leads the
Way
 Ultimate
goal is to improve teacher
quality and impact achievement for
all students.
 Data provides the means to do this.
 Relational database is the engine
that makes this possible.
For more information please contact:
Vicki L. Cohen [email protected]
Director, School of Education
Marlene Rosenbaum [email protected]
Associate Dean, University College
Joshua Cohen [email protected]
Database Administrator