Topic 2: Quality Metrics
Download
Report
Transcript Topic 2: Quality Metrics
Topic 2
-
Quality Metrics
Bart Vannieuwenhuyse
Senior Director Health Information Sciences
Janssen R&D
Bart Vannieuwenhuyse
1
Brussels, 20 March 2013
Topic 2 – Quality Metrics
Scope of the project –
Purpose – “improving”
“what gets measured, gets done”
What are “Quality Metrics”?
A “metric” is a measure.
“Quality” is something a “customer” defines.
A “Quality Metric”, therefore, is a measure of quality as defined by the
customer.
NOTE 1: A “customer” might be defined as anybody with an expectation of
receiving something of value in exchange for something else of value, either
external to or internal to an organization.
NOTE 2: Not all “Metrics” are “Quality Metrics”
http://www.capatrak.com/Files/PresH%20-%20Metrics.pdf
Bart Vannieuwenhuyse
2
Brussels, 20 March 2013
Topic 2 – Quality Metrics
Contributing projects
Bart Vannieuwenhuyse
3
Brussels, 20 March 2013
Topic 2 – Quality Metrics
Convergence challenges
Define scope – agree on areas with highest need
“Internal” vs “External” application of metrics
Potential opportunities to leverage (tbd)
Improving efficiency of collaboration in project
Process to improve project deliverables
Measuring quality of (external) data
Identifying quality of (sub)contractors
Bart Vannieuwenhuyse
4
Brussels, 20 March 2013
Topic 2 – report back
• Quality Metrics – domains:
– Project quality
• Quality of deliverables – internal “peer review” generally adopted
• Project management – “on time – on budget” generally adopted
– Project impact
• Uptake of solutions – need for further development of metrics
(e.g. Service registry using text mining in BioMedBridges)
• Scientific impact – publications, possibility to further improve on
speed and breadth of sharing results
• Societal / health care impact – need for further development of
more standardized approaches
– Data quality …
Bart Vannieuwenhuyse
5
Brussels, 20 March 2013
Data Quality
“Data quality is the end product of a whole process”
Quality of
Solution
Quality of
Usage
Metrics 1
Metrics 2
“All elements need to be of the right quality”
A Rolls Royce with 3 wheels is a crappy car
Bart Vannieuwenhuyse
6
Brussels, 20 March 2013
Data quality - process
• Context of data creation – meta-data
– Should be made explicit
– Provenance must be clear
• “medical context” - clarity on reimbursement and “medical
practice”
• Clarity on who created the data
– Mapping to common ontologies
• Type of use drives selection of data
– Data should be fit for intended use – Care vs Research
– Options to select data sources on available meta data
Bart Vannieuwenhuyse
7
Brussels, 20 March 2013
Data quality - metrics
• Quality of solution – metrics 1
– Adopt existing standards e.g. ISO 25000
•
•
•
•
•
SDLC like approach (engineering)
Functional suitability
- Reliability
Performance efficiency
- Security
Compatibility
- Maintainability
Usability
- Portability
– STEEEP – Safe Timely Efficient Effective Equitable Patient-centered
(IOM – US)
• Quality of usage – metrics 2
• Effectiveness
• Efficiency
• User satisfaction
Bart Vannieuwenhuyse
- Freedom of risk
- Context coverage
8
Brussels, 20 March 2013
Data quality - dimensions
• Accuracy
• Quantitative vs Qualitative data (origin of data)
• Benchmarking to check accuracy (TransForm, OMOP, EU-ADR)
• Completeness
• Needed granularity – data available? (TransForm selection tool)
• “Longitudinality” – length of available Hx
• Timeliness
• Data “freshness” – latest update
• Reliability
• Who created the data – who is responsible
• Trustworthiness – traceability (versioning, time-stamping)
• Structured - Unstructured
Bart Vannieuwenhuyse
9
Brussels, 20 March 2013
Next steps
• Data Quality Metrics community
– Convene individuals from all EU projects dealing with re-use of existing data
– Consolidate existing approaches across EU projects – share current solution
– Classifications of data quality metrics – check availability of ISO standards for
eHealth data – if not, consider developing one? (ISO 8000 general data quality)
– Consolidate available quality standards of solutions (e.g. ISO 25000)
– Recommendation for projects to focus on data quality even before projects starts
– Develop common approaches to evaluating data quality – “benchmarking”
analogy of computer chips // radar-graph
– Have guidelines on Data quality – e.g. when creating new data / attention to
meta-data (training)
– Develop and share analytical methods that deal with “imperfect data”
Data quality is a journey
And even the longest journey starts with the first step
Bart Vannieuwenhuyse
10
Brussels, 20 March 2013