Transcript VA Data Lifecycle

Reading for Fun – Official History
 VistA*/U.S. Department of Veterans Affairs national-
scale HIS
 Steven H. Brown, Michael J. Lincoln, Peter J. Groen,
Robert M. Kolodner
 International Journal of Medical Informatics 69 (2003)
135/156
 VistA Document Library (VDL)
 www4.va.gov/vdl
(With Pharmacy Additions)
Richard Pham, PharmD
Enterprise Architect
OI&T Corporate Data Warehouse – Architecture
[email protected]
The Health Care System Is MUCH MORE COMPLICATED Than
Other Business Processes (~7% of VistA)
Each Process Is A World Unto Itself
DHCP/VistA/CPRS
 VistA – Veterans Health Information Systems and
Technology Architecture - Refers both to the architecture
and the database which the architecture supports
 DHCP - Decentralized Hospital Computer Program – The
DOS (Unix-like) system where many of VistA’s non-clinical
entries take place
 CPRS - Computerized Provider Record System – A user-
friendly GUI providing access to clinical order entry
functions
Objectives
 The main objective is to understand the data lifecycle
of VA’s VistA/CPRS and the user experience of
VistA/CPRS
 A high-level overview of VistA Internals
 Learn about data structures and outputs in VistA
 Learn where data enters and travels throughout the VA
 Try to make sense of data resources within the VA and
how they are accessed
The VA Data Lifecycle
The VA Data Lifecycle
Core Patient Care Functionality
 VistA is first and foremost an Electronic Medical
Record. The architecture design supports veteran
health care.
Core Patient Care Functionality
 VistA Internals
 DHCP
 CPRS
VistA Internals 101
 MUMPS
 Server and Operating System
 Kernel
 “Three Wise Men (Managers)”
 TaskMan
 MailMan
 FileMan
 Modules
Massachusetts General Hospital Utility
Multi-Programming System (MUMPS or M)
 My definition in English: M is a programming language
designed for hierarchical databases that is convenient for
medical applications or anything else where speed and data
storage upkeep are a problem and programmer
intelligence/organization is not
 My technical definition: M is a Turing-complete, low and
high-level, imperative, machine-compiled (no longer
interpreted) programming language utilizing a hierarchical
global array file structure
 Used commonly in healthcare and financial industry
settings
Structure of The Veterans Administration
Data Efforts (Late 1970s)
VHA Ancestor
Department of Medicine
and Surgery (DMAS)
VHA-OI Ancestor
Computer Assisted System
Staff (CASS)
OI&T Ancestor
Office of Data
Management &
Telecommunications
(ODM&T)
Comparing The Two Offices
CASS
ODM&T
 Decentralized design
 Centralized design
philosophy
 Rapid, agile development
philosophy
 Bureaucratic, process-focused
development
 SME-involved development
 Development without SME’s
Highlights of ODM&T Development
 Took 6 years to deploy APPLES Pharmacy at 10 sites
 A 1980 paper detailing ODM&T’s transactional patient
treatment file (PTF) system promised an interactive
national solution by 1990.
 Navigating the mandated 17 steps between system
specification and deployment alone is said to have
required at least 3 years.
Beginnings of DHCP
 There were subject matter experts that believed that
they could put out useful applications faster than the
ODM&T sloth
 Development of the testing and principles was done
unofficially throughout the late 1970s
 The Second
Original DHCP Design Principles
 A commitment to rapid prototype development
 All use ANSI MUMPS
 Modular Design
 Actively Maintained Data Dictionary
 Code Sharing/Portability
 Involve the SME’s
DHCP Kernel
 Functions as both an operating system for VistA
applications and an M virtual machine
 Kernel shields DHCP modules from needing to know
hardware and OS configurations on the server
 Isolates M to the ANSI standard (1995)
 Provides a toolbox of standard functions for most
programmers
VistA to Relational Database
Terminology
VistA (Example)
Relational Database (Example)
Namespace (VHAFRE)
Database (VA Fresno)
“Package” – Not hardcoded
Schema (RxOutpatient)
File (50.68 – VA PRODUCT)
Table (NationalDrug)
Field (.01 – NAME)
Column (DrugNameWithDose)
Domain (cardinal/decimal, setofcodes,
freetext/wordprocessing)
Field Type (numeric, boolean, varchar)
Internal Entry Number (IEN or .001)
~Key (9722)
Record
Tuple/Row (ISOSORBIDE
MONONITRATE 120MG TAB,SA)
MUMPS Classic Database
 One Data Type
 String (Text)
 Other types
 Cardinal Numbers
 Float Numbers
 $H Dates
 One Data Storage Type
 Multidimensional Array aka Globals
 Dynamic (duck) typing
VistA Data Organization
 Namespace
 File

Field
 Record
 654 (VAMC Reno)
 File 120.5 (GMR Vitals)

Field 0.1 (DATE/TIME VITALS TAKEN)
 IEN-1, BP, 140/90
 Most Files have an entry at the 0.001 Field called “IEN” or
“Internal Entry Number” as an identity key to mark the
record as unique
Upside of Using Globals
 Faster - No joins
 Faster – All parameter pointers built in
 Faster – Direct and planned programmatic access to
database (Look at SQL execution plans)
 Less Data Storage Overhead and faster paging – If the
data point does not exist in the array, there does not
need to be a fixed point like in relational
Downside of Using Globals
 No Intrinsic Structure and No Enforcement* - M
believes whatever you put into the globals (most M
programmers view this as an advantage while
relational programmers have an MI)
 ACID-compliance not mandated
 (Il)logical data structures guaranteed – There are
many interesting* ways that the M programmers
modeled the data that does not make sense to later
viewers
MUMPS Quirks
 Whitespace (Space) matters
 Requires knowledge of kernel and sometimes lower-
level concepts
 Programming Without Type or Structure Enforcement
 VA programming standards and conventions
The Three Wise Men (Managers)
 TaskMan – The man(anger) that schedules tasks to the
kernel
 MailMan – The man(anger) that messages between
the user, TaskMan, and any other two-way
communication between packages
 FileMan – The man(anager) that controls internal file
(data structure) interactions
TaskMan
 TaskMan handles application processing:
 Creation of application processing tasks
 Scheduling these tasks
 Monitoring health/statistics of these tasks
 If kernel is the brain, then TaskMan is the body of the
operation
 If programming, NEVER EVER use the TaskMan global.
This subverts TaskMan’s scheduling queue, and can cause a
system memory leak. Use the calls instead…
MailMan
 VistA needs a way to pass and receive data from the
database to other areas
 MailMan fulfills this function in the pre-TCP/IP days
 “Electronic mail” doesn’t mean just email
 Practically any message between the database and
anyone else (the end-user, another site, or application,
etc.) can be moved this way
 Gives programmers methods to both receive and
return data to the database
FileMan
 A higher-level method to access the VistA database
without exposing a programmer interface
 Mostly menu-driven
 One can use limited programming
 Serves as the model for all other modules that interact
with the VistA database
ODM&T Initial Action Plan To DHCP
Development (1980)
 Ordered that development stop
 Fired the developers
 Removed the hardware
 Cut the DMAS budget so it would never happen
again…
The official history
Development Goes Underground
 Developers that survived the ODM&T purge
continued their work as a black project in DMAS
 During 1980 and 1981, the survivors (Underground
Railroad) continued work on developing modules for
system integration
Modules
 Modules are programmed to interact with the VistA
database
 Most use Fileman as a model for programming
Some of the Many Modules
Medicine
Surgery
Dentistry
Nursing
Pharmacy
Laboratory
Care
Management
Patient Care
Encounters
ADT
Mental Health
EDIS
Oncology
Nutrition and
Food Service
Imaging/PACS Prosthetics
Not really in the scope of this presentation to cover each
module .
Try the VistA Documentation Library:
http://www4.va.gov/vdl/
Or
VHA eHealth University (VeHU): http://www.vehu.va.gov/
Acceptance and DHCP 1.0
 Once there was a critical mass of packages that were
shown to be useful, the tide turned and the project was
blessed…
 Initial testing done in 1980-83
 1.0 installation was in 1985
 Most of the underlying packages can still be
recognized by the original programmers
Special Topic – The Pharmacy
Package (File 50 Series)
 Where are the files?
 File 50 Series is the main line, though there are other
places
 How many files are there? - 437
 Look at screen capture on next side for File 50 series.
 How many columns? - 3174
How Many File 50 Tables Are
There? (1096 – 659 = 437 Tables!!)
How Many File 50 Series Columns
Are There? (8526 – 5352 = 3172!!)
Further Information On The
Background
 For the VA Base M Training
 http://vaww.vistau.med.va.gov/VistaU/MTraining/Def
ault.htm
 For the VA Programming Standards and Conventions
 http://vista.med.va.gov/sacc/
 For the VA Document Library
 http://vista.med.va.gov/vdl/
Computerized Patient Record
System (CPRS)
 A Real-Time Order Checking System that alerts clinicians during the ordering session
that a possible problem could exist if the order is processed
 A Notification System that immediately alerts clinicians about clinically significant
events
 A Patient Posting System, displayed on every CPRS screen, that alerts clinicians to issues
related specifically to the patient, including crisis notes, warning, adverse reactions, and
advance directives
 The Clinical Reminder System, which allows caregivers to track and improve preventive
health care for patients and ensure timely clinical interventions are initiated
 Remote Data View functionality that allows clinicians to view a patient’s medical history
from other VA facilities to ensure the clinician has access to all clinically relevant data
available at VA facilities
 CPRS DOES NOT STORE DATA!!!
CPRS Internals
 Written in Embarcadero Delphi (NOT in MUMPS)
 Connects from the Graphic User Interface to the VistA
database using a Remote Procedure Call (RPC) Broker
 This Remote Procedure Call Broker translates
instruction sets from other languages into M
Present State of VistA
 Large MUMPS database
 Over 50+ Main Clinical Packages
 Over 10,000 + Tables
 Each medical center runs somewhere between 2-4 TB
worth of data over 30 years (mostly imaging)
 Many processes
 300+ MB of running executable at any given time
 Over 20,000 subroutines (VDL)
 Many simultaneous users
The VA Data Lifecycle
National Analytic Systems
 A list of systems that support policy, planning, and
congressional needs
 There are more extracts than this, but I have chosen
the most common ones…
Local VistA
Installations
Local VistA
File
Site 1
Local VistA
File
Site N
Host
Location
DSS – Austin, Tx
PBM – Hines, IL
NPCD – Austin, Tx
Etc.
Local Extract
Software
Site 1
VistA Extracts
Load and
Translate
Software
Local Extract
Software
Site N
VistA Extracts
DSS Extract Software
PBM Extract Software
NPCD Extract Software
Etc.
DSS Extract Files
PBM Extract Files
NPCD Extract Files
Etc.
Diagram of Data
Sources Available to
VA Researchers
DSS Production Database
PBM Database
NPCD Database
Etc.
National
Database
DSS Build Software
PBM Build Software
NPCD Build Software
Etc.
Build SAS
Datasets
DSS NDE SAS Datasets
Medical SAS Datasets
Etc.
Custom
Extract By
Database
Owner
Custom
Extract
SAS Datasets
Research
Database
PBM Custom Extract
Medicare Data
National Death Index
Etc.
External
Data
Researchers
46
Systems to Support Planning
 Decision Support System (DSS)
 Supports accounting and costing for the OIG, GAO,
CBO, and other auditing agencies
 Allocation Resource Center
 Supports personnel and resource allocation at the
medical center level
 Workload capture, resource allocation
 Basis for the VERA (VA’s Fund Control Point) Model
VistA DSS Data Feeds
ADM
DEN
IVP
LAR
MTL
PAS
PRO
SUR
UDP
Admissions
Dental
Pharmacy: IV
Lab Results
Mental Health Test
Patient Assessment
Prosthetics
Surgery
Pharmacy Unit Dose IP
CLI
ECS
LAB
MOV
NUR
PRE
RAD
TRT
Clinic Visits
Event Capture System
Lab
Patient Movement – Inpatient
Nursing
Pharmacy OP
Radiology/Nuclear Med
Patient Treatment Specialty
May be transmitted by site at any time of the month, ideally
around the 25th of the month prior to the processing month.
Systems to Support Research
 National Patient Care Database
 An integrated set of data that captures a patient’s care
encounter with the VA
 Corporate Data Warehouse –
 A near real-time accumulation of much of the same data
 The result of the Health Data Repository process
NPCD Data Flow Diagram
VistA
MailMan
•NPCD data is sent from the facilities to the AAC via
MailMan messaging
•Once a message reaches the AAC MailMan server,
It automatically moves to the Data Management
Interface System (DMI)
•NPCD and other applications retrieve their respective
data from DMI for use
• Acknowledgement messages are sent to facilities
z900
SD
PROLIANT
8000
Data
Data
extracted
extracted
&
backed
by application
up nightly M-F
ESC
Austin
MailMan
Server
Data Stream
DMI
SD
DLT
NPCD
Acknowledgement
message
Data received
in DMI 24x7
Acknowledgement
message
HL7 data to Oracle DB
NPCD Processing
UNIX
Daily
Data
Loading
Flat files are indexed
and loaded into the
database daily
Oracle on Unix
NPCD
DSS
data
extracted
z900 (MAINFRAME)
Master
Extract File
(MEF)
SAS
Data is checked
for duplicates bimonthly
WINDOWS
VSSC/
KLF Menu
Data is extracted
and filtered for
reporting twice a
month
Resources for Further Information
 VA Information Resource Center (ViREC)
 http://www.virec.research.va.gov
 National Patient Care Database –
 (Internal) http://vaww.aac.va.gov/npcd/
 National Data Systems (NDS)
 (Internal) http://vaww4.va.gov/NDS/DataAccess.asp
Secrets of the VA Data Universe
 This was an extremely brief introduction to a
complicated area
 I have another presentation on the availability of
databases in the VA and how to access them for
operations and/or research
The VA Data Lifecycle
Regional Remote Data Processing
Center Shadow Systems
 A offsite backup process to ensure continuity of
operations for VistA Patient Care
Regional Data Processing Centers
(RDPCs)
 Read only backup VistA systems are set up to take
journaling files
 When a record is written or altered to a local medical
center’s VistA, a journal file with that entry is prepared and
sent to a Regional Data Processing Center
 This maintains an active backup in case the local medical
center’s VistA goes down
 VistA goes down
 OpenVMS goes down
Regions and RDPCs
 Region I RDPC – Sacramento (SAC) and Denver (DEN)
 Region IV RDPC – Philadelphia (PHI) and Brooklyn
The Day VistA Died
Northern Cal Without VistA
 The medical staff was forced to write discharge instructions and notes






on paper.
The electronic lists of instructions and of medications were not
available for the patients being discharged.
Patients being discharged could not be given follow-up appointments
at the time of discharge.
The appointments had to be made later and the patient notified by
phone.
There were delays in obtaining discharge medications and patients
remained on the wards longer than would normally be required.
The nurses administered medications to the patients and used the
paper Medication Administration Record, or MAR, to record the
administration events.
Initial medication passes were interrupted and delayed until the paper
copies of the (MAR) could be printed.
RDPC Denver and Brooklyn
The VA Data Lifecycle
Business Intelligence
Business Intelligence in the VA –
Making the Data Work For Us
 VistA has a wealth of clinical and administrative data
available
 In the past, giving a value-added, timely VistA dataset
was hard
 Querying the active system with minimal impact
 Needed an interface between M and analyst languages
(SAS, SQL, etc.)
 Easy to read reports was hard to build
Corporate/Regional Data
Warehouse
 Takes a copy of the journal file that goes into the backup
shadow system
 Translated from the M array to a relational database format
using Intersystems Cache’s class mapping program
 Staged in a Feeder-Collector system for collection
 Indexed and value-added columns produced and loaded to
an VISN RDW Server
CDW Governance
VHA Business
Owners/SME’s
Communicates
Organizational
Priorities
CDW Governance Board
Organizes SMEs
and Data
Stewards
Sets and monitors
domain, work priorities,
and timelines for
completion.
OI&T
VHA-OI
Data Quality
10N, OIA, VBA
Provides
Documentation
and
Clarification of
Business Logic
Corporate Data
Warehouse
CDW Governance Is In VHA’s Hands
 Ordered By VHA
 Domain and Work Prioritization By CDW Governance Board
 Chair – KLF (OIA)
 Vice-Chair – Larry Mole (Public Health SHG)
 Monitored and Accountable To VHA
 Project management provided by John Quinn (National Data
Systems) and KLF (OIA)
 Supported By VHA
 OI Data Quality
 Business Owners
 PBM’s Data Steward is Rob Silverman
“As the number of eyes goes up,
the number of bugs goes down.”
 Writing documentation about the business logic of the
files and fields
 Answering end user questions about the data
 Data validation
 Preferably before
 Inpatient Pharmacy
 ADR/Allergy Package
1st category models are simple –
V Health Factor
FMFile
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
V HEALTH FACTORS
Source Mapping
FMField
ResolveFld DWTableName
HEALTH FACTOR
HealthFactor
HEALTH FACTOR
0.01 HealthFactor
PATIENT NAME
HealthFactor
EVENT DATE AND TIME
HealthFactor
VISIT
0.01 HealthFactor
VISIT
0.01 HealthFactor
LEVEL/SEVERITY
HealthFactor
VISIT
HealthFactor
ENCOUNTER PROVIDER
HealthFactor
COMMENTS
HealthFactor
DWFieldName
HealthFactorTypeIEN
HealthFactorType
PatientIEN
EventDateTime
VisitVistaDate
VisitDateTime
LevelSeverity
VisitIEN
EncounterStaffIEN
Comments
2nd category models require
transformation – Prescription
Prescription
and 1st fill
Refill
Partial
Fill
Fileman
Prescription
Only
All Fills
Data Warehouse
3rd category models not usable
without transformation - PCMM
Levels of Data
 National – Corporate Data Warehouse (CDW)
 Region – Regional Data Warehouse (RDW)
 VISN – VISN Data Warehouse (VDW)
 Medical Center – Local Data
Entities Who Produce Business
Intelligence Products
 National – VSSC, PSSG, DMDC, HEC, ARC, DSS, BIPL, OQP, PCS, PBM
 Region – Regional BISL Teams
 VISN – VISN Data Warehouse, VISN PBM
 Local – DSS
 Bolded are ones that have substantial resources in clinical business
intelligence
 PSSG handles much of the GIS and Statistical Demography for the VA
Data Access
 VISN and Station Level – Contact Your VISN Database
Manager
 Regional/Corporate Access – Contact NDS for the 9957
Permissions
Operational Challenges of VistA
 System Resources
 $8 Billion investment over 20 years
 New needs for new domains
 MUMPS Programmers must be internally trained (and many of
them are retiring or dying)
 Communication with Other Systems
 HIMISS compliance with data interchange
 E-functions (billing, prescribing, verification)
 Interagency Cooperation – DoD and NHIN
 Business Intelligence
 Closing the data lifecycle and bringing back clinical data for
knowledge discovery
Why Is Pharmacy ALWAYS Picked
As An IT Test Case Project?
 Pharmacy is data savvy
 Local data quality control from ADPACs
 Federated data quality control from VISN PBM, CMOP, PBM
SHG
 Pharmacy is one of the few domains that have active business
logic SME’s
 Pharmacy did not contract institutional memory out the door
 Pharmacy gets things done
 Pharmacy has more technology success stories as a collective
than any other PCS office (BCMA, Automation, Central Fills)
 Pharmacy actively mines data
Acknowledgments
 Kernel – Jack Schram (Oakland OIFO)
 SQLI – Ellen Zufall (SF IRMS)
 RPC Broker and MUMPS coding – Perry Richmond
(VISN 18 BI)
 Regional Data Process – Vincent Bui and Ken Koenig
(Region I SQL Back Office Team)
Acknowledgments
 OI&T Business Intelligence Product Line (BISL)
 Jack Bates – Manager, OI&T BIPL
 Stephen Anderson – Lead Data Architect
 Mike Baker – Lead ETL Architect
 Denver Griffith/Ken Fuchsel – Server Administrators
 Dave Fackler
 Ron Talmage
 Dan Hardan, Jeff King, Jeff Price
Questions