Transcript course introduction

Safety Analysis of
Software-intensive Systems
Tor Stålhane
IDI / NTNU
What is safety
A system is safe if it behaves in such a way
that it does not harms people, equipment
or the environment.
Safety is a relationship between a system
and its environment
Safety is not an add-on to a system but an
integrated part that needs to be
considered from day one of a
development project.
2
What is safety analysis - 1
Safety analysis is the totality of activities that
are used to identify
• Hazards that may rise when a system is
put into operation.
• Ways to remove these hazards or reduce
their consequences to an acceptable level.
• Actions needed throughout the system’s
development to ensure that all safety
requirements are implemented.
3
What is safety analysis - 2
The soft side of safety analysis: Collecting
and analyzing info. The problems are
human related.
• Collecting info from all stakeholders
• Organize it in such a way that it can be
used to create
– Safety requirements for development
– Safety tests
– Safety routines and procedures for the
operation and maintenance of the system
4
What is safety analysis - 3
The hard side of safety analysis: Defining
barriers. The problems are related to both
humans, software and hardware:
• How can we construct barriers against
hazards in the software?
• How can we define operating procedures
for handling crises?
5
Collecting info - 1
All stakeholders must be involved in the
safety analysis since they all possess vital
info.
Safety analysis is thus a people intensive
process – critically dependent on
• The participants’ experience and
knowledge.
• Our ability to elicit relevant info
6
Collecting info - 2
We need to identify
• All potentially dangerous events - hazards.
• The events’ consequences.
• The events’ probability or frequency – at
least in qualitative terms.
• Important scenarios. The quality of the info
from a person increases when the
questions are related to a scenario.
7
Tools and methods - 1
The methods that we use in safety analysis
– especially in the early phases – must be
able to involve all stakeholders.
We need methods that are easy to
• Learn and understand
• Use on real-life problems
• Apply to software, hardware, people and
routines and procedures.
8
Tools and methods - 2
Which tools and methods to use depend on
who participate in the process, the info
available and how it is represented.
The info available will depend on where in
the development process we are.
The way the info is represented is, at least
partly, something that we can influence.
We have good experience with using UML
diagrams in all phases.
9
Tools and methods - 3
As we move from a concept to a high level
design and then on to detailed design and
implementation, more and more
• Information will be available
• Decisions will be made and thus leave us
with less freedom when making new
decisions.
Thus, we will need different analysis
methods in different phases of the
system’s life cycle.
10
Project time and decisions
Knowledge
Freedom of decisions
Experience
Time
Concept
HLD
LLD
Implementation
TD
11
The concept phase
Most systems start as a concept, e.g.:
• Automatic shut-down of production when
we discover a gas leakage.
• All patient info kept in a central database
and be available for all that need it through
a data network.
• Complete overview of all our trains –
where they are, their speed and so on.
12
Electronic patient journal – Concept
Physician
Patient journal
system
Primary Physician
Nurse
Lab system
Top level view – system and stakeholders
13
System
concept
Experience
Operational
environment
Knowledge
Experience
Knowledge
Tools and
methods
Stakeholder
Stakeholder
Hazards and
barriers
14
Preliminary Hazard Analysis - 1
The preliminary hazard analysis is used
early in the process. This is reflected in
the level of details required in the PHA
table.
We can include both hazards and the
corresponding preventive actions –
barriers.
Barrier descriptions are converted to
system requirements.
15
Preliminary Hazard Analysis - 2
Hazard
Cause
Main effect
Double check
all patient info
inserted
Wrong info
inserted
Somebody
retrieve
wrong info
Stored info
corrupted
Kill or hurt
patient
:
Double store
and check
Redundant
patient info
required for
retrieval
Wrong patient id
used at insertion
or retrieval
:
Preventive
action
:
:
16
Requirements
Once we have decided to go ahead with the
project, we need to elicit and document
the requirements. These consist of two
components:
• The functions used to fulfil the customer’s
needs
• Barriers against hazards identified in the
PHA
17
Use Case for Electronic patient
journal
Medication
Physician
Diagnosis
Primary Physician
Documents
Orders and
responses
Nurse
Lab system
Treatment plan
18
Needs
Operational
environment
Hazards and
barriers
Customer
Requirements
Experience
System
concept
Expectations
Knowledge
Experience
Knowledge
Methods and tools
Stakeholder
Stakeholder
New hazards
and barriers
19
Safety in the requirements phase
Functional requirements – which services
should the system offer to its users?
Use case diagrams and textual use cases
have turned out to be two efficient ways of
documenting this. They
• Are easy to understand for all
stakeholders.
• Can be used as input to several safety
analysis methods.
20
Use case for medication
21
Functional FMEA
Component Id
Treatment plan
Function Failure
mode
Local
effect
Return
wrong data
Check
Return data
current
for another
treatment patient
Return no
data
Wrong
Update
treatment update
plan
No update
Sys effect
Actions
Wrong
decision
Patient can
get hurt or
killed
Check
against other
data
available on
this patient
None
Suspend
decision
Alternative
data source
Wrong
data in
journal
Wrong
treatment
Implement
update
receipt
Seriousness
H
H
L
H
H
22
Misuse case
Review
treatment
plan
Review
drug data
Doctor
Review
documents
Review
diagnosis
Wrong
update
Delete
data
Unlucky
doctor
Data is
lost
Network
is down
Faulty
system
23
High level design
When we enter high level design, all
identified hazards and barriers have been
converted to requirements.
The high level design can be documented
for instance as
• Package diagrams
• High level class diagrams
• High level sequence diagrams
24
Part of electronic patient journal
Patient
documents
General
patient info
Treatment
plan
Patient drug
data
Patient
diagnosis
25
System
concept
Experience
Operational
environment
Extended
requirements
Knowledge
Experience
Knowledge
Tools and
methods
Stakeholders
New
hazards
Stakeholders
Barriers and
tests
26
Safety and design
Packages and classes can be viewed as
components and we can thus make our
safety analysis much more detailed.
Important methods that can be used at this
stage are for instance:
• HazOp, for architectural design.
• Component FMEA
27
HazOp - 1
HazOp uses study nodes as units of
investigation and guide words to help in
the hazard identification process. This
makes the method quite efficient for
identifying hazards
On the other hand, HazOp also requires
more information – the system’s
architecture – to define the study nodes.
28
HazOp - 2
Guideword
Less
Study
node
Consequences
Causes
General Incomplete info Missing updates
patient
which can lead
info
to wrong
Lost updates
treatment
Incomplete
updates
Possible
solutions
Check and
sign-off for
updates
Mirror
database
This is a simple version – more elaborate
versions gives more info and requires more work.
29
Failure Mode Effect Analysis - 1
FMEA will systematically check each system
component
• How can this component fail?
• What are the consequences for the
component?
• What are the consequences for the
system?
• How can we handle the hazard?
30
Failure Mode Effect Analysis - 2
Component
Patient drug
data
Failure
mode
Give
wrong
data
Effect
Wrong
medication
description.
e.g. dosage
Incorrect Outdated
or missing medication
update
description,
e.g. dosage
Handling or
barrier
Seriousness
Check dosage High
against
medication
rules database.
Prevent too
high dosage
High
31
Failure Mode Effect Analysis - 3
The failure Mode Effect Analysis:
• Offers a systematic walk-through of one or
more system components.
• Focuses on preventions – barriers - rather
than cures and fixes.
• Produces an easy-to-use list of hazards
and ideas on how they can be removed or
handled.
32
Detailed design
Just as high level design, the detailed
design can be documented for instance as
packages, class diagrams and sequence
diagrams.
We have more info than we had during high
level design and we can thus make a more
detailed safety analysis.
33
Patient
info
Treatment
plan
Patient
drug data
Patient
documents
Drug DB
Test results
Current
treatment
If changes necessary
Drug
description
Update drug data
34
Operational
environment
Barriers
Experience
Detailed
design
Knowledge
High level design
Experience
Knowledge
Tools and
methods
Stakeholder
New
hazards
Stakeholder
Barriers and
tests
35
Component FMEA
Component
Id
Treatment plan
Failure mode Local effect
Sys effect
Return wrong Wrong info to Wrong
treatment
doctor
decision
Update wrong
drug data
Update drug
data for
wrong
patient
:
Actions
Sanity check
Seriousness
H
H
Wrong info in
Wrong
patient’s
medication
journal
Update receipt
:
:
:
H
:
36
Implementing barriers
All hazard analyses must lead to barriers
that have one of the following effects:
• Prevent a hazard from leading to a
problem.
• Prevent a problem from causing a
dangerous event.
• Reduce the effect of a dangerous event if
it cannot be prevented.
37
Barrier roles
Event
Reduction
Reduce effect
of event
Barrier 6
Prob.
Barrier 5
Handling
Prevent event from having
bad consequences
Barrier 4
Barrier 3
Barrier 2
Risk
Barrier 1
Prevention
Prevent risk from becoming
a problem
38
Barrier reliability
All barriers work
as planned
Minimum
achievable
risk
Acceptable
risk
Barr. n
Barr. 3
Unmitigated risk
from EUC
Barr. 2
Barr. 1
Barrier failure
RM
RA
Ru
Risk
39
Realizing barriers
Barriers in software can be realized in
several ways. It is important that they do
not lead to a large increase in complexity.
One way to realize barriers is to use
patterns such as:
• Façades or wrapper façades
• Protected single channel
• Sanity checks on values
• Monitor - actuator
40
Façade pattern
41
Façade pattern sequence diagram
42
Wrapper façade pattern
43
44
Sanity check pattern
45
Monitor - actuator
46
Safety analysis research - 1
Research on safety analysis are concerned
with some broad problem areas: How to
• Implement barriers to prevent or reduce
the effect of dangerous events?
• Create safety analysis patterns?
• Elicit the necessary information from all
stakeholders?
47
Safety analysis research - 2
Our current research in the area of software
safety has focused on:
• Which methods are the easier to
understand, learn and use?
• What is the relationship between method
and system representation – is it e.g.
easier to base an analysis on scenarios
than on a requirements list?
48
Safety analysis research - 3
How can we
• Improve the safety analysis by making
earlier experiences on similar systems
available to all stakeholders?
• Most efficiently move from identified
hazards to
– Prevention, e.g. barriers
– Tests – do the barriers work as intended?
49
Last but not least
It is possible to be too safe. A chainsaw with a fully
protected blade is
• Absolutely safe
• Absolutely useless
It is not possible to be absolutely safe. Whatever
you do or don’t do, the probability of dying
during the next hour is more than 10-6.
Make sure you have a nice day.
50