Transcript Presentation 1

Toward A Reasonable
Programmer Standard
Responsibility and Negligence in
Software Design
Overview of Presentation

It’s an ethical issue – ask questions; make
comments

Why do we need a definition?

New technologies = new legal issues

Software, though not new, is applied in new
areas

What to think when software causes harm?
Why a Definition?
 Objection:
you gain nothing by stating a
definition
 Objection:
I know in advance that your
definition is wrong
 Objection:
Standard of negligence
doesn’t apply to software
The Law and Negligence

“Failure to be sufficiently careful in a
matter in which one has a moral
responsibility to exercise care...”
Online Ethics Center: The Online
Ethics Glossary Center. (2003,
March 21).
Classic
example: leaving a rake on your
walkway – kills the mailman
Failing
to shovel snow; doctors who leave
surgery tools in a patient; etc.
When is a person negligent?

The reasonable person standard: “A phrase
used to denote a hypothetical person who
exercises qualities of attention, knowledge;
intelligence, and judgment that society requires of
its members for the protection of their own
interest and the interests of others.”
National Association for Court
Management: Glossary of Terms.

Invokes different requirements in
different societal roles – the
reasonable doctor is very different
from the reasonable homeowner

What is a reasonable programmer?
Test Case 1: Freeware

Software is distributed freely and
with a carefully worded license

What’s the worst that can happen?
Computer crashes -> data loss

Holding a programmer responsible
for the data is too demanding – not
everything is negligent
Test Case 2: Free Algorithm

Programmer posts implementation of a
sorting algorithm

Algorithm doesn’t work on negative numbers

Company uses algorithm in air traffic control
software

Moral intuition: company’s fault, not the
algorithm programmer; notion of direct
responsibility
Test Case 2: Ethical Analysis

Company had contractual and moral
obligation to test their software sufficiently

Programmer had no intention of using
algorithm in critical environment – no
obligation

Conclusion: expectations of performance
derive from moral and contractual obligation
Test Case 3a: Virus Junkie
 A programmer
Q gets high off of
creating viruses
Q
unwittingly unleashes virus,
costing U.S. companies millions
 Should
Q be held responsible?
Test Case 3b: Router

A properly functioning antivirus
program at company C would have
stopped Q’s virus

Is C in any way responsible for the
harm caused by the virus?
Test Case 4: Therac-25

Atomic Energy of Canada Limited: faulty
software leads to overdose of radiation
-> death

AECL probably was ‘sufficiently careful’ –
post-accident review showed many hours of
testing

Contrasted with bug that fails to save internet
books mark: serious consequences important
to definition, as is amount of testing
proportional to potential harms
Test Case 5: Patriot Missile

Programming flaw -> system operates over 20 hours
= failure -> military issues patch, deployment is slow

Barracks destroyed, software patch arrives the next
day

Raytheon’s fault? Testing revealed the flaw

Army’s fault? Slow deployment of patch, vague
memo on proper operation, project extended beyond
intended lifetime

New definition: direct causation of flaw, or indirect
causation of conditions that lead to flaw
Final Definition
 Unreasonable,
direct failure to be
sufficiently careful in software design
and testing, or causation of such
conditions leading to serious harm,
wherein a programmer has a moral
obligation to do so from a contractual or
otherwise reasonable expectation.
Test Case 6: More Viruses
 Lots
of subtle issues, what can we
reasonably expect from each party?:
 Obviously,
punish those who create virus
 OS designer let flaw propagate?
 Owners and designers of networks that
allow propagation?
 User who downloads attachment with
subject “Haven’t heard from you in a
while!”?