Transcript Existential Risk – Diplomacy and Governance

Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Plan for the day and introductions
9:30-10:30
We are here to create a prioritised shortlist
of concrete objectives for the international
community to pursue on a 5-10 year
horizon to reduce existential risk.
Concrete objectives are specific
programmes which can be realised in the
medium term and lead towards a reduction
in existential risk.
Strategic objectives are desirable states of
affairs which, were they achieved, would
reduce existential risk.
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
Introductions
 Name
 Institution or background
 Source of interest or expertise in
existential risk or governance
 Your personal goals from this session
Norms and working styles
 Please do not discuss attendees names
or institutions or attribute claims to
attendees without their explicit consent.
 Unless an urgent clarification is needed,
leave questions to the end.
 In full group discussions, the moderator
will call on people who show an interest
to speak.
 If you have suggestions for process
improvement, feel free to share especially
between sessions.
Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Establishing shared knowledge
10:30-11:30
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
What is existential risk?
Source: Bostrom (2013)
What are some existential risks?
 Natural
 Pandemic
 Asteroid strike
What are some existential risks?
 Natural
 Pandemic
 Asteroid strike
 Anthropogenic
 Nuclear war
 Artificial pandemic
 Extreme climate change
 Bad outcomes from artificial intelligence
What are some existential risks?
 Natural
 Pandemic
 Asteroid strike
 Anthropogenic
 Nuclear war
 Artificial pandemic
 Extreme climate change
 Bad outcomes from artificial intelligence
 Some threaten extinction directly, some indirect
What are some existential risks?
 Natural
 Pandemic
 Asteroid strike
 Anthropogenic
 Nuclear war
 Artificial pandemic
 Extreme climate change
 Bad outcomes from artificial intelligence
 Some threaten extinction directly, some indirect
Why care about existential risks?
 Moral case
 Billions of lives at stake – and far more future
people.
 First period in history where we may have
power to affect continuation of humanity.
Why care about existential risks?
 Moral case
 Billions of lives at stake – and far more future
people.
 First period in history where we may have
power to affect continuation of humanity.
 Existential risk reduction is under-supplied
 Global, intergenerational public good
 Unprecedented and uncertain
So what can we do?
So what can we do?
So what can we do?
So what can we do?
Concrete objectives
are specific
programmes which
can be realised in
the medium term
and lead towards a
reduction in
existential risk.
Strategic objectives
are desirable states of
affairs which, were
they achieved, would
reduce existential
risk.
Where do we stand?
Naive
Current
Goal
Where do we stand?
Naive
Current
Goal
All answers are provisional!
Strategic objectives – desiderata
 Lead to lower existential risk
 Specific enough to target and assess
 Plausibly neglected or high-leverage
Strategic objectives – process
 Internal meetings at the Future of Humanity
Institute and Global Priorities Project
 Brainstormed strategic objectives
 Comments and group discussions
 Individual assessment
 Aggregation
 Order-of-magnitude estimates of value
Mechanisms for strategic objectives
Mechanisms for strategic objectives
Strategic objectives overview
• Accountability to future generations
• Transparency in development of risky technologies
Prevention
• Agreement on liability for catastrophe
• International regulation of biotechnologies
• Reduced capacity for nuclear conflict
• Agreements on stockpiling of food
Response
• International pandemic scenario planning
• Agreements to intervene on tail-risk climate change
Recovery
Capacity-building
• Secure storage for key knowledge and capabilities
• Increased research into alternate food
• Increased research on existential risks
• Increased attention on existential risks within governments
Strategic objectives – Prevention
 Governance mechanisms for accountability to
rights of future generations
 International transparency on development of
offensive and dual-use technologies
 International agreement on liability in case of
global catastrophe
 Monitoring and international regulation of
dangerous biotechnologies
 Reduced capacity for accidental or intentional
nuclear conflict
Strategic objectives – Prevention
 Governance mechanisms for accountability to
rights of future generations
 Helps correct incentives for intergenerational public
goods.
 Useful at national or international levels.
 Value in precedent-setting, and experimenting with
governance structures.
Strategic objectives – Prevention
 International transparency on development of
offensive and dual-use technologies
 Decreases likelihood of arms races, and decreases
incentives to unilaterally pursue risky technologies for
state or non-state actors.
 Could be via agreements on international monitoring
and surveillance.
 Could be by credible commitment to rewarding
whistleblowing.
Strategic objectives – Prevention
 International agreement on liability in case of
global catastrophe
 Explicit liability in case of catastrophe caused by risky
activity would help to correct incentives for global
public good of risk reduction.
 Making nations liable for catastrophic activity within
their borders would help incentivise them to clamp
down on it.
 Could be pursued in specific domains (e.g.
biotechnology)
Strategic objectives – Prevention
 Monitoring and international regulation of
dangerous biotechnologies
 We already have capabilities to produce potentially
pandemic pathogens.
 Trend is towards capabilities growing more powerful
and more accessible.
 International agreement on how to monitor and
regulate technology would prevent tragedy of the
commons or unilateralist’s curse scenarios.
Strategic objectives – Prevention
 Reduced capacity for accidental or intentional
nuclear conflict
 Nuclear war remains one of the largest risks of global
catastrophe in the coming decade.
 Further reduction in stockpiles could reduce existential
risk without threatening national security.
 Further safeguards to prevent unintentional or
mistaken launch extremely valuable.
Strategic objectives – Prevention
 Governance mechanisms for accountability to
rights of future generations
 International transparency on development of
offensive and dual-use technologies
 International agreement on liability in case of
global catastrophe
 Monitoring and international regulation of
dangerous biotechnologies
 Reduced capacity for accidental or intentional
nuclear conflict
Strategic objectives – Response
 Agreements on stockpiling and distribution of
food or other essentials for catastrophic scenarios
 International pandemic scenario planning,
including outbreaks of engineered pathogens
 International agreements for decisive intervention
in case of tail-risk climate change
Strategic objectives – Response
 Agreements on stockpiling and distribution of
food or other essentials for catastrophic scenarios
 Most risk from asteroids, supervolcanoes, or nuclear
war comes from atmospheric effects preventing food
growth.
 Stockpiling food could reduce deaths directly, but also
reduce likelihood of large-scale unrest.
 Better supply chains and the end of the cold war have
led to reductions in existing stockpiles of food.
Strategic objectives – Response
 International pandemic scenario planning,
including outbreaks of engineered pathogens
 Pandemics present a window-of-opportunity, where
intervention early on can be much more valuable than
the same intervention later.
 Explicit planning for the worst-case scenarios could
enable fast high-quality responses.
 Pandemics will quickly be international, so
international coordination in responding valuable.
Strategic objectives – Response
 International agreements for decisive intervention
in case of tail-risk climate change
 There is significant uncertainty in climate models, and
experts suggest a non-negligible chance of > 6°C
warming, which could precipitate catastrophic effects.
 If we discover we are headed for such a scenario,
dramatic geoengineering could be crucial, but it is still
poorly understood and may pose risks of its own.
 Agreement on when to and when not to use
geoengineering ahead of time could help timely
decision-making.
Strategic objectives – Response
 Agreements on stockpiling of food and other
essentials for catastrophic scenarios
 International pandemic scenario planning,
including outbreaks of engineered pathogens
 International agreements for dramatic
intervention in case of tail-risk climate change
Strategic objectives – Recovery
 Secure storage for key knowledge and capabilities
to rebuild after massive catastrophe
 Increased research and development into
alternate food sources
Strategic objectives – Recovery
 Secure storage for key knowledge and capabilities
to rebuild after massive catastrophe
 Many global catastrophes do not pose a direct
existential risk, but could be one if civilisation collapses
and does not recover.
 Providing key knowledge or capabilities (such as
Svalbard Global Seed Vault) could facilitate recovery.
 Challenges in identifying what is actually key, and in
preserving it past a catastrophe that destroys other
stores of the knowledge.
Strategic objectives – Recovery
 Increased research and development into
alternate food sources
 Some catastrophes could impact long-term agricultural
prospects. To reduce risk of mass starvation, and
increase capability to rebuild, technology which allowed
production of non-traditional food could be very
helpful.
 There has been some preliminary research into this, but
very little has been thoroughly explored.
 [This helps with both Response and Recovery]
Strategic objectives – Recovery
 Secure storage for key knowledge and capabilities
to rebuild after massive catastrophe
 Increased research and development into
alternate food sources
Strategic objectives – Capacity-building
 Increased funding for research to understand
existential risks
 Increased attention on existential risks within
governments and other decision-making bodies
Strategic objectives – Capacity-building
 Increased funding for research to understand
existential risks
 Massive uncertainty around existential risks, but with
some exceptions they receive relatively little study.
 Further study could reveal which posited threats are
real and significant, and which could be ignored.
 Research could also suggest better interventions.
Strategic objectives – Capacity-building
 Increased attention on existential risks within
governments and other decision-making bodies
 Existential risk reduction is a global public good and
naturally provided by large bodies rather than
individuals.
 Some risks may present key moments for intervention.
Better chances of good outcomes if decision-makers are
already aware of and engaged with the issues.
 Could include establishing small dedicated groups
within governments, or improving fora for discussion of
the issues.
Strategic objectives – Capacity-building
 Increased funding for research to understand
existential risks
 Increased attention on existential risks within
governments and other decision-making bodies
Strategic objectives overview
• Accountability to future generations
Medium
• Transparency in development of risky technologies
Specialised
Prevention • Agreement on liability for catastrophe
Response
Recovery
Capacitybuilding
Specialised
• International regulation of biotechnologies
Medium
• Reduced capacity for nuclear conflict
Medium
• Agreements on stockpiling of food
Specialised
• International pandemic scenario planning
Medium
• Agreements to intervene on tail-risk climate change
Specialised
• Secure storage for key knowledge and capabilities
Specialised
• Increased research into alternate food
Medium
• Increased research on existential risks
Broad
• Increased attention on existential risks within governments
Broad
Strategic objectives – Discussion
For questions or comments on:
1. General approach, Strategic/Concrete split
2. The strategic objectives we included
3. Any strategic objectives we didn’t include
Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Brainstorming concrete objectives
11:30-12:30
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
Concrete objectives are specific
programmes which can be realised in the
medium term and lead towards a reduction
in existential risk.
Strategic objectives are desirable states of
affairs which, were they achieved, would
reduce existential risk.
Example concrete objectives
 Extending the idea of crimes against
humanity to include crimes against future
generations.
 Developing a UN forecasting body
specifically tasked with identifying
existential risks.
 Creating an international body tasked
with knowledge sharing in risk reduction
strategies.
When identifying concrete objectives think…
 Which existential risks does this
address?
 Which strategic objectives does this
address?
 Which tools or institutions would this
rely on?
At this stage we are not filtering by…




Ease of implementation
Plausibility of support/consensus
Funding constraints
Existing expertise
Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Reviewing concrete objectives
1:30-3:00
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
Reviewing concrete objectives
 In small groups (~30 mins):
 Consider the other group’s concrete
objectives.
 Do they inspire new concrete
objectives?
 How could they be improved?
 In whole group (~60 mins):
 Consolidate into a single list and
discuss
Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Fleshing out concrete objectives
3:00-4:00
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
Fleshing out concrete objectives
 Identify mechanisms that could be used
to pursue the objective.
 Who are the key stakeholders?
 Who could lead this process?
 Approximately what time-line could
you imagine?
 What similar processes have happened
before or are happening now?
Existential Risk – Diplomacy and
Governance
Monday 8th February 2016
Prioritising by impact
4:20-6:00
Schedule for the day
Time
Activity
9:30-10:30
Plan for the day and introductions
10:30-11:30
Establishing shared knowledge
11:30-12:30
Brainstorming concrete objectives
12:30-1:30
Lunch
1:30-3:00
Reviewing concrete objectives
3:00-4:00
Fleshing out concrete objectives
4:00-4:20
Break and refreshments
4:20-6:00
Prioritising by impact
6:00
Dinner
Prioritising by impact
 Aim of this session is to assess concrete objectives
by how good they would be, if achieved.
 Tomorrow we will consider how tractable they
are, and integrate into an overall assessment.
 We provide a skeleton framework as a startingpoint for evaluating the impact of concrete
objectives.
Framework for evaluating impact
1. Identify the strategic objectives that would be
helped by this concrete objective.
2. Estimate how much the concrete objective would
help on the strategic objective.
3. Estimate how good scaling up the strategic
objective would be (estimates already exist).
4. Combine estimates, and consider carefully.
Estimating how much things help
 Extremely uncertain and imprecise.
 We think explicit estimates are useful to counter
scope-insensitivity.
 We provide order-of-magnitude bands to help this.
Bands for estimating impact of Concrete
objectives on Strategic objectives
Band
Represents
Surprising if more Effect on Strategic
often than
objective
Massive step
Huge
breakthrough
Once-in-a-decade Doubles progress
Significant step
Major success
Once-in-a-year
+10% progress
Small step
Minor success
Once-a-month
+1% progress
Tiny step
Non-negligible
help
Twice-a-week
+0.1% progress
1. Estimate bands independently and discuss
2. Modify up one band if strategic objective is broad,
down one band if strategic objective is specialised.