Transcript Addressing Perceptions of Case-Based Reasoning

1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Addressing Perceptions of Case-Based Reasoning
David W. Aha
Head, Adaptive Systems Section
Navy Center for Applied Research in AI
Naval Research Laboratory, Code 5514
Washington, DC
[email protected]
Invited Talk
2007 International Conference on Case-Based Reasoning
13 August 2007
Belfast,
Northern
Ireland  13 August 2007  Belfast, Northern Ireland
Addressing Perceptions of Case-Based Reasoning
 David
W. Aha  ICCBR-07
1
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Goals of this presentation
1. Raise awareness on how to assess CBR R&D methods
2. Assess CBR R&D methods we’re publishing
3. Relate CBR’s R&D methods to those used in AI
4. Beg for your forgiveness?
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
2
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1.Perceptions
2.Objectives
3.Survey
4.Findings
5.Interpretation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
3
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1. Perceptions
• Story: Gnats, envy, & self-doubt
• Quest
2.Objectives
3.Survey
4.Findings
5.Interpretation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
4
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
What perceptions of case-based reasoning (CBR) exist?
• Among active CBR researchers/practitioners
• Among others
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
5
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
My perception
…
Artificial Intelligence
…
Case-Based Reasoning
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
6
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Gnats, envy, & self-doubt
Gnat
UK Gnat
Observation: CBR perceived differently by others
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
7
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Gnats
In CBR (Pal & Shiu, 2004; Kolodner, 1993) expertise is embodied
in a library of past cases… <long, accurate description of CBR>
The major problem with CBR is that it lacks a sound
theoretical framework for its application and has only
achieved limited success.
- Anonymous senior AI researcher/proposer, 2005
“Case based reasoning is often limited to surface features
that may not be relevant to the operational military
situation. (There is a need for deeper underlying reasoning,
including analogical reasoning.)”
- Anonymous ONR Program Manager, 2007
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
8
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Gnats
Artificial Intelligence: CBR not taught?
• AIMA (Russell & Norvig, 2002-)
– 90% market share (1000+ universities, 91 countries)
• CBR: Not discussed
• IBL (3 pages)  Statistical_Learning
ML: Prevailing view is CBR = Instance-based learning  ML
• No (e.g., 61% of papers in ECCBR-06 not related to ML)
• Yet there is a relationship
– e.g., “CBR is a technique within the field of machine learning…”
(Beltrán-Ferruz et al., ECCBR-06)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
9
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Gnats
Why isn’t some AI-related CBR research published at IC/ECCBR?
Computational analogy
– “CBR systems…tend to use only minimal first-principles
reasoning…[and] rely on feature-based descriptions…[or] use
domain-specific and task-specific similarity metrics. This can be
fine for a specific application, but being able to exploit similarity
computations that are more like what people do could make such
systems…more understandable to their human partners.” (Forbus et
al., IAAI-02)
Episodic memory
– “Episodic memory can be thought of as the mother-of-all CBR
problems – how to store and retrieve cases about everything
relevant in an entity’s existence. Most CBR research has avoided
these issues.” (Nuxoll & Laird, ICCM-04)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
10
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Gnats
Some Stereotypical perceptions of CBR
USA funding agencies
Of questionable merit (?)
Cognitive Architectures
Informal/incomplete models of episodic memory
Cognitive Psychology
Cognitively implausible exemplar models
Artificial Intelligence
A subfield, once dominated by speculative
evaluation methodologies (Hall & Kibler, AIM 1985)
Machine Learning
Case-based algorithms for supervised learning
Statistics
A target?
Knowledge Management A panacea (still true?)
Business
A mysterious technique whose name is rarely
mentioned by its practitioners (still true?)
Us
A discipline worthy of research & application
How could any misperceptions be addressed?
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
11
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Not interested in giving you yet another content survey
We have existing surveys of CBR (e.g., KER 2005 special issue)
#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Grouping
Introduction
Techniques
Task Areas
Topic Areas
Applications
Title (CBR = "Case-Based Reasoning")
CBR commentaries: Introduction
CBR foundations
Representation in CBR
Retrieval, reuse, revision, and retention in CBR
Integrations
Advances in conversational CBR
Textual CBR
Distributed CBR
Soft CBR
Design, innovation, and CBR
CBR for diagnosis applications
Case-based planning
Medical applications in CBR
CBR and law
CBR-inspired approaches to education
Knowledge management in CBR
Image processing in CBR
Case-based recommender systems
Fielded applications of CBR
Emergent CBR applications
Authors
Aha, Marling, & Watson
Richter & Aamodt
Bergmann, Kolodner, & Plaza
López de Mántaras et al. (13 authors)
Marling, Rissland, & Aamodt
Aha, McSherry, & Yang
Weber, Ashley, & Brüninghaus
Plaza & McGinty
Cheetham, Shiu, & Weber
Goel & Craw
Goker, Howlett, & Price
Cox, Muñoz-Avila, & Bergmann
Holt, Bichindaritz, Schmidt, & Perner
Rissland, Ashley, & Branting
Kolodner, Cox, & González-Calero
Althoff & Weber
Perner, Holt, & Richter
Bridge, Goker, McGinty, & Smyth
Cheetham & Watson
López de Mántaras, Perner, & Cunningham
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
12
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Envy
Am I (unnecessarily) wishing for something?
Formal foundations envy?
• e.g., Bayesian, first-order logic, decision theory, COLT, …
• But we have this:
– e.g., Cover & Hart, 1967; Richter, FLAIRS-07; Richter & Aamodt, 2005 KER
– And we’re the ultimate chameleons, even within AI
Methodological approach envy?
• e.g., Experimental study of ML (Langley & Kibler, 1991), Crafting
papers on ML (Langley, ICML-00), …
• Possibly:
– We haven’t had received much proselytizing on this…yet
– My awareness of these issues has increased; worth reviewing
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
13
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Envy
Crafting papers (e.g., on ML) (Langley, ICML-00)
• Content
• Evaluation strategy
• Communication
Paper content recommendations
•State the research goals and evaluation criteria
•Specify the component (e.g., learning) & overall perf. task
•Describe rep’n and organization of knowledge & data
•Explain the system components (if any)
•Evaluate the approach
–Empirical, theoretical, psychological, novel functionality
•Describe related work
–Explain similarities/differences with your work
•State the limitations
–Propose solutions
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
14
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Self-Doubt
Summary questions
• How should (royal) we respond to possible
misperceptions of CBR?
– i.e., Other than to survey the field’s contents and its foundations
• Why are some folks ignoring CBR?
• How can we attract them?
• Does this concern our research methodologies (and/or
their communication) rather than our research focus?
Proposal: Examine our research methodologies
Realization: This requires a framework for investigation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
15
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Quest: Identify, characterize, & compare CBR research methods
Don Quixote (Scott Gustafson)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
16
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1.Perceptions
2. Objectives
• Questions
• Conjectures/Hypotheses
3.Survey
4.Findings
5.Interpretation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
17
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Questions
1. How should we describe CBR to others?
• i.e., in the context of AI
2. What R&D methodologies are we using?
3. Does CBR R&D differ from AI R&D?
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
18
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Conjectures/Hypotheses
 AI research is dominated by two methodologies (Cohen, 1991)
• Model-centered (neat) (i.e., proving theorems on formal models)
• System-centered (scruffy)
1. CBR research is not (currently) dominated by both
• Dominated only by system-centered papers, which often lack models
for deriving claims, generating predictions, and explaining behavior
2. CBR research suffers from similar methodological problems
• Model- and system-centered papers differ in whether they conduct
evaluations, assess performance, and describe expectations
3. The designation of CBR conference publications are
distinguished by their research methodologies
• Oral vs. poster presentations
• Best paper nominees from others
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
19
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1.Perceptions
2.Objectives
3. Survey
•
•
•
•
Case base
Retrieval
Reuse
Revision
4.Findings
5.Interpretation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
20
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Case Base
Frameworks for assessing AI R&D Methods
Read 150
Papers!
(can you imagine?)
Case #1
Cohen, Paul R. (1991). A Survey of the Eighth National
Conference on Artificial Intelligence: Pulling together or
pulling apart? AI Magazine, 12(1), 16-41.
Paul R. Cohen
(circa ~1991)
Summary of (Cohen, 1991)
Paul R. Cohen
(circa ~2007)
• Conclusion: AI research follows two incomplete, complementary methodologies
• Proposes: MAD (Modelling, Analysis, & Design) mixed methodology
Recommendation: Make this required reading for AI researchers
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
21
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
(Cohen, 1991): 40 citations (Google Scholar, 8/1/07)
Many of our suggestions are similar to the excellent points made by Cohen (1991) in his discussion of AI, but they
seem worth instantiating for the field of machine learning (Langley & Kibler, 1991 “Experimental Study of ML”)
There are two ways in which the fields proceed. One is through the development and synthesis of models of
aspects of perception, intelligence, or action, and the other is through the construction of demonstration systems
(Brooks, 1991 Science).
As Cohen (1991) demonstrated in his analysis of the papers presented at AAAI90, we are, as a discipline, just
learning how to perform real, systematic experimentation. One hears a lot of talk about AI as an experimental
science, but typically the “experiments" amount merely to writing a computer program that is supposed to validate
some hypothesis by its very existence. (Pollock, 1992 Artificial Intelligence)
The importance of this link has been highlighted by several researchers, some even going so far as to state that
AI will not advance as a science until the gap between those who construct models and those who build
systems is closed. (Jennings, 1995 Artificial Intelligence).
Cohen (1991) discovered that only 43% of the papers that described implemented systems report any kind of
analysis of their contributions. Even of the papers that do describe evaluatory experiments, very few go beyond
evaluating the programs to analyzing the scientific claims that the programs were written to demonstrate.
(Ram & Jones, 1995 Philosophical Psychology)
Methodological issues are by no means resolved (Cohen, 1991), but they are much discussed and a consensus
is emerging on the importance of combining theoretical and empirical investigations. (Bundy, 1998 book)
As Cohen (1991) points out, most research papers in AI, or at least at an AAAI conference, exploit benchmark
problems; yet few of them relate the benchmarks to target tasks. (Howe & Dahlman, 2002 JAIR)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
22
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Retrieval
My query:
• Identify R&D methodologies being used in CBR
• Compare results with general AI and other AI subfields
Case #1: (Cohen, 1991)
• Develop and apply framework for analyzing AI R&D methodologies
• Identify R&D methodologies being used
• Propose novel R&D methodology (MAD)
Framework for assessing AI R&D Methods
My Query
Case #1
Cohen, Paul R. (1991). A Survey of the Eighth National
Conference on Artificial Intelligence: Pulling together or
pulling apart? AI Magazine, 12(1), 16-41.
1. Retrieve
Case #1
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
23
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
MAD Framework (Cohen, 1991)
3. Models (Define, extend, generalize, provide semantics)
4. Theorems/Proofs for models
Purpose
5. Present algorithm(s)
Complexity
Formal
Informal
Complexity
Formal
Informal
9. Example task
Natural
Synthetic
Abstract
10. Task type
Natural
Synthetic
Abstract
Embedded
Not embedded
6. Analyze algorithm(s)
7. Present system
8. Analyze aspects of system
11. Task environment
12. Assess performance
Argument
13. Assess coverage
14. Comparison
15. Predictions, hypotheses
16. Probe results
Note: It does not
(completely) eliminate
subjective assessments!
17. Unexpected results
18. Negative results
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
24
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
MAD Framework: Purpose fields
Field
Description
3. Models (Define, extend,
generalize, provide semantics)
• Abstract, typically formal description of behavior and/or
environmental factors that affect behavior.
• Purpose of building a model is to analyze its properties
4. Theorems/Proofs for models
• e.g., Complexity, soundness, completeness, decidability
5. Present algorithm(s)
6. Analyze algorithm(s)
• e.g., Complexity, soundness, completeness, decidability
7. Present system
• Describes components, control flow
8. Analyze aspects of system
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
25
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
MAD Framework: Argument fields
Field
Description
9. Example task
• Natural, synthetic, or abstract
10. Task type
• Used iff multiple trials are described
• Natural, synthetic, or abstract
11. Task environment
• Used iff multiple trials are described
• Embedded or not embedded (e.g., in other s/w, env’t)
12. Assess performance
• Weak criterion: One perf. measure over many examples
• e.g., a bakeoff is an assessment
13. Assess coverage
• Solve instances of some problems in a defined problem space
• Not a demo on superficially different problems w/o justification
14. Comparison
• Goal: Study relative strengths/limitations of multiple techniques
• e.g., a bakeoff is not a comparison
15. Predictions, hypotheses
• Indicate reason to implement/test an idea
• Not “My algorithm solves this problem”, or simple perf. demos
• Many papers are vague as to why empirical work is described
16. Probe results
• Go beyond central results (e.g.,follow-up expt’s, explanations)
17. Unexpected results
• Infrequent
18. Negative results
• Rare (if ever)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
26
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Reuse
My Query
Case Base
(Cohen, 1991)
1. Retrieve
MAD Framework
AAAI-90
AAAI-90
Data
Analyze
Results
MAD
Methodology
Metrics
2. Reuse
MAD Framework
ECCBR-06
Hypotheses
ECCBR-06
Data
Analyze
Adapted
Hypotheses
Results
ICCBR
Audience
(at lunch)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
27
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06: 36 Papers
# First Author
1 Gabel
2 Ros
3 Watson
4 Karoui
5 McDonnell
6 Weber
7 Gu
8 Nicholson
9 Hervas
10 Gupta
11 Minor
12 McCarthy
13 Kofod-Petersen
14 Recio-Garcia
15 Herrera
16 Freyne
17 Bergmann
18 Baccigalupo
19 Perner
20 Gomez-Gauchia
21 Massie
22 Wiratunga
23 Stahl
24 Coyle
25 Bogaerts
26 Chakraborti
27 Lamontagne
28 Althoff
29 Beltran-Ferruz
30 Kuchibatla
31 Funk
32 Montani
33 Mendez
34 Bergmann
35 Hefke
36 Goker
Title
Multi-agent CBR for cooperative RLs
Retrieving and reusing game plays for robot soccer
Self-organizing hierarchical retrieval in a case-agent system
COBRAS; Cooperative CBR system for bilbliographic reference recommendation
A knowledge-light approach to regression using CBR
CBM for CCBR-based process evolution
Evaluating CBR sytems using different data sources: A case study
Decision diagrams: Fast and flexible support for case retrieval and recommendation
CBR for knowledge-intensive template selection during text generation
Rough set feature selection algorithms for textual case-based classification
Experience management with case-based assistant systems
The needs of the many: A case-based group recommender system
Contextualised ambient intelligence through CBR
Improving annotation in the semantic web and case authoring in textual CBR
Unsupervised case memory organization: Analysing computational time and soft computing capabilities
Further expeirments in case-based collaborative web search
Finding similar deductive consequences: A new search-based framework for unified reasoning from cases and general knowledg
Case-based sequential ordering of songs for playlist recommendation
A comparative study of catalogue-based classification
Ontology-driven development of conversational CBR systems
Complexity profiling for informed case-base editing
Unsupervised feature selection for text data
Combining case-based and similarity-based product recommendation
On the use of selective ensembles for relevance classification in case-based web search
What evaluation criteria are right for CCBR? Considering rank quality
Fast case retrieval nets for textual data
Combining multiple similarity metrics using a multicriteria approach
Case factory: Maintaining experience to learn
Retrieval over conceptual structures
An analysis on transformational analogy: General framework and complexity
Discovering knowledge about key sequences for indexing time series cases in medical applications
CBR for autonomous service failure diagnosis and remediation in software systems
Tracking concept drift at feature selection stage in SpamHunting: An anti-spam instance-based reasoning system
Case-based support for collaborative business
A CBR-based approach for supporting consulting agencies in successfully accompanying a customer's introduction of KM
The PwC connection machine: An adaptive expertise provider
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
28
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1.Perceptions
2.Objectives
3.Survey
4. Findings
• Results
• Analysis & Patterns
• Followup
5.Interpretation
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
29
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Findings: Results
1
#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
3
Informal
Model to
frame the
research
4
Define, Extend,
Generalize,
Differentiate,
Theorems
Semantics for and proofs
Formal Models re: Model
1
1
1
1
1
5
6
7
Analyze Algorithms
Present
Algorithms Complexity
1
1
Formal
Informal
1
1
1
1
1
1
Analyze aspect(s) of system
Example type
1
1
Formal
Informal
Natural
Synthetic
Task
Abstract
Natural
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
9
Present
system Complexity
1
1
1
1
1
1
1
1
1
8
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
1
30
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Findings: Examples
Field
#
Example
3. Models (Define, extend, etc.)
9 Similarity/deductive reasoning (Bergmann & Mougouie)
4. Theorems/Proofs for models
0
Purpose
5. Present algorithm(s)
6. Analyze algorithm(s)
7. Present system
8. Analyze aspects of system
21 Retrieve k cases from a DD (Nicholson et al.)
7 Unsupervised algs. (Fornells Herera et al.)
21 PwC Connection Machine (Göker et al.)
7 Fast CRNs (Chakraborti et al.)
9. Example task
22 Song playlists (Baccigalupo & Plaza)
10. Task type
26 European skiing holidays (McCarthy et al.)
11. Task environment
Argument
12. Assess performance
13. Assess coverage
4 Game plays for robot soccer (Ros et al.)
27 Integrated CCBR evaluation (Gu & Aamodt)
1 Complexity profiling (Massie et al.)
14. Comparison
15 SpamHunting (Méndez Reboredo et al.)
15. Predictions, hypotheses
19 Rough set feature selection (Gupta et al.)
16. Probe results
7 Ave. diversity of decision trees (Coyle & Smyth)
17. Unexpected results
6 Pima’s precision results (Bogaerts & Leake)
18. Negative results
0
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
31
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Categorizing Papers: AAAI-90 (Cohen, 1991)
Field
Papers
3. Models (Define, extend, etc.)
4. Theorems/Proofs for models
5. Present algorithm(s)
6. Analyze algorithm(s)
7. Present system
Models
(3  4)
Algs
(5  6)
8. Analyze aspects of system
Categories
Model-Centered: (M  A) S
Hybrid: (M  A)  S
System-Centered: (M  A)  S
Systems
(7  8)
AAAI-90
25
Models
43
1
4
36
Algs
3
37
Systems
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
32
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Categorizing Papers: ECCBR-06
Field
Papers
3. Models (Define, extend, etc.)
4. Theorems/Proofs for models
5. Present algorithm(s)
6. Analyze algorithm(s)
7. Present system
Models
(3  4)
Algs
(5  6)
8. Analyze aspects of system
Categories
Model-Centered: (M  A) S
Hybrid: (M  A)  S
System-Centered: (M  A)  S
Systems
(7  8)
ECCBR-06
1
Models
5
0
3
7
Algs
6
13
Systems
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
33
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Comparing MAD Categorizations of Papers
50% M AAAI-90
ECCBR-06 25% M
25
Models
43
4
1
36
3
Algs
1
Models
5
6% Hybrid
17%
29%
Models
1%
3%
Algs
3%
14%
Models
0%
8%
Algs
6
13
Systems
24%
2%
3
0
37
Systems
7
25% Hybrid
19%
17%
25%
36%
Systems
Systems
Algs
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
34
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Results: ECCBR-06
Legend
A=Algorithms
M=Models
S=Systems
A
A
A
Field 9:
A
Example
B
B
Field 10: Eval B
B
Task
C
Field 11:
C
Embedded
C
Task
Fields 12-14: D
D
Demo
Fields 15-18: E
E
Followup
Distribution by fields 3-8
Natural
Synthetic
Abstract
None
Natural
Synthetic
Abstract
None
Embedded
Not embedded
None
Demo
No demo
Expectations
No expectations
Contingency table
M
1
1
0
0
0
0
0
0
1
0
0
1
1
0
0
1
Model-Centered
M+A
A
5
7
2
3
0
1
2
0
1
3
4
6
0
1
1
0
0
0
0
1
5
6
0
0
5
7
0
0
4
4
1
3
M+S
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Hybrid
M+S+A
3
1
0
0
2
2
0
0
1
2
0
1
2
1
1
2
S+A
6
2
1
1
2
4
1
0
1
1
3
2
5
1
4
2
System-Centered
S
13
7
1
0
5
6
0
0
7
0
6
7
6
7
6
7
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
Totals
35
16
3
3
13
22
2
1
10
4
20
11
26
9
19
16
35
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Results: ECCBR-06
Legend
M-C: Model-Centered
H: Hybrid
S-C: System-Centered
A
A
A
Field 9:
A
Example
B
B
Field 10: Eval B
B
Task
C
Field 11:
C
Embedded
C
Task
Fields 12-14: D
D
Demo
Fields 15-18: E
E
Followup
M-C
Distribution by fields 3-8
Natural
Synthetic
Abstract
None
Natural
Synthetic
Abstract
None
Embedded
Not embedded
None
Demo
No demo
Expectations
No expectations
H
13
6
1
2
4
10
1
1
1
1
11
1
13
0
8
5
S-C
9
3
1
1
4
6
1
0
2
3
3
3
7
2
5
4
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
13
7
1
0
5
6
0
0
7
0
6
7
6
7
6
7
36
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Analysis: Comparing ECCBR-06 with AAAI-90

Source
ECCBR-06
AAAI-90
Model-Centered Hybrid System-Centered
13
9
13
104
8
37
χ2(2)=19.0, p<0.0001
Could this distribution of M-C, Hybrid, and S-C methodologies have arisen
by chance, or does it reflect a real difference between ECCBR and AAAI?
• The ECCBR/AAAI distinction is not independent of the research
methodology class

Source
ECCBR-06
AAAI-90
Model-Centered
Hybrid
System-Centered
M
M+A
A M+S M+S+A S+A
S
1
5 7
0
3
6
13
25
43 36
1
4
3
37
χ2(6)=24.1, p<0.006
Legend
A=Algorithms
M=Models
S=Systems
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
37
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Analysis: Examining ECCBR-06
M-C
Distribution by fields 3-8
Any Example
No Example
H
13
9
4
S-C
9
5
4
13
8
5
χ2(2)=0.4, p>0.8
 Unlike AAAI-90, the methodological choice of an example is
independent of the paper’s class.
M-C
H
S-C
Distribution by fields 3-8
13
9
13
Any evaluation
No evaluation
12
1
7
2
6
7
χ2(2)=7.0, p<0.003
 The methodological choice of whether an evaluation was
conducted is not independent of the paper’s class.
• Model-centered and hybrid papers include evaluations
significantly more frequently than do system-centered papers.
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
38
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Analysis: Examining ECCBR-06 (cont.)
M-C
Distribution by fields 3-8
Demonstration
No demonstration
H
13
13
0
S-C
9
7
2
13
6
7
χ2(2)=9.9, p<0.007
 Like AAAI-90, Model-centered and hybrid papers are more likely
than system-centered papers to include (any type of)
performance assessment.
M-C
Distribution by fields 3-8
Expectations
No expectations
H
13
8
5
S-C
9
5
4
13
6
7
χ2(2)=0.6, p>0.7
 Surprisingly, and unlike AAAI-90, model-centered and hybrid
papers do not provide (any types of) expectations more frequently
than do system-centered papers.
• Perhaps this warrants a follow-up analysis
• Perhaps system-centered researchers make predictions not
derived from models, which would be dangerous, or perhaps
they are simply not stating the models, which is more likely.
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
39
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Patterns: Comparing ECCBR-06 with AAAI-90
Pattern
Observed frequencies
AAAI-90
ECCBR-06
p(M)
0.49
0.25
p(S)
0.30
0.61
p(SM)
0.03
0.08
p(SM)
0.89
0.86
p(No or abstract examples | M-C)
0.76
0.46
p(Test implementations | M-C)
0.33
0.92
p(Prediction/hypothesis)
<0.21
0.53
p(Evaluation)
0.30
0.72
p(prediction/hypothesis | evaluation)
p(Negative, surprising, or probe)
0.16
Generous?
0.69
Generous!
0.25
Näive
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
40
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06: Distinguishing Papers from Posters
“Now we tread on hallowed ground” - Anon
Hypothesis: Reviewers are human and subjective. While
there’s probably a trend that oral presentations show more
“maturity” than do posters, exceptions exist and this trend is
probably not significant.
Results of analysis: I was wrong…
• …assuming the presentation/use of models is
indicative of a paper’s level of maturity
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
41
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06
50% M Oral Papers (22)
1
Models
5
2
0
14% Hybrid
Posters (13) 15% M
5
1
Algs
0
Models
0
5%
23%
Models
0
9%
Algs
0%
0%
Models
0%
8%
Algs
5
5
Systems
23%
5%
1
0
8
Systems
2
46% Hybrid
15%
38%
36%
38%
Systems
Systems
Algs
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
42
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06: Distinguishing Papers from Posters
Model-Centered
Hybrid
System-Centered
M
M+A
A
M+S M+S+A S+A
S
Oral
1
5
5
0
2
1
8
Poster
0
0
2
0
1
5
5
2(5)=9.3, p<0.1
 The poster/paper designation of an accepted paper at
ECCBR-06 was not independent of the paper’s class.
• Tentative conclusion: If you want your accepted
paper to be an oral presentation, then present your
work in the context of a model.

Distribution by fields 3-8
Oral
Poster
M-C
11
2
H
S-C
3
8
6
5
2(2)=6.0, p<0.05
So: Will you think about this, and want to learn more?
But maybe you are unconvinced…
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
43
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06: Distinguishing Papers from Posters




Example
None
Oral Poster
14
8
8
5
2(1)=0.02, p>0.9
Eval
None
Oral Poster
16
9
6
4
2(1)=0.05, p>0.8
Demo
None
Oral Poster
17
9
5
4
2(1)=0.28, p>0.5
Demo
None
Oral Poster
11
8
11
5
2(1)=0.44, p>0.5
Nothing else (so far) distinguishes papers from posters
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
44
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
ECCBR-06: Distinguishing Best Paper Nominees?
• But there were only 5
• Future work: Analyze after adding 9 ICCBR-07 nominees
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
45
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Summary
Hypotheses revisited
1. Unlike AAAI-90, CBR research is not dominated by both
model-centered and system-centered methodologies
 Dominated only by system-centered papers
2. CBR research suffers from similar methodological problems
• Model- and system-centered papers differ in whether they:
 Conduct evaluations
 Assess performance
 Describe expectations
3. The class of a paper in the MAD framework distinguishes
 Oral vs. poster presentations
 Best paper nominees from others
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
46
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Outline
1.Perceptions
2.Objectives
3.Survey
4.Findings
5. Interpretation
• A new case
• Caveats
• Next steps
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
47
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
A new case…hopefully
How can we assess CBR
R&D Methodologies?
Case Base
Frameworks for assessing
AI R&D Methods
(Aha, 2007?) (Cohen, 1991)
1. Retrieve
MAD Framework
AAAI-90
2. Reuse
MAD Framework
4. Retain
ECCBR-06
MAD mixed
methodology
Today’s
Results
3. Revise
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
48
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Many of Cohen’s (1991) Points for AI Apply to CBR
• A goal of AI research is to develop science & technology to
support the design and analysis of intelligent systems
• Model- and system-centered methods are complementary
– Model-centered researchers typically develop algorithms for simpler
problems, but with deeper analysis, expectations, and demos
– System-centered researchers typically build large systems to solve
realistic problems, but w/o explicit expectations, analyses or demos
– See the MAD methodology (Cohen, 1991)
•
•
•
•
Models are used to derive hypotheses & expectations
Few systems merit attention on the basis of existence alone
It is impossible to evaluate a system without predictions
Creating benchmarks will not fix AI’s methodological problems
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
49
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Caveat #1: No cases for ECCBR-90, AAAI-06, etc.
AAAI-90
?!
ECCBR-06
Note: CBRW-91’s R&D methods differ greatly from ECCBR-06’s
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
50
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Caveats (that could influence the results)
• The case base is small to compare CBR with AI
– AAAI has changed since 1990! Compare to AAAI-07 (and IAAI-07!)
– ECCBR’s differences may reflect a higher acceptance rate
– No analysis of other subfields; how does CBR relate?
• No reliability data: Subjective classification of papers!
– e.g., I gave up distinguishing “informal” and “no” system analysis
– “Volunteers welcome!” ((Cohen, 1991), which I repeat)
• Not representative of CBR community’s work?
– Is ECCBR-06 an aberration? (Wait until next year?)
– Perhaps we publish model-centered work elsewhere (e.g., COLT)
– ECCBR readers’ expectations match ECCBR-06’s class distribution?
• Lack of page-space places limitations on what is presented
– e.g., hypotheses not made explicit
• Science/research works iteratively
– Earlier exploratory research (e.g., involving surprising results) resulted in
changes to the model, algorithm, or system; we see only the end result
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
51
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Caveat finale: I’ve ignored many issues!
• Relation of evaluation to:
– Investigating the claims, if any
– The predictions, if any
• Results of formal analysis
– e.g., average- vs. worst-case
• Quality of the empirical evaluation (e.g., scale)
• Significance of evaluation’s results
•…
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
52
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Next Steps
Other potential applications of the MAD framework
• A decision aid for predicting whether an accepted paper
should be categorized as an oral presentation
– Or to ensure diversity among the presentations
• Selection of best paper nominees
• To assist reviewers with spotting novelty and/or expected
characteristics in a submission
• Explaining/characterizing CBR methodologies to others
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
53
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Next Steps
Conjecture: CBR has shifted from heavily modelcentered to heavily system-centered research
– Needs analysis to provide evidence, but it’s obvious
– AI needs both to achieve balance
– We should consider this in our reviewing processes
Roger Schank
Milestones: We are halfway to our 25th anniversary
David Leake
– A time for reflection
– We should make it our goal that, by the 25th, we will
achieve a better balance
Agnar Aamodt
(1st ICCBR Co-Chair)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
54
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Next Steps
• Refine our research methodologies
– Ensure program PCs reflects them broadly
– A CBR-related journal could assist (by providing feedback)
• Change perceptions by improving communication
– WWW site (maybe AAAI, but probably not, as we have
different specifications in mind)
– Co-locate our conferences with others
• e.g., IJCAI, ECAI, IR conference
• Reach out in new ways (e.g., Video)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
55
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Next Steps
AAAI-07 AI Video Competition (Co-Chairs: Thrun & Aha)
• Goal: Encourage the interest of prospective students
• Results (see aivideo.org)
– Quick funding
– 30 Submissions (in a short time period)
– Large turnout for awards ceremony
– Invited, and will be held, for AAAI-08
• Two CBR videos
1. k-nearest Neighbor Classification (Antal van den Bosch)
2. Invisible Threats (Rosina Weber & André Testa)
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
56
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Takeaway Message
And now: The 25 Re’s!!!
• e.g., Reuse, Regurgitate, Repulse, …
• Beats previous record of 13 by 12 (Bridge, ICCBR-05)
Derek Bridge
Bridge Gauntlet: 13 Re’s!
Just kidding – I’ll spare you
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
57
1. Perceptions  2. Objectives  3. Survey  4. Findings  5. Interpretation
Concluding remarks
1. Goal: Raise awareness of CBR R&D methodologies
2. Current CBR methods are unlike traditional AI’s
• CBR’s is not dominated by model-centered work
• We must beware system-centered limitations
• But there’s much more to learn
3. Paper/poster distinction relates to use of models
4. MAD framework has several potential uses
5. We have work to do to address perceptions
Thanks for listening
†This
presentation is dedicated to my late colleague John Urban and the late great Donald Michie, for their support.
Addressing Perceptions of Case-Based Reasoning  David W. Aha  ICCBR-07  13 August 2007  Belfast, Northern Ireland
58