Transcript Память - Meetup

Memory and Q&A systems
Review
Vladislav Belyaev
DeepHackLab
22 November 2015
Summary
1.
2.
3.
4.
Memory
Neural nets with memory
Q&A methods & systems
DL and NLP (+ discussion)
Memory
Память — одно из свойств нервной системы, заключающееся в способности какое-то время сохранять
информацию о событиях внешнего мира и реакциях организма на эти события, а также многократно
воспроизводить и изменять эту информацию
Память является неотъемлемой частью таких процессов, как
• обучение
• прогнозирование будущего и воображение несуществующего (повидимому, оба процесса являются процедурами «нарезания и
перетасовки фрагментов воспоминаний») [1]
• сознание и самоидентификация индивидуума
Stages in the formation and retrieval of memory:
• Encoding or registration
• Storage
• Retrieval, recall or recollection
• Forgetfulness
1. Hassabis D, Kumaran D, Vann S.D, Maguire E.A. Patients with hippocampal amnesia cannot imagine new experiences // PNAS
104 (2007) pp.1726-1731
Memory, few facts
Short-term memory:
• 7±2 items (Miller, 1956) and 4–5 items (Cowan, 2001)
• memory capacity can be increased by chunking
ФСБКМСМЧСЕГЭ
ФСБ КМС МЧС ЕГЭ
Long-term memory:
• potentially unlimited duration
• capacity is immeasurable
Making memories can be enhanced by sleep:
1. Acquisition which is the process of storage and retrieval of new information in memory
2. Consolidation
3. Recall
Neural nets with memory
All neural nets have memory. But some nets have more memory than others
Memory mechanism
+
−
Data
Hopfield
Associative (attractors)
Fast and simple learning
Low capacity - 0.15*neurons
No practical use
Images or other objects with
fixed states
NN
Weights
Great for classification and other
relevant tasks
Static systems
Not suitable for sequences
Any static data
RNN
Weights + Recurrence
Can cope with sequences
Vanishing gradients for longterm dependencies
Any sequence
LSTM/GRU
Weights + Recurrence +
Architecture (hidden
states)
No vanishing gradients problem
Can choose what to remember
and forget (language structure)
Hard to recall exact facts or
exact phrase (small memory)
Need a lot of data
Any sequence
MemNN
Weights + Memory +
Architecture (IGOR)
Real memory with raw text
Can generalize + Inference
Work with supervision
Artificial or easy tasks (e.g.
14M sentences)
Text, artificial dataset or
factorial Q&A (one)
NTM
NN controller (read and
write heads) and a memory
bank
Real memory with raw text
Can generalize
Limited memory size (128
locations) Complex model
Sorting, copying and recall
Any sequence
Q&A methods & systems
Q&A mechanism
+
−
Data
IR
Bag of words - statistics
(Whoosh engine
Apache Lucene)
Simple and straightforward relevance
calculations
No inference or reasoning
Any set of documents or other
objects (image, audio, video)
KB
IE: Sematic parsing,
Graph of facts;
Query generator
Inference
Transparency
Complexity limitations –
when many facts/operations
needed + “understanding”?
Yago, Freebase, DBpedia
ARISTO
KB
Inference supporting knowledge base
Transparent
Based and aimed at rules,
features (mostly handcrafted)
Q&A tests for schools
Study guide textbooks
Embedding
Cosine similarity
between Q&A
BiLSTM with attention
Straightforward and fast to implement
No inference or reasoning
“Low” level of reliability
Any questions (e.g.
InsuranceQA DS + word2vec)
Quanta
Dependency-tree
recursive NN
Can handle non-obvious clues
Combine sentential representations
into paragraphs
No inference or reasoning
Need an answer to occur at
least 6 times
Quiz bowl questions (raw text
to entities) + Wikipedia
Neural
Programmer
Select source and
operation in sequence
of steps
Simple arithmetic and logic operations
(complex reasoning)
No rules for program selection
Synthetic dataset of tables
Easy tasks
Triples: a question, a source
and an answer + operations
bank
Q&A main theme
Oleksandr Kolomiyets, Marie-Francine Moens (2011), A survey
on question answering technology from an information retrieval
perspective
Clark P., et. Al (2015), Automatic Construction
of Inference-Supporting Knowledge Bases
ARISTO
1
Specification
2
Extraction
3
Semantic
Interpretation
4
Question
Answering
ARISTO (knowbot)
Hixon B., et. Al (2015), Learning Knowledge Graphs for Question Answering through Conversational Dialog
Embedding
Tan M., et. Al (2015), LSTM-based Deep Learning Models for non-factoid answer selection
Neural Programmer
Le Q., Sutskever I. (2015), Neural
Programmer:
Inducing
Latent
Programs with Gradient Descent
DL and NLP
Yann LeCun: “The next big step for Deep Learning is natural language understanding, which aims to give machines
the power to understand not just individual words but entire sentences and paragraphs.”
Geoff Hinton: “I think that the most exciting areas over the next five years will be really understanding text and
videos. In a few years time we will put [Deep Learning] on a chip that fits into someone’s ear and have an Englishdecoding chip that’s just like a real Babel fish.”
Yoshua Bengio, has increasingly oriented his group’s research toward language, including recent exciting new
developments in neural machine translation systems
Unsolved problems
Applications
Future
Discussion
Bubble?
Unfeasible features
Current trends
Do we need it?
C.D. Manning (2015), Computational Linguistics and Deep Learning, http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
Thank you for attention!
Questions?
Bibliography
Key refs
1. Neural Programmer: Inducing Latent Programs with Gradient Descent (11.2015)
http://arxiv.org/abs/1511.04834
2. LSTM-based Deep Learning Models for non-factoid answer selection (11.2015)
http://arxiv.org/abs/1511.04108
3. Memory Networks (2014)
http://arxiv.org/abs/1410.3916
4. Neural Turing Machines (2014)
http://arxiv.org/abs/1410.5401
5. AI institute - papers
http://allenai.org/papers.html
6. A survey on question answering technology from an information retrieval perspective (2011)
https://yadi.sk/i/5Aq-cYmjkbPDe
7. Computational Linguistics and Deep Learning
http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
Other useful refs
1. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015)
http://arxiv.org/abs/1506.07285
2. A Neural Network for Factoid Question Answering over Paragraph (2014)
https://cs.umd.edu/~miyyer/pubs/2014_qb_rnn.pdf
3. Automatic Construction of Inference-Supporting Knowledge Bases (2015)
http://allenai.org/content/publications/clark_balasubramanian.pdf
4. Learning Knowledge Graphs for Question Answering through Conversational Dialog (2015)
http://allenai.org/content/publications/hixon_naacl_2015.pdf