Transcript 1___text_mining_v0a

Text mining
[email protected]
Extract from various presentations: Temis, URI-INIST-CNRS, Aster Data …
www.decideo.fr/bruley
Information context

Big amount of information is available in
textual form in databases and online sources

In this context, manual analysis and effective
extraction of useful information are not
possible

It is relevant to provide automatic tools for
analyzing large textual collections
www.decideo.fr/bruley
Text mining definition
The objective of Text Mining is to exploit
information contained in textual documents in
various ways, including … discovery of
patterns and trends in data, associations
among entities, predictive rules, etc.
The results can be important both for:
 the analysis of the collection, and
 providing intelligent navigation and browsing
methods
www.decideo.fr/bruley
Text mining pipeline
Unstructured Text
(implicit knowledge)
Structured content
(explicit knowledge)
www.decideo.fr/bruley
Text mining process
Text preprocessing
Syntactic/Semantic text
analysis
Features Generation
Bag of words
Features Selection
Simple counting
Statistics
Text/Data Mining
Classification- Supervised
learning
Clustering- Unsupervised
learning
Analyzing results
Mapping/Visualization
Result interpretation
www.decideo.fr/bruley
Iterative and interactive process
Text mining actors
Publishers
Enriched content
Annotation tools
Tools for authors
New applications based on annotation layers
Richer cross linking based on content…
Analysts
Empowers them
Annotating research output
Hypothesis generation
Summarisation of findings
Focused semantic search…
www.decideo.fr/bruley
Libraries
Linking between Institutional repositories
Access to richer metadata
Aggregation
Aids to subject analysis/classification …
Challenges in text mining






Data collection is “free text”, is not well-organized (Semistructured or unstructured)
No uniform access over all sources, each source has separate
storage and algebra, examples: email, databases, applications,
web
A quintuple heterogeneity: semantic, linguistic, structure,
format, size of unit information
Learning techniques for processing text typically need
annotated training
XML as the common model, it allows:
– Manipulation data with standards
– Mining becomes more data mining
– RDF emerging as a complementary model
The more structure you can explore the better you can do
mining
www.decideo.fr/bruley
Data source administration
Intranet
File System
Databases
EDMS
Internet
Web
Crawling
On-line
Databank
XML Normalisation
-subject
-Author
-text corpora
-keywords
Information Provider
Format filter
www.decideo.fr/bruley
Text mining tasks
Name Extractions
Term Extraction
Text Analysis
Tools
Feature extraction
Abbreviation Extraction
Categorization
Relationship Extraction
Summarization
Clustering
Hierarchical Clustering
Binary relational Clustering
TM
Text search engine
Web Searching
Tools
NetQuestion Solution
Web Crawler
www.decideo.fr/bruley
Information extraction
Keyword Ranking
Link Analysis
Query Log Analysis
Metadata Extraction
Intelligent Match
Duplicate Elimination
www.decideo.fr/bruley
Extract domain-specific
information from natural language
text
– Need a dictionary of
extraction patterns (e.g.,
“traveled to <x>” or
“presidents of <x>”)
• Constructed by hand
• Automatically learned
from hand-annotated
training data
– Need a semantic lexicon
(dictionary of words with
semantic category labels)
• Typically constructed
by hand
Document collections treatment
Categorization
www.decideo.fr/bruley
Clustering
Text Mining example: Obama vs. McCain
www.decideo.fr/bruley
Aster Data position for Text Analysis
Data
Acquisition
Gather text from relevant
sources
(web crawling, document
scanning, news feeds,
Twitter feeds, …)
Pre-Processing
Mining
Analytic
Applications
Perform processing
required to transform and
store text data and
information
Apply data mining
techniques to derive
insights about stored
information
Leverage insights from
text mining to provide
information that improves
decisions and processes
(stemming, parsing, indexing,
entity extraction, …)
(statistical analysis,
classification, natural
language processing, …)
(sentiment analysis, document
management, fraud analysis,
e-discovery, ...)
Aster Data Fit
Third-Party Tools Fit
Aster Data Value: Massive scalability of text storage and processing, Functions for text processing, Flexibility to develop diverse
custom analytics and incorporate third-party libraries
www.decideo.fr/bruley
Aster Data Value for Text Analytics
•
Ability to store and process massive volumes of text data
– Massively parallel data stores and massively parallel analytics engine
– SQL-MapReduce framework enables in-database processing for
specialized text analytics tools
•
Tools and extensibility for processing diverse text data
– SQL-MapReduce framework enables loading and transforming diverse
sources and types of text data
– Pre-built functions for text processing
•
Flexible platform for building and processing diverse analytics
– SQL-MapReduce framework enables creation of flexible, reusable
analytics
– Embedded MapReduce processing engine for high-performance analytics
www.decideo.fr/bruley
Aster Data Capabilities for Text Data
Pre-built SQL-MapReduce functions for text processing
•
Data transformation utilities
-
•
Unpack: extract nested data for further
analysis
Custom and Packaged Analytics
Aster Data nCluster
App
App
Web log analysis
-
•
Pack: compress multi-column data into a
single column
Sessionization: identify unique
browsing sessions in clickstream data
Text parser: general tool for tokenizing,
stemming, and counting text data
-
nGram: split text into component parts
(words & phrases)
-
Levenstein distance: compute “distance”
between words
www.decideo.fr/bruley
App
App
Aster Data Analytic Foundation
Text analysis
-
App
App
SQL-MapReduce
SQL
Data
Data
Data