Transcript cult_analytics_DH09 - Software Studies Initiative

Cultural Analytics: theory,
methodology, practice
Dr. Lev Manovich
Director, Software Studies Initiative, Calit2 + UCSD
Professor, Visual Arts Department
with Dr. Jeremy Douglass
Post-doctoral researcher, Software Studies Initiative, Calit2 + UCSD
You can capture this lecture using any media and share it.
You can also download this presentation @ softwarestudies.com
presented on June 22, 2009 as keynote @Digital Humanities
2009
PRESENTATION CONTENTS:
analysis and visualization of large cultural data:
- theoretical implications
- interfaces / visualization techniques
- methods for analysis of visual media and borndigital culture
Our research is made possible by the support from:
NEH Office of Digital Humanities
Singapore Ministry of Education
California Institute for Telecommunication and
Information (Calit2), UCSD Division
Center for Research in Computing and the Arts
(CRCA)
UCSD Visual Arts Department
UCDARnet
UCHRI
Software Studies Initiative Researchers:
Lev Manovich | Director
Noah Wardrip-Fruin | Associate Director
Jeremy Douglass | Postdoctoral Researcher
Cicero Silva | Software Studies Brazil
William Huber | Researcher | PhD student, UCSD
Chanda Carey | Researcher | PhD student,
UCSD
Daniel Rehn | Researcher | MFA student, UCSD
Software Studies Initiative Collaborators:
Yuri Tsivian, Department of Art History, University of Chicago: cinemetrics.lv | film
analysis
Adele Eisenstein: Digital Formalism project (Department for Theatre, Film and
Media Studies (TFM), Vienna University; the Austrian Film Museum; Interactive
Media Systems Group, Vienna University of Technology) | film analysis
Isabel Galhano Rodrigues, University of Porto, Portugal | gesture analysis
David Kirsh, Cognitive Science, UCSD | dance video analysis
Jim Hollan, Cognitive Science, UCSD | visualization | cultural analytics software
Falko Kuester, Structural Engineering, UCSD + Calit2 | visualization | cultural
analytics software
Yoav Freund, Computer Science and Engineering, UCSD | machine vision | cultural
analytics software
Kay O’Halloran, Multimodal Analysis Lab, National University of Singapore |
Mapping Asian Cultures project
Giorgos Cheliotis: Communication and New Media, National University of Singapore
| Mapping Asian Cultures project
Matthew Fuller: Goldsmiths College, University of London | software studies
Benjamin H. Bratton: Visual Arts, UCSD + Calit2 | software studies
BACKGROUND
The joint availability of large cultural data sets
(the exponential growth of user generated
content, digitization efforts by museums and
libraries, digital traces and self-presentation ),
cultural information (web presence of all
professional cultural agents), the tools already
employed in the sciences to analyze and
visualize big data, and the techniques developed
in new media art makes feasible new
methodologies for the study, teaching and
presentation of cultural processes and artifacts including contemporary cultural production,
sharing and consumption.
fMRI of a global “cultural brain”
today neuroscience combines single neuron recordings, tracking activities of neural
networks (1mm) and neural maps (1cm), fMRI of the neural activity of the whole brain,
and other methods.
Traditional analysis of culture can be compared to recording and analyzing activity of a
single cell or a small cell population.
We need to start tracking, analyzing and visualizing larger cultural structures (including their connectivity and dynamics over space and time) - equivalents of neural
networks, maps, cortical columns, and the whole brain.
Applying other techniques from neuroscience such as staining cells.
Applying the basic method of contemporary neuroscience: combining results from
different research methods which measure cultural processes at diff. resolutions
(MRI, PET, staining cells, genomics, etc.)
new scale:
1. digitization of existing cultural assets
2. web presence
3. social media
4. cultural globalization
The rapid growth of professional educational and cultural institutions in many newly
globalized countries along with the instant availability of cultural news over the web,
the availability of software tools, and cheap travel has also dramatically increased the
number of "culture professionals" who participate in global cultural production and
discussions.
Hundreds of thousands of students, artists, designers have now access to the same
ideas, information and tools. In many cases, it is no longer possible to talk about centers
and provinces.
The students, culture professionals, and governments in newly globalized countries are
often more ready to embrace latest ideas than their equivalents in "old centers" of world
culture.
growth of a global culture space after 1990. Example: Fashion Weeks, 2005.
growth of a global culture space after 1990. Example: cumulative number of new art
biennales, 1895-2008.
world heat map made up from 35 million Flickr geo-coded photos).
http://www.cs.cornell.edu/~crandall/photomap/
NEW SCIENCE OF CULTURE?
Until now, the study of human beings/cultural processes relied on two types of data:
shallow data about many people/objects (statistics, sociology) or deep data about a
few people/objects (psychology, psychoanalysis, ethnography, “thick description,”
humanities).
The development of high performance computing, mobile devices, web and social
media allows for a fundamentally new methodology for the study of human beings and
society:
We can now collect detailed data about very large numbers of people/objects/cultural
processes. We no longer have to choose between size and depth. (Example: reality
mining project at MIT; Citysense; Blogpulse.) Software, hardware, and networks
people use can capture detailed data about some dimensions of peoples’ cultural
behavior, imagination and thinking (creating, remixing, reusing, communicating, etc.)
CULTURAL
ANALYTICS
Why “Cultural Analytics”?
existing terms:
knowledge discovery
from data to knowledge
Google Analytics
web analytics
business analytics
visual analytics = “the science of analytical
reasoning facilitated by interactive visual
interfaces”
Our goals:
- being able to better represent the complexity, diversity, variability, and uniqueness of
cultural processes and artifacts than the current cultural technologies allow for
- democratize cultural research by creating sets of open-source tools for cultural
analysis and visualization (in particular, study of visual and interactive media)
- develop techniques to describe the dimensions of cultural artifacts and cultural
processes which until now received little or no attention
- create much more inclusive cultural histories and analysis - ideally taking into account
all available cultural objects created in particular cultural area or time period (“art
history without names”)
- create representations and interfaces for visual exploration which operate across
multiple scales - from details of structure of a particular individual cultural
artifact/processes to massive cultural data sets/flows
INTERFACES
Platform for Cultural Analytics research environment: HIperSpace (287 megapixels).
What kind of interface do we need to create “situational awareness” for “cultural
analysts”?
The interfaces used today whenever a person/group monitors a performance of
system/machine/process, makes decisions and controls (i.e., a “human-machine
system) all share the common principle:
- multiple displays which present information about a system/process using diff. visual
techniques.
Examples: vehicle interfaces, patient monitoring in a hospital, control room of a plant,
information dashboards, financial news.*
Culture is a complex system / process / environment - therefore we should use similar
interface design for studying and monitoring it.
Text
Example of an existing interface: Barco’s iCommand
Example of an existing interface: AT&T control center (2001).
Cultural Analytics research environment: interface mockup. Focus: geo map.
IDEOLOGY OF DIGITAL TRACES. Not all cultural and social activities leave digital traces on
the web. A significant part of today’s global culture is “digitally invisible.” Therefore, we can’t
only do projects which rely on web data or existing databases - we also need to take on
“digitally invisible” cultural activities using available and original etnographic research.
Example: envisioned analysis of the development of “shanty towns” in Mexico city (using
research of the local architects working with these communities.)
Cultural Analytics research environment: interface mockup. Focus: long tail.
Example of long tail visualization: looks on lookbook.nu (sorted by number of hypes).
Created by softwarestudies.com
Example of long tail visualization: looks’ titles on lookbook.nu (sorted by number of
hypes). Created by softwarestudies.com.
Close-up of visualization of looks’ titles on lookbook.nu (sorted by number of hypes).
Created by softwarestudies.com.
Map of Science visualization
Cultural Analytics research environment: interface mockup. Focus: relationships map.
Cultural Analytics research environment: interface mockup running on HIPerWall
Cultural Analytics research environment: interface mockup running on HIPerWall
INTERFACES which combine media browsing
and visualization to enable visual exploration of
data
“The aim pursued with visual exploration is to give an overview of the data and to
allow users to interactively browse through different portions of the data. In this
scenario users have no or only vague hypotheses about the data; their aim is to
find some. In this sense, visual exploration can be understood as an undirected
search for relevant information within the data. To support users in the search process,
a high degree of interactivity must be a key feature of visual exploration techniques.”
Christian Tominski, Event-Based Visualization for User-Centered Visual Analysis, PhD
Thesis, Institute for Computer Science, Department of Computer Science and
Electrical Engineering, University of Rostock, 2006.
“Exploration denotes an undirected search for interesting features in a data set.”
Kreuseler, M., Nocke, T., and Schumann, H. A History Mechanism for Visual Data
Mining. In Proceedings of the IEEE Symposium on information Visualization
(infovis'04) - Volume 00 (October 10 - 12, 2004). INFOVIS. IEEE Computer Society,
Washington, DC, 49-56. 2004.
source: www.infovis-wiki.net.
Cultural Analytics software running on HIPerSpace (May 2009)
Cultural Analytics software running on HIPerSpace (May 2009)
Cultural Analytics software running on HIPerSpace (May 2009)
Visualization of complete text of Hamlet. Source: project Guthenberg.
Visualization of complete text of Hamlet. Lines of text are rendered as solid rectangles.
Visualization of complete text of Hamlet. Hamlet = red, King = black.
NBC TV news montage. data: 2 shows per year, 1960-2008.
Representation of video structure: Betty Boop cartoon (left) vs. music videos (center and
right)
THEORETICA
L ISSUES
SOME THEORETICAL ISSUES AROUND
CULTURAL DATA MINING / CULTURE VIS:
Culture does not equate cultural artifacts. How can we automatically analyze
“context” in a meaningful way? If cultural process/activity more important than the
“outputs” being produced, how to visualize it?
Statistical paradigm (using a sample) vs. data mining paradigm (analyzing the
complete population). Modernity/normal distribution vs. Software Society/power law.
Pattern as a new epistemological object. From meaning to pattern: humanities have
been focused on interpreting the meanings of a cultural artifact/process. Today we can
easily uncover the meanings of each cultural artifact - but we don’t know the larger
patterns they form. The new scale of culture points toward a pattern as a new unit of
analysis (because we can not afford to consider meanings of every single artifact.)
New digital divide - between social and cultural activities/people which leave digital
traces and whose that do not.
From small number of genres to multi-dimensional space of features where we
can look for clusters and patterns.
Learning from a Search Engine:
size:
cultural criticism: very small samples of cultural production
a search engine: every accessible web document
categorization:
cultural criticism: cultural objects are placed into small number of genres/categories
a search engine: analysis of each web document to generate its unique description (not
reducing to small number of categories)
links:
cultural criticism: analysis of some links (“influences”) between a given object and others
a search engine: systematic consideration of all links between a given object and others
features:
cultural criticism: manual analysis of a small number of features of cultural objects
a search engine: automatic analysis of the same (very large) set of features of web
documents and links between them. Examples: Sense Networks “attributes 487,500
dimensions to every place in a city. Google: “PageRank reflects our view of the importance
of web pages by considering more than 500 million variables and 2 billion terms.” “Our
technology analyzes the full content of a page and factors in fonts, subdivisions and the
precise location of each word. “
Analysis of Born Digital Culture:
1. While patterns of interaction between “readers” and cultural objects/processes
have always been a part of culture, recent technologies offers possibilities to capture
and analyze such interactions.
-> “Big data” has a new meaning in the case of interactive digital media. For
example, we can deal with a single video game – but include in our analysis hundreds
of hours of video which represent gameplay sessions by many users.
2. Because digital objects are driven by software, they have self-describing property
- their structure and behavior are specified in their "software layer" (HTML code, project
file of a AE composition, game engine, etc.)
-> in addition to) analyzing the "surfaces" of digital objects (as we will do with the
books or films), we also need to develop methods to analyze, summarize and visualize
the structures and dynamics of "software layers" of these objects.
3. Media objects today are encountered by users/authors in a larger digital
environment - the commands and interface of a desktop media viewer/authoring
application, the commands and HCI of a social media site such as YouTube, etc. We
need to develop methods for the analysis of digital environments - including functions
and UI of desktop and mobile applications, webware, web sites, social media sites,
etc.
4. Rather than existing in final fixed form as media inscriptions (film print, printed
photo, printed book, etc.), digital objects are sets of possibilities.
In the case of commercial interactive media, software specifies what can happen at
every moment in interaction. This can take such simple form as a tree graph - for
instance, in early games the player is given a choice to go left or right etc. The fields
of HCI, interaction design and information architecture already developed a range of
graphical and non-graphical techniques to describe the structures of possible
interaction - so we can adopt some of them for the humanities, and also develop
new ones.
In the case of generative/procedural art and design, software specifies possible
range of work "states." For example, a program which generates an abstract image
implicitly specifies all possible images which it can produce - with each actual output
being one instance from this sea of possibilities. (Think of typical works done with
Processing.) In the case of time-based procedural media, such range of possibilities
will change over time.
TECHNIQUES
AND
APPLICATION
S
media: cinema
view: macro
data: average shot
length stats on 1100+
feature films from
cinemetrics.lv
types of moving images
view: micro
data: Dziga Vertov, A
Man With a Movie
Camera (1929); Betty
Boop cartoons (19321936); contemporary
motion graphics.
Shot lengths and a distribution of particular shot type over time in A Man With a
Movie Camera. Each bar corresponds to one shot. Bar height = shot length.
Scale: complete film (68 minutes)
scale: complete film
zoomed view
Shot lengths and distribution of different shot type over time in A Man With a
Movie Camera (film length = 68 min). Each bar corresponds to one shot. Bar
length = shot length.
Combining manual annotations with automatic measurements or
visual/spatial/temporal structure. Example: visualization of temporal patterns in A Man
with a Movie Camera.
scale: 1 minute
Comparing different temporal patterns on a micro-level in A Man With a Movie
Camera. X axis = time. Y axis = brightness mean (dark grey line).
Vertical bar = shot boundaries (light grey line).
Temporal patterns of low-level visual features in Common Go music video. Shown:
brightness mean, brightness median, saturation median, hue median.
Comparative visualization: center of mass in Betty Boob cartoon (black) vs. music video
(red). The center of mass for all frames are projected on top of each other.
Visualizations created by UCSD undergraduate students on HIPerSpace (June 2009)
Visualizations created by UCSD undergraduate students on HIPerSpace (June 2009)
media: visual art
view: macro + micro
data: 35 paintings by
canonical artists
representing transition
from realism to
modernism,1849-1917.
source: artstor.org
FROM
MAPPING TO
SIMULATION
from visualization of cultural patterns to
modeling and simulation to include all
professional cultural content produced in the
world
Studying cultural processes:
from taxonomy to evolutionary biology
We usually think of culture in terms of taxonomies - styles, genres, market segments equivalent of biology in the 18th century.
The exponential increases in the number of cultural objects (lifestyle products, films,
photographs, songs, graphic designs, etc.) created by both professional producers
and non-professionals - and the similar increase in the number of producers - create a
new cultural universe.
The sheer size of the new cultural universe, and its much higher connectivity[1]
makes appropriate to apply scientific paradigms and methods (including
mathematical models and computational modeling) used in evolutionary biology,
environmental biology, genomics, bioinformatics, and other live sciences to study
large-scale biological phenomena. [2]
--------------------------------------[2] Future collaboration with Calit2 Center for Algorithmic and Systems Biology
(CASB)
----------------------------------------------------------------------------------[1] Higher cultural connectivity:
“globalization”: access to the same ideas and tools + more people using English as a
global cultural platform;
large-scale globally distributed production in games industry, film industry, etc.;
social media software which encourages remix and “media mobility”;
desktop software production environment which similarly makes it very easy to borrow
elements from other cultural products;
increase in travel;
access to cultural objects produced by everybody else (Youtube, Flickr, portfolio web
sites of professional designers, etc.)
Mathematical simulations of global cultural flows
based on the existing data?
Blue Brain project @Henry Markram's Brain and Mind Institute at the École
Polytechnique (EPFL) in Lausanne
Henry Markram anticipates that a simulation of a complete human brain down to the
molecular level (based on all existing experimental neuroscience data) will be available
before 2020.
Text
“a forest of neurons”: a dye is injected into each neuron and then developed in order to
reveal the morphology. Image by Blue Brain project.
If we do simulation-based research into global cultural flows, what kinds of models
would we need? More complex than brain simulation? More simple?
If simulation-based research is now being applied to new scientific areas, let us use it
to study global cultural flows and dynamics.
What new analytical categories can we develop if we aim for an equivalent of a Blue
Brain project - simulating all of contemporary global cultural developments and
relationships using all available data?
Thank you.
follow Cultural Analytics at
softwarestudies.com