Transcript Planting Data

USDA Open Data / Big Data
Presentation
for the
Xcelerate Solutions
Federal Big Data Working Group
Presented by the
Office of the
Chief Information Officer
March 16, 2015
1
Overview
• Open Data
–
–
–
–
–
–
Implementation of OMB’s M-13-13- Open Data Policy
USDA Last Quarter’s Accomplishments
Open Data Projects
Open Data Success Stories
Data Quality
Open Data Summary
• Big Data
– Intersection of Open and Big Data
– Why Is It Important To Agriculture
– USDA Big Data Projects
• Food Resilience Theme of the Climate Data Toolkit Initiative
• Using Administrative Data for Statistical Purposes
– Big Data Summary
2
Open Data
3
USDA Implementation of M-13-13
• Developed 3-Step M-13-13 Implementation Strategy
• Dataset Collection/Publishing
•
•
•
•
Created and maintained an Enterprise Data Inventory (EDI)
Submit EDI to OMB only (not to public)
Maintain a Public Data Listing (PDL)
Published datasets to PDL to USDA.gov/data and Data.gov
Enterprise Data
Inventory
Public
Data
listing
• Developed and documented the USDA Data Publishing Process
•
Develop Dataset Prioritization, Privacy/Legal Review and IT Security Review Processes
• Developed Customer Feedback and Engagement Plan
•
Conduct Internal/External Engagement Sessions
4
Last Quarter’s Accomplishments
 November 2014 – Submitted 61 Datasets
 February 2015 – Submitted Datasets


14 New Datasets
61 Updates
 The Enterprise Data Inventory (EDI) consists of
651 datasets


506 Release to the Public (Public Data Listing)
145 Non-Public
 December 2014 – Received Government/Industry
Roundtable Final Results
 Continue to support the USDA Science Council
Committee on Statistics, we formed a cross Agency
Administrative Data Categorization Working Group
(ADCWG)

Determine ways to categorize USDA Data in a standardize
manner

Determine how to collect, disseminate and access scientific
statistical data for use within the Department
5
Open Data Projects
•
Office of the Secretary (OSEC) Disaster Clearinghouse
Project
•
•
•
•
•
Census Find it/Connect it Project
•
•
•
Develop a pilot using USDA, Census and NOAA data to determine
the economic impact of a flood on Farmers, Ranchers, Producers and
Consumers
Working with the National Institute Standards and Technology
(NIST) to establish a common set of standards that can be shared
across the federal government
Census One City, One Problem – Software
Development Kit (SDK) Project
•
•
•
•
Develop a one-stop application for disaster victims to find assistance
before, during and after a disaster
Project sponsor - Dr. Ann Mills and Ms. Rebecca Shively of OSEC
Successfully completed requirements and preliminary design
User’s Test and Web-design will occur in March 2015
Unlocking the value of open data to promote economic growth, fuel
innovation and entrepreneurship
Enabling cities and counties to use data to deliver tangible and
transformative results for their communities
Select one city and one problem and solve it through the use of
Census, USDA and NOAA’s open data
Census Center for Applied Technology (CAT) Lab
•
•
•
Real-Time Disaster Intelligence Map with Crowdsourcing
Pollinator Field Data Collection with Citizen Science
Farm Service Administration Common Land Unit (CLU) Hackathon
for Open Application Programming Interface (APIs)
6
Open Data Success Stories
.
• WhyHunger
– The WhyHunger organization is
using the Food Nutrition Service
(FNS) data as one of the resources
in providing information for the
Summer Feeding Program for
School Children, as well as healthy
food sourcing in emergency food
programs. Featuring program
profiles, resources and strategies
employed by food banks and
agencies, including building
relationships with farmers, creating
voucher programs, teaching people
how to grow their own food, and
supporting an organizational culture
that prioritizes nutrition.
http://www.whyhunger.org/getInfo/showArticle?arti
cleId=3674&utm_source=Clearinghouse+Connectio
n++May+2014&utm_campaign=CCMay2014&utm_m
edium=email
7
Open Data Success Stories
.
• Farmers Market Directory
– The Open Data concept was used
to develop applications such as
the Farmers Market Directory,
which gives developers and
designers direct access to the
wealth of farmer’s market
information housed in the online
database.
• Rails-to-Trails
– The Rails-to-Trails Project
extensive mapping tool mixes
data from the National Household
Travel Survey and USDA’s
Economic Research Service
(ERS) to provide a better
understand pedestrian and bicycle
mode patterns in rural areas.
8
Data Quality
•
When is the Data Good Enough for Release
•
•
•
Request for Forest Service for Trails Data
Should we consider a Disclaimer?
Data Quality Improvement (Data Cleansing)
•
Cleansing of Mata data – we’ve improved the
quality of our datasets by 36% since our last
submission.
•
•
•
•
Cleansing of the Actual Data to make it more open.
We are adopting the five star maturity methodology
•
•
•
•
•
•
Spelling
Broken links
Duplicate entries (etc)
Stage 1 - PDF Files
Stage 2 – Excel Files
Stage 3 – CSV Files
Stage 4 – XML Files
Stage 5 - Data that can be link to other data
Data Management Policy
•
FY2015 – develop a policy to govern the data
management of USDA data
9
Open Data Summary
• USDA Open Data Team has made tremendous progress
– Data Collection
– Process Development
– Customer Feedback and Engagement
• Challenges Facing the Open Data Team
– Increasing Agencies and Administrative Offices participation in submitting
datasets to be published (Internal and External)
– Engaging the public to improve the quality and quantity of USDA datasets
– Data Quality and Data Management
– Dedicating resources to move the Open Data Initiative to the next level
• Permanent Chief Data Officer
• Data Scientist
10
Big Data
11
Intersection of Open and Big Data
12
What Is Generating Big Data?
Sensor
Technology
and
Networks
(measuring all
kinds of data)
Scientific Instruments
(collecting all sorts of data)
Mobile Devices
(tracking all objects all the
time)
13
Value of AgBig Data
Value Generated by Ag Big Data
Making the
link between
 On-Farm Optimization
• Data-based decision-making for many more decisions
• Early problem identification for management response
• Custom solutions to minimize inputs and maximize
yields
 Input Product Innovation for Unique
Conditions
• Biotech / seed research
• Equipment R&D
• Other input supplies
Market Linkage
• Improves transparency and
predictability of markets
16
Big Data In Agriculture
• Big data starts with precision ag, but is much larger
than on-farm data alone
– On-Farm Precision Ag (fertilizer, planting, crop
protection)
– 3rd Party Data (weather, satellite / aerial imagery, soil
fertility, topographic data)
– Research & Development (Academic / Land grant data,
biotech and equipment company publications, software and
analytics
– Commodity Markets (local, regional, global)
15
Importance of This Issue
• Why is Big Data in agriculture emerging now?
– Computer capacity has now expanded to process vast
volumes of data
• Why is Big Data an important issue?
– Enables the promise of Precision Agriculture to be realized
– Makes critical agronomic decisions on small areas within
each field
• What is the level of urgency?
– Several major Ag companies have just launched or soon
will launch Big Data
16
Decision Support for Farmers
Enabling Farmers to Make 40 Critical Decisions
Critical Decision Sets:
Planning
Data
• Seed
Selection
• WeedControl
Program
• InsectControl
Program
Pre-Planting
Data
• Fertility
Program
• Tillage
Program
Planting
Data
• Plant
Population
Dynamics
• Seed
Depth
In-Season
Data
• PostEmergenc
e Pest
Control
• Crop
Diagnostic
s
Harvest
Data
• Equipment
Goal:
• Crop
Marketing Maximize
Net Return
Per Acre
Productivity Tools:
Seed
Factors
• Seed
is
lynchpin
decision – key to
establishing
yield potential
Planting
Factors
•Focus on
best field
configuration,
preparation
and planting
elements
Pest-Control
Factors
• Focus on
insect and
weed
control
regimes
In-Season
Decisions
•Diseases,
nutrient, etc.
approaches
based on infield
environment
19
Farmer
Profitability
Strategy
Pioneer Farmer Profitability Strategy
Whole Farm Profitability Services
Current Services Offering
•
•
•
•
•
•
Field-by-field crop plans
Planting maps
Harvest maps
Crop insurance
Financial services
Grain marketing
Harvest /
Evaluate
Plan
Monitor
Plant
Tools to deliver additional insights for advanced farmer profitability.
20
USDA Big Data Projects
 Food Resilience of the Climate Data Toolkit Initiative
 Led by the USDA OSEC and OSTP
 Tool Kit Focuses on four types of Data;
 Production:




How will crop yield and production of other food products be affected by changes in climate?
What crops and locations will be most vulnerable to changes in climate?
How will changes in climate affect the types of crops that can be successfully grown in a location?
Under what conditions (i.e., for what crops and in what locations) will indirect effects of climate change
(e.g., pests, pathogens, invasive species, fire) overwhelm direct effects of increasing temperature and
changes in precipitation?
 Supply
 What parts of the food supply chain will be most vulnerable to climate change?
 How will processing, storage, and transportation need to be modified to continue to supply safe food
products under a changing climate?
 Nutrition
 How will climate change impact food borne illness that includes chemical as well as microbial sources?
 How will an increase in pesticide or veterinary drug use with climate change impact the contamination
of food?
 Security
 How will climate change affect the nutrient content of staple foods?
 How will the cost of a balanced diet be impacted under a changing climate?
 What segment of the population will be most vulnerable to changes in food prices?
 What countries are most vulnerable to climate change that affects food yield, quality, and availability?
 Launch of the Food Resilience Theme – July 2014
19
USDA Big Data Projects
 Using Administrative Data for Statistical Purposes
 Driven by Memo M-14-06 – Guidance For Providing and Using Administrative
Data for Statistical Purposes
 Led by USDA Science Council Committee on Statistics
 Convenes monthly to discuss the progress on each Agency’s initiatives
 Improving how agencies leverage existing and Federal data to facilitate
agencies' programmatic work and enhance the value of those data to the
American public
 Facilitating the collection and publishing of USDA Agency administrative data
to enable full, appropriate use of non-public administrative data for statistical
purposes
20
Big Data Summary
• The Value of Investing in Open and Big Data:
– Open and Big Data will create the next major technological
“sea change” in agriculture
– Will enable completely different ways to innovate and
invent new business models
– Enable the mass market to source, supply and consume it
with better outcomes
– Innovators and inventors can build stuff that matters and
build stuff that works
– Unlock more value in the existing data market and has the
potential to create a new industry of businesses
21
Questions
Open Data Points-Of-Contact
Joyce Hunter
USDA Acting Chief Information Officer (CIO)
and
Deputy CIO for Policy and Planning
202-720-8833
[email protected]
Bobby Jones
USDA Senior Advisory to the
Deputy CIO for Policy and Planning
and
Acting Chief Data Officer
202-690-4305
[email protected]
22