Transcript - bYTEBoss
Data Warehousing & Data Mining A review of Data Warehousing and Data Mining Aqeel Al-Jishi Nick Farley Masaki Osada Data Warehousing Defined: Data Warehouse – The main repository of an organization historical data, it corporate memory It contains the raw material for management’s decision support system Data Mining – (Knowledge Discovery) The process of analyzing data from different perspectives and summarizing it into useful information History In the late 1980s to early 1990s distinct computer databases were created These databases were designed to meet the data analysis needs that Operational Systems were failing to support Operational Systems failed for many reasons: Long report generation time Inability to handle loads and not optimized Many organizations had multiple Operation Systems which was a reporting nightmare Custom applications were required for reporting which slowed reporting and increased costs Goals of Data Warehousing Make information easily accessible Provide endless views and combinations of data (Slicing & Dicing) Query results returned with minimal wait time Be adaptive and resilient to change Designed with the correct users in mind (business users and management) Keep information secure but allow access to insiders Common Issues Problems with Data Acquisition may arise 80% of the time building a data warehouse will be spent on extracting, cleaning, and loading data Errors with data can be rampant: Incomplete Data (missing fields) Incorrect Data (wrong calculations) Readability Issues (strange formatting) Differing Design Views, Relational Relational Model - Bill Inmon The data warehouse is but one part of the Business Intelligence system An enterprise has one data warehouse and data marts source their information from it. Uses 3rd Normal Form to store information in the database Differing Design Views, Dimensional Dimensional model - Ralph Kimball Data warehouse is the conglomerate of all data marts within the enterprise Uses Star or Snowflake schema to emulate a multi-dimensional database Overall information is NOT normalized in the database Terminology Data Mart – A data mart is a subset of data from the data warehouse, typically used when the broad scope of the data warehouse isn’t needed Business departments commonly create, use, and alter their own data marts. Level of Granularity – The amount and level of data brought in to the data warehouse during acquisition Dimensional Model Dimension Table – A table with a single-part primary key and descriptive attribute columns. Describes the business entities of an enterprise, represented as hierarchical, categorical information such as time, departments, locations, and products Dimensional Model Fact Table – A table with numeric performance measures (metrics) characterized by a composite key The elements for the composite key come from the foreign keys from a dimensional table What is Data Mining? Basic Concept Getting “Useful Information” out of a large amount of “Data” In Business World Getting “Business Intelligence” out of a large amount of “Information” What is the difference between “Business Intelligence” and “Information”? Evolutionary Step Business Question Enabling Technologies Characteristics Data Collection (1960s) “What was my total Computers, tapes, Retrospective, revenue in the last disks static data five years?” delivery Data Access (1980s) “What were unit sales in New England last March?” Relational Databases, Structured Query Language Retrospective, dynamic data delivery at record level Data Warehousing & Decision Support (1990s) “What were unit sales in New England last March? Drill down to Boston.” On-line analytic processing, multidimensional databases Retrospective, dynamic data delivery at multiple levels Data Mining (Emerging Today) “What’s likely to happen to Boston unit sales next month? Why?” Advanced algorithms, multiprocessor computers, massive databases Prospective, proactive information delivery Focus of Data Mining Predict the future trends and behaviors using the past information Prospective analysis Uniqueness of Data Mining Heuristic in nature Capable of finding hidden patterns users would never think of Modeling is the key technology to find patterns Self-guiding An example of Modeling Yesterday Today Tomorrow Static Information Known Known Known Dynamic Information Known Known Target Build a model based on known combinations of static and dynamic information to see if it indicates the result of a dynamic information in the future. Real world examples of Data Mining “Diaper and Beer” VISA Blockbuster Entertainment Personalized Recommendation Suppliers to Wal-mart Fraud Detection Sales prediction NBA Teams Strategy analysis Data Warehouse Data Mart ERP DM ERP SCM DM CRM DM Questions? Sources • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • BOOKS: Mastering Data Warehouse Design, Relational and Dimensional Techniques Claudia Imhoff Nicholas Galemmo Jonathan G Geiger © 2003, Wiley Publishing, Inc. Indianapolis, Indiana The Data Warehouse Toolkit second edition, The Complete Guide to Dimensional Modeling Ralph Kimball Margy Ross © 2002, Wiley and Songs, Inc. New York, NY Statistical Data Mining and Knowledge Discovery Edited by Hamparsum Bozdogan © 2004 CRC Press LLC, Boca Raton, Florida Data Mining and Business Intelligence: A Guide to Productivity Stephan Kudyba and Richard Hoptroff © 2001 Idea Group Publishing, London, England Modern Systems Analysis and Design, fourth edition Jeffrey A. Hoffer, et al © 2005, 2002 by Pearson Education, Inc. Upper Saddle River, New Jersey Database Systems An Application-Oriented Approach Michael Kifer, Arthur Bernstein, Philip Lewis © 2005 Pearson Education, Inc., New York, New York ELECTRONIC: http://www.1keydata.com/datawarehousing/inmon-kimball.html http://www.fortunecity.com/skyscraper/oracle/699/orahtml/dbmsmag/9807d05.html http://en.wikipedia.org/wiki/Data_warehouse http://en.wikipedia.org/wiki/Dimension_(data_warehouse) http://en.wikipedia.org/wiki/Data_mining http://www.anderson.ucla.edu/faculty/jason.frand/teacher/technologies/palace/datamining.htm http://www.dwinfocenter.org http://www.thearling.com/text/dmwhite/dmwhite.htm http://www.learndatamodeling.com/dimension.htm http://www.learndatamodeling.com/fact.htm