Data Warehouses with Big Data - Term Paper Example

? Data Warehouses with Big Data I.D. of the Data Warehouses with Big Data In the last two decades, database management technologies have continuously evolved to yield more powerful systems for large scale storage and retrieval of data. Basic database administration is traditionally based on Relational Database Management System (RDBMS) and database administration suites like Microsoft SQL and Access, Oracle, MySQL, etc. are used (Ricardo 2011). However, technocrats and industries were not satisfied with RDBMS, and consequently data warehousing (DW) technologies were introduced during the 2000s. Basically, data warehousing is a relatively more intelligent and refined database administration system which can handle large amount of data. At the commercial level, data warehousing is further evolving in the form of Big Data which is aimed at storing and retrieving enormous amount of asynchronous and disparate data across distributed computing systems (Kusnetzky 2010). According to Ricardo (2011), traditional form of database administration is based on the fundamental technique of identifying and manipulating the different characteristic entities in a given dataset. These entities can be termed as database elements. Most important database elements are forms, fields, tables, and queries. In an RDBMS framework, these elements of a database are related with each other using simple administrator defined relationships. RDBMS can thus help an administrator to organize data in an intelligent and retrievable way. However, this technology is not always helpful to arrange data into multiple layers so as to facilitate more efficient stacking, less errors, and context aware distribution. Although the basic concepts of RDBMS are still in extensive use in different high level database applications (Kimball and Ross 2011), large scale information storage services are now progressing toward multidimensional management of data. Evolution of DW schema took place with regard to the needs of the industries and research institutes. It can be stated that evolution of data warehousing was initially aimed to mitigate the limitations of preexisting database management systems. According to Baru et al (2013), the database management industry has considerably matured and developed its own dynamics and techniques over the last twenty years. But in the past few years, data warehousing technologies have become commercially important. With the advent of Big Data, the database management industry has now developed “increased volume, velocity, and variety” of data storage, retrieval, and even processing systems (Baru et al 2013, p. 60). Experts like Devlin (2011) have gone to the extent of stating that Big Data is a better and independent form of database administration technology vis-a-vis data warehousing. But the author appears to be more critical toward the traditional data warehousing technologies. From a holistic viewpoint, synchronization of data warehousing with cloud computing facilities is precondition to Big Data (Baru et al 2013; Kusnetzky 2010). Consequently, “taxonomy of data” (see Devlin 2011, section 2) in the realm of Big Data can be regarded as a viable cornerstone in the evolution of contemporary DW schema. For more details, refer to Figure – 1. Figure – 1: Taxonomy of data as viewed at the point of transition from data warehousing to Big Data techniques. The figure shows six main varieties of data named multiplex, textual, compound, derived, atomic, and measurement. (Devlin 2011, section 2) RDBMS is a database management system that is based on defining, linking, and organizing different database elements like tables, forms, queries, etc. However, a standard DW schema gives maximum importance to the data tables. As such, data tables are organized with the help of “dimensional modeling” (Kimball and Ross 2011, p. 16). This is a method of database administration which is based on simplicity and architectural coherence of distributed database systems with complex warehousing capabilities. Dimensional modeling culminates at multidimensional data structures, which can be stratified and stacked inside limited space too (Jensen, Pedersen, and Thomsen 2010). Consequently, DW schema is based on the basic technique of stacking and layering huge amounts of data and then interconnecting them in a most suitable and intelligent way. There are three major configurations in this data management system: Star schema, snowflake schema, and fact constellation schema. The construction of star schema is done on a central data table termed as the fact table. Several dimensional tables are then connected to this fact table. These dimension tables can be used to define and stratify huge amount of information around a single fact table. In snowflake schema, the dimension tables are further normalized to provide even more layers of data storage and processing. The most complicated configuration is the fact constellation schema. This kind of schema has bigger and more complex star schema arrangement of a large number of smaller star schema data sets. Consequently, a fact constellation schema must possess large number of internal links and connectors. (Kimball and Ross 2011; Jensen, Pedersen, and Thomsen 2010, pp. 10-15) New storage paradigms are emerging with the lapse of time. Although experts like Devlin (2011) tend to focus more on the differences between data warehousing and Big Data (see Figure – 2), the two technologies can hardly be regarded to have evolved differently. Figure – 2: In his article, Devlin (2011, section 1) refers to the figure above as a traditional data warehouse design. However, the middle portion of this diagram shows a layered configuration of enterprise data warehouse. This is a direct consequence of dimensional modeling, which is an imperative in Big Data implementation as well. In strict sense, more complex data warehousing synchronized with cloud computing can give rise to Big Data storage mechanisms. For complex sets of data, mere enhanced memory may not be enough. Sorting, storage, retrieval, searching, and navigation are some of the key processes that are embedded in modern database management. Consequently, a key paradigm of Big Data implementation is based on increased processor speeds. For very high processing speeds, shared hardware access through cloud computing is utilized in Big Data facilities. (Baru et al 2013; Kusnetzky 2010) Another key paradigm in modern data warehousing is dimensional modeling. This is a key concept that is seminal in implementing a Big Data facility and develops into multiple layers of data storage system. Dimensional modeling gives rise two multidimensional interpretation of data. In RDBMS, data is organized in two dimensional tables having rows and columns. Location of a given piece of data set would be traced on the basis of cell number and column number. But in advanced data warehousing, data can be organized in three dimensional cubes. Each cell of a cubical arrangement must be defined with three parameters instead of two. Hence, chances of errors are lessened and more data is stored. Application of the concept of dimensional modeling taken from traditional data warehousing is a fundamental requisite of Big Data. (Kimball and Ross 2011; Jensen, Pedersen, and Thomsen 2010) Furthermore, the main drive behind Big Data is due to the desire of accomplishing extremely complicated tasks for advanced research purposes. For example, research and development in the field of bioinformatics calls for high speed processing of enormous genomic data. Big Data helps the geneticists to achieve this aim (for more details, see Figure – 3). So Big Data also involves the paradigm of developing continuously expanding storage system to accomplish the previously impossible storage and retrieval tasks. Big Data or advanced data warehousing systems of today are thus being benchmarked on the basis of continuous performance and versatility (Baru et al 2013). Figure – 3: The figure above shows an example of Big Data use in bioinformatics. As per the industry standards, even a general Big Data storage and retrieval system can process one peta-byte of data in just 750 days! Since bioinformatics data generated in the realm of genomic research and development are extremely large, only a Big Data implementation system can meet the processing requirements that were regarded to be unattainable even only a few years ago. (Trelles, Prins, Snir, and Jansen 2011, p. 224) Majority of computer science experts like Kimball and Ross (2011) agree upon the fact that Big Data is the advanced version of data warehousing. While traditional data warehousing still puts emphasis on the use of forms and queries, advanced data warehouses are based on data tables that are extensively interlinked with each other in a multidimensional environment. When this kind of complex arrangement is combined with high speed cloud computing, commercial level Big Data utilities are created. According to Baru et al (2013), the main qualities of Big Data include high speed processing and enormous memory units. Big Data signifies industry level usage of pervasive computing with the help of multilevel information processing, storage, and retrieval. References Baru, C., Bhandarkar, M., Nambiar, R., Poess, M., & Rabl, T. 2013, Benchmarking Big Data Systems and the BigData Top100 List. Big Data, 1, pp. 60-64. Devlin, B. 2011, Will data warehousing survive the advent of big data? - Strata. Strata - Making Data Work. Available: http://strata.oreilly.com/2011/01/data-warehouse-big-data.html. Last accessed on September 22, 2013 Jensen, C.S., Pedersen, T.B. and Thomsen, C. 2010, Multidimensional Databases and Data Warehousing. San Rafael: Morgan and Claypool. Kimball, R., & Ross, M. 2011, The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling. Hoboken: Wiley. Kutnetzky, D. 2010, What is "Big Data?" | ZDNet. ZDNet | Technology News, Analysis, Comments and Product Reviews for IT Professionals. Available: http://www.zdnet.com/blog/virtualization/what-is-big-data/1708. Last accessed on September 22, 2013 Ricardo, C. 2011, Database Illuminated. Sudbury: Jones and Bartlett. Trelles, O., Prins, P., Snir, M., & Jansen, R. C. 2011, Big data, but are we ready? Nature reviews Genetics, 12, p. 224 Read More