Relevance of Big Data Analytics in Traffic Management - Case Study Example

Relevance of Big Data Analytics in Traffic Management: A case of Traffic management; California Introduction Background In the recent past, cities have seen their economy grow so fast. Amongst the things that accompany this growth is the traffic (which is very big). The city of California show an average traffic flow of over 300 million daily [1]. With heavy traffic come heavy problems of traffic offence. Monitoring and controlling the traffic is the only solution. This monitoring is effectively through the surveillance cameras that have been installed at several check points. The cameras relay images and videos to the servers of traffic department. As can be imagined, this is a lot of data that is sent to servers and the information so unstructured and unclassified. This data is so huge to the tune of terabytes of data per day [2] hence the name Big Data. The data is only useful when it has been analysed to something meaningful. Given the need to analyse this data under its characteristic properties is what poses to be a database management problem that will be pursued by the study. Problem statement The main problem that must be solved is the sufficient traffic monitoring and control process. The only way is through the efficient big data handling and eventually big data analytics [3]. Given the nature of the big data, handling it by organizations is a usual challenge. Among the characteristics include, but not limited to, heterogeneity (A single entity may have very many dimensions), unstructured/unclassified, [4]. Understanding whether, and how the traffic department has/or will manage the big data effectively is the main aim of the study. Importance of the study This study is essential in understanding the challenges of handling big data (including analytics, storing and structuring) .Understand if any current undertakings in research can be dependable for big data analytics. Understand the future of big data management in handling the problems that accompany management of traffic data. In order to realise the importance of the research, the study is guided by the research questions in the next subsection [5]. Research questions What are the challenges facing big data management? Is there any research progress that is important in big data handling? Based on the current status of research, what is the future of application of big data handling? Theoretical framework This study has an underlying theoretical framework. It is the based on the ‘big data integration theory’. Theory of Database Mappings, Semantics and Programming Languages. The theory identifies that the big data integration differs from the traditional data integration in 4V,s. It includes volume, velocity, veracity and variety. This forms the basis of the characteristics of the big data [6, 7]. Literature Review Characteristics of the big data The big data is commonly characterised by the 5 Vs. Volume is a key characteristic feature. Big data is characterised by big volumes of data. Not just the big volumes come heterogeneity. For example data about a car comes with number plate, color, speed of motion, model. We need to cope with biased, noisy data and multisource data streams. Others are generic of this one. Large volumes of data require a lot of space on the database for storage. Alongside the same, analysing this data becomes a big challenge since big records have to be sorted through. Data velocity (motion) is another important aspect to consider. Big data is characterised by almost/if not real-time data production and usage, i.e in-memory analytics. The challenge is to cope the speed with which data is real-time created and used. Big data has a big variety aspect. There are different types of data and each of them requires different analytical methods [6, 7, 8, 9 ]. Overall conceptual conclusion The Big Data phenomenon presents opportunities and perils. Optimistically speaking, massive data amplifies the algorithmic inferential power that have been seen to be successful on modest-sized data sets. The challenge has been development of the theoretical principles needed for scaling inference and learning algorithms to massive scale. Pessimistically, massive data can amplify the error rates that form part and parcel of any inferential algorithms. The biggest challenge is to control such errors even in the face of the uncontrolled sampling processes and heterogeneity underlying many huge data sets. One other major issue associated with Big Data is that problems often come with temporal constraints, where a slowly obtained high-quality answer becomes less useful compared to a quickly obtained medium-quality answer. Overall, there is a problem in which the classical resources of the theory of computation—e.g., time, space and energy—trade off in complex ways with the data resource. Various aspects of this general problem are faced in the theory of statistics, computation and relating disciplines—where topics such as dimension reduction, Monte Carlo sampling, distributed optimization, low-rank matrix factorization, compressed sensing, streaming and hardness of approximation are of clear relevance [10]. Challenges and opportunities of big data There are challenges that face the handling of big data on a capacity angle. They include lack of experts of know-how, lack of finances to cater for the expensive infrastructural needs [11]. The characteristic challenges posed by the big data are a big deal and can easily render the desired outcome of the data almost unattainable. Big data is has high dimensionality. According to [11], high dimensionality brings noise accumulation, spurious correlations, and incidental heterogeneity. A combination High dimensionality with large sample size creates issues such as heavy computational costs and algorithmic instabilities [12]. The massive samples in Big Data are typically aggregated from multiple sources at different time points using different technologies. This creates issues of heterogeneity, experimental variations, and statistical biases, and requires us to develop more adaptive and robust procedures. The noisy Big Data could be more valuable than tiny samples because general statistics obtained from frequent patterns and correlation analysis usually overpower individual fluctuations and often disclose more reliable hidden patterns and knowledge. Further, interconnected Big Data forms large heterogeneous information networks, with which information redundancy can be explored to compensate for missing data, to crosscheck conflicting cases, to validate trustworthy relationships, to disclose inherent clusters, and to uncover hidden relationships and models. Mining requires integrated, cleaned, trustworthy, and efficiently accessible data, declarative query and mining interfaces, scalable mining algorithms, and big-data computing environments. At the same time, data mining itself can also be used to help improve the quality and trustworthiness of the data, understand its semantics, and provide intelligent querying functions. As noted previously, real-life medical records have errors, are heterogeneous, and frequently are distributed across multiple systems. This is structuring the unstructured. There are many techniques that could be used to attack big data projects. It all depends on the available technologies. Among the technologies include the non-relational databases such as the Mongo database. Cloud computing is also another technology that makes big data handling a success for programmers. There are several software developments to enhance big data analytics. Hadoop is key analytical technology in the big analytics [13, 14]. One possible solution to problems associated with big data is hardware. Some vendors are using increased memory and powerful parallel processing to crunch large volumes of data extremely quickly. Another method is putting data in-memory but using a grid computing approach, where many machines are used to solve a problem. One other solution to this challenge is to have the proper domain expertise in place. Make sure the people analyzing the data have a deep understanding of where the data comes from, what audience will be consuming the data and how that audience will interpret the information [15]. Methodology This research is mainly a survey research on the case study. The study used a survey research of the interviews type. The information was obtained from appropriate key respondents from the traffic department. Based on the reviewed literature questions were formulated based on the four categories as seen in the literature review section. In order to find proper information, some questions were formulated so as to compare for status with the current one. They include how the unstructured information is structured for useful information and how the information is analysed to suite different end users. Based on the outcome, it would be projected on the possible database future. The questions and question structure are as shown in the appendix section of this paper. The information sought aimed at looking at the database management of big data to give desired end user results vis a vis the challenges. It also seeks to understand the current research trends in the sector. On this basis, the analysis section explores the outcome of the case study. The results of this case can be extended to other departments if positive. The outcome of the survey are recorded and analysed in the section below. Findings and analysis From the case study, this study finds out a few things about databases and big data. To start with, the traffic department admits that in the recent past, it was hard to store the big volumes of data. As such, monitoring traffic was a very serious challenge. Currently, although not sufficiently, the department is able to store all the data. This is attributed to the new databases that have been resorted to by the department. One of the correspondents (technician) says there was a change to Mongo database from structured DB for data storage which can store a lot of data per unit time. Somehow, the same volume challenge has been alleviated by cloud computing techniques. However, the correspondence agrees that they still face expert crisis for handling the complex database situation. The study finds out that the data received is highly heterogeneous. This comes by due to the various aspects and dimensions of the traffic including the registration number, color, speed, behaviour, type of the car/or otherwise and several others. This, they say is a bit of a challenge when it comes to analysing. The correspondence also says that most of the data received in the process is unclassified/unstructured. This makes the analysis process. It is also noted that the there’s a variety of information of information collected by the cameras. It ranges from information of cars to pedestrians and respective dimensions as talked about in the heterogeneity aspects of the data. However, the study finds out that since there’s a predefined source of information, the errors arising from differential source errors is reduced. The biggest challenge, it says is reducing the errors and systematically analysing the information for relevant end-user utilization. However, the correspondence implies that in the recent past, there has been pursuit of software that could make the analytical process easier and faster to improve the usefulness of the data. Apache Hadoop is the software lately embraced for analytical process. They say it has made the querying process so easy and fast (real-time). The software structures the unstructured which is then analysed for timely usefulness of the data which searches through millions of records to give results in real-time. In the appendix section, there is a detailed hadoop modules. In addition to the new software, cloud computing has been highly embraced for making the data storage manageable. Apache hadoop offers a free comprehensive and cohesive platform that encapsulates data integration, monitoring, data processing and workflow scheduling [ 16]. “With Apache Hadoop, we were able to not only store the massive volume of image and video data, but also to enable a large number of users to access the data quickly. Traffic violation data can now be stored for 24 months instead of only three months. And it now takes less than a second to accurately search for the plate number or driving track from the over 2.4 billion records of vehicle data.”, one correspondent says. It can be concluded that database management of big data is taking a good shape in ensuring that the much need of big data is satisfied. This means that with continued research, the future of big data is bright. References [1] T. Wang, Reducing greenhouse gas emissions and energy consumption using pavement maintenance and rehabilitation. . [2] H. Technology, "Improving traffic management with big data analytic", 2015. [3] F. Ohlhorst, Big data analytics. Hoboken, N.J.: John Wiley & Sons, 2013. [4] W. Inmon and K. Krishnan, Building the unstructured data warehouse. Bradley Beach, NJ: Technics Publications, 2011. [5] M. Nascimento, T. Sellis, R. Cheng, J. Sander, Y. Zheng, H. Kriegel, M. Renz and C. Sengstock, Advances in Spatial and Temporal Databases. [6] Z. Majkić, Big data integration theory. Cham: Springer, 2014. [7] S. Kumar and D. Toshniwal, "A data mining framework to analyze road accident data", Journal of Big Data, vol. 2, no. 1, 2015. [8] K. Krishnan, Data warehousing in the age of big data. [9] J. Hurwitz, A. Nugent, F. Halper and M. Kaufman, Big data for dummies. Hoboken, N.J.: Wiley, 2013. [10] P. Holland, "Characteristics of Big Data – Part One | Data Intensity", Dataintensity.com, 2016. [Online]. Available: http://www.dataintensity.com/characteristics-of-big-data-part-one/. [Accessed: 26- Mar- 2016]. [11] J. Hurwitz, M. Kaufman and A. Bowles, Cognitive computing and big data analytics. [12] C. Rao, "Analysis of high-velocity data streams", India, 2014. [13] "MongoDB for GIANT Ideas | MongoDB", Mongodb.org, 2016. [Online]. Available: https://www.mongodb.org/. [Accessed: 26- Mar- 2016]. [14] D. Agrawal, S. Das and A. El Abbadi, Data management in the cloud. [San Rafael, Calif.]: Morgan & Claypool, 2013. [15] "Big Data: What It Is and How to Overcome Its Challenges", Askingsmarterquestions.com, 2016. [Online]. Available: http://www.askingsmarterquestions.com/big-data-what-it-is-and-how-to-overcome-its-challenges/. [Accessed: 26- Mar- 2016]. [16] Ishwarappa and J. Anuradha, "A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology", Procedia Computer Science, vol. 48, pp. 319-324, 2015. Read More