Data Quality: Issues, Processes and Importance - Literature review Example

Running Head: DATA QUALITY Data Quality: Issues, Processes and Importance Introduction This essay is based on the concept of data quality and discusses several aspects which relate to this phenomenon. To start with, a number of issues and problems that affect the quality of data are discussed. Secondly, different ways in which the issues related to data quality as identified in the discussion can be resolved are presented. Emphasis is on how solving the problems can help improve the overall quality of data. Thirdly, the relevant processes, tools and technologies that are used to improve the quality of data are presented. Lastly, the various ways in which quality data is important to the public sector are discussed. Data Quality: Inherent Issues In general, the quality of data that is used by an organisation can be understood in terms of the extent to which the data is highly consistent, completely comprehensible and relevant to particular situations (Singh & Singh, 2010, p. 41). The extent to which data is of quality with regard to particular situations depends on the particular organisation that is using the data. For example, educational institutions have different requirements for the data that is used as opposed to organisations operating in the other sectors. It is only when data adheres to these needs that it can be said to be of high quality. Further, an organisation that uses high quality data stands to benefit a lot in terms of the quality of decisions made and their overall outcomes (Rhind, n.d., p. 3). This implies that the issue of the quality of data is of great importance to organisations. There are several issues that are inherent to data quality. The first one is the subject of accuracy. For data to be of high quality, it must represent the actual values that it stands for (Watson, Kraemer & Thorn, 2006, p. 3). This implies that since data is supposed to represent actual reality, it must not fail to do so. There are several ways in which data may fail to be accurate, thus losing its overall quality. For example, when data lacks linkages that allow two or more separate systems to access and edit it, it usually fails to reflect the reality (Watson et al., 2006, p. 4). Furthermore, issues of accuracy of data may arise as a result of large scale errors in the data itself. The result of this is that the overall quality of the data is compromised because of lack of accuracy in individual values. The second issue that is related to quality of data is its completeness. Generally, completeness of data can be understood in terms of whether or not the data represents all the information that is available and should be covered by it. In order for data to be termed as complete, all the values must be available and should be in a state that is usable (Singh & Singh, 2010, p. 41). The importance of completeness in data cannot be overemphasised. According to Chapman (2005, p. 4) organisations can guarantee the quality of their data by prioritising the use of little but complete sets of data over large amounts of data that are not complete. Therefore, completeness is an important aspect of the quality of data. The third aspect of data quality is consistency. For data to be of high quality, it has to be completely consistent. This means that all values that are similar should relate to the same kind of information. There are two types of consistency of data: structural and semantic (Chapman, 2005, p. 15). For structural consistency, the entities, attributes and types that are used in the presentation of data should have a uniform format. This is usually achieved by ensuring that the database is well designed and has good attributes all through. On the other hand, data that is said to be semantically consistent is presented in such a manner that it is clear and completely unambiguous (Chapman, 2005, p. 15). The fourth aspect of quality of data is validity. Typically, data should be correct and completely reasonable for it to be regarded as of high quality (Singh & Singh, 2010, p. 42). This suggests that the data that is collected, processed and stored must be able to support the analysis that it is meant to facilitate. Another issue that is inherent in data quality is the degree of flexibility of the data. Generally, data that is of high quality is able to capture the actual and constant changes that happen in the real world. This is applicable to different scenarios. For instance, Watson et al. (2006, p. 4) point out that flexibility of data that is used within the context of a learning institution should reflect the rapid changes that take place in the day-to-day operations within such an institution. Chapman (2005, p. 16) points out that flexibility is required even in collecting and processing data about scientific phenomena. Therefore, the level to which data is flexible determines its overall quality. The issues inherent in data quality can be illustrated as shown in figure 1. Figure 1: Issues inherent in data quality Solutions to the Issues Inherent in Data Quality There are several solutions that have been recommended to tackle the issues of accuracy, validity, consistency, flexibility and completeness of data. This is because the different types of errors that arise from these issues compromise the overall quality of the data. It is therefore necessary for organisations and other stakeholders to solve these issues as a way of avoiding the consequences of using data that is of poor quality. First of all, solutions to the issues of data quality are meant to be applied to the entire process that data is subjected to between its reception and final use. This is due to the point that errors in data can occur at any of the stages of the data handling process including reception, entry, integration, maintenance and extraction (Singh & Singh, 2010, p. 42). Therefore, theoretically, organisations seek to eliminate all these areas by ensuring that the correct processes and procedures are used during all the stages of the data management process. The data management process that is used by organisations is represented in figure 2. Figure 2: The data management process used by organisations In addition to this, there are other specific methods that have been developed to correct different forms of errors in data. Although all these methods fall under the data cleaning process, their general approach, the specific issues they seek to address and the tools that they employ vary slightly from each other. For instance, Cong, Fan, Geerts, Jia and Ma (2007, p. 315) developed a complicated model for solving the problems of accuracy and consistency in data. In this model, it is observed that since the problems of inconsistencies and inaccuracy in data result from failure to follow the right procedures in preparing the data, the problems can be resolved by three primary methods: repairing individual values in the data set, use of incremental methods on the values and ensuring that the repair process is accurately done (Cong et al., 2007, p. 316). Such an approach ensures that the data cleaning process takes matters of consistency and accuracy into consideration. According to Rahm and Do (2000, p. 2), in order for the data cleaning process to be effective in removing all the errors that arise from issues of consistency, accuracy and the others, the process must take into consideration several requirements. To begin with, it is stated that the cleaning process should be able to detect all manner of errors in individuals and integrated sources (Rahm & Do, 2000, p. 2). This is necessary for all the errors to be removed from the data. Secondly, the process should be performed together with other specific operations that are meant to improve the overall quality of data. This denotes that data cleaning works best when it is performed in conjunction with other procedures that are meant to reduce the level of errors in data. One of the other procedures that should be used with data cleaning is data validation. Through data validation procedures, potential errors that may not be detected by the data cleaning procedures are pointed out, analysed and resolved (SOA & LL Global, 2011, p. 19). The importance of this procedure is that it is completely dependent on how well the data cleaning process is carried out. Therefore, by ensuring that data cleaning is carried out in conjunction with data validation, all the different types of errors that have been identified can be resolved. This in turn ensures that the data that is stored and utilised is of the highest possible quality. Relevant Processes and the Latest Tools and Technologies According to Rahm and Do (2000, p. 1) such a process is usually divided into three main phases: operational sources of the data, extraction, transformation and loading phase and the last phase of storage of the data. In all these phases, complex processes of extraction, translation, matching and integration are applied to transform the data to a form that is ready for use. In order to enhance this process and improve the overall quality of the data that is obtained from it, organisations use the Total Quality Data Management (TQDM) approach. In this approach, a four-stage cycle involving defining, measuring, analysing and improving data is applied to the entire data handling process for continuous improvement (Wang, Ziad & Lee, 2001, p. 6). With regard to tools and technologies, Rahm and Do (2000, p. 8) identify three areas in data quality for which special tools have been developed as follows: data analysis, cleaning and the phase of extraction, transformation and loading. The following is a list of data management tools that can be used by organisations. a) Copy Manager b) QM Software c) QuickAddress d) Trillium Software e) Integrity f) Data QUALITY Tool g) Dataflux. Importance of Data Quality in the Public Sector There are several ways in which the quality data is of importance in the public sector. To begin with, quality data increases the productivity of organisations in general. Improvement of the level of productivity of an organisation as a result of availability of high quality data occurs in several ways and across different sectors of the economy. For example, availability of data of good quality can be used to help organisations reallocate resources appropriately (Yiu, 2012, p. 16). This leads to attention being given to areas of great concern in the operations of the organisation. Secondly, data that is of good quality can be used to enhance the productivity of organisations by being used to make improvements in the way the organisations offer their services (Bujak, Carvalho & Sriramulu, 2012, p. 100). For example, organisations operating in the public health sector can considerably improve their service delivery by relying on more accurate and complete data about the health condition of their patients. It is by relying on continuous improvements in service delivery, which in turn is dependent on the use of quality data, that organisations are able to increase their productivity to the general public. Usually, the decision-making process plays a very important role in the way organisations carry out their responsibilities. For example, organisations rely on well developed predictions to plan for how they can respond to possible future scenarios (Yiu, 2012, p. 15). By forecasting their future operations, organisations are able to respond to changes in policy quite effectively. In order for organisations to make such decisions, it is important that they have access to high quality data. Therefore, quality data is important for organisations in the public sector in that it helps them make decisions about their future courses of action. This is important because by doing so, organisations are able to plan on how to provide high quality services to the public (Health Information and Quality Authority, 2011, p. 12). This is applicable to different public sector organisations in that they can use quality data to predict trends in the market and respond by instituting the most appropriate risk management strategies. The third way in which high quality data is important in the public sector regards the benefits that members of the public and organisations can derive from the process of sharing information on the operations of organisations that operate in the sector. In general, the public is in support of the practice of organisations sharing important data on how they operate (Shakespeare, 2012, p. 24). For this process to be successful, organisations need to share with the general public data that is of the highest possible quality. When this happens, individuals are able to understand the way organisations are conducting their operations. This enhances the issue of accountability of organisations in the public sector. This is helpful to not only individuals but also to different organisations. For example, when one is accessing specific documents, the input of several public organisations is required at different stages of the process. When such organisations share high quality data among themselves and the general public, this makes the process much easier and transparent. According to SOA & LL Global (2011, p. 5), poor quality data usually leads to undesirable consequences to organisations which include the risk of losing profitability, the manner in which the organisation manages its capital and its overall rating by the public. When applied to the public sector, it can be seen that organisations require high quality data as a way of avoiding the consequences of using data that is of poor quality. For example, an organisation that fails to use accurate data to present its performance to the public risks losing its overall rating and profitability over the long-term. Conclusion Data quality is important in the public sector because it helps the public understand the way organisations conduct their operations. Further, by using high quality data, organisations are able to reallocate their resources and make general improvements on the way they conduct their operations. Also, in order to ensure that they have and use high quality data, organisations apply the TQDM approach to the data handling process. With regard to dimensions of quality data, organisations seek to get and use data that does not have errors related to consistency, relevance, validity and accuracy. This is because the benefit of having quality data exceeds by far that of using data which is erroneous. References Bujak, A., Carvalho, W., & Sriramulu, R. (2012). Lean management and operations in the global professional services industry. In U. Baumer, P. Kreutter & W. Messner (Eds.), Globalisation of professional services (pp. 95-104). Berlin: Springer-Verlag. Chapman, A. D. (2005). Principles of data quality. Report for the Global Biodiversity Information Facility. Retrieved from http://www.niobioinformatics.in/books/Data%20Quality.pdf. Cong, G., Fan, W., Geerts, F., Jia, X., & Ma, S. (2007). Improving data quality: Consistency and accuracy. Retrieved from http://win.ua.ac.be/~adrem/bibrem/pubs/CongFGJM07.pdf. Health Information and Quality Authority. (2011, April). International review of data quality. Dublin: Health Information and Quality Authority, Rahm, E., & Do, H. H. (2000). Data cleaning: Problems and current approaches. Retrieved from https://www.ki.informatik.hu-berlin.de/mac/lehre/lehrmaterial/Informationsintegration/Rahm00.pdf Rhind, G. (n.d). Poor quality data: The pandemic problem that needs addressing. Independent White Paper Commissioned by Postcode Anywhere. Retrieved from http://www.grcdi.nl/PCAwhitepaper.pdf Shakespeare, S. (2012). Shakespeare review: An independent review of public sector information. Retrieved from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/198752/13-744-shakespeare-review-of-public-sector-information.pdf Singh, R., & Singh, K. (2010). A descriptive classification of causes of data quality problems in data warehousing. International Journal of Computer Science Issues, 7(3), 41-50. Retrieved from http://ijcsi.org/papers/7-3-2-41-50.pdf SOA & LL Global. (2011). Experience data quality: How to clean and validate your data. Retrieved from https://www.soa.org/research/research-projects/life-insurance/research-2011-12-data-quality.aspx Wang, R. Y., Ziad, M., & Lee, Y. W. (2001). Data quality. New York: Kluwer. Watson, J. G., Kraemer, S. B., & Thorn, C. A. (2006). Data quality essentials: Guide to implementation – resources for applied practice. Centre for Educator Compensation Reform. Retrieved from http://cecr.ed.gov/pdfs/guide/dataQuality.pdf Yiu, C. (2012). The big data opportunity: Making government faster, smarter and more personal. Retrieved from http://www.policyexchange.org.uk/images/publications/the%20big%20data%20opportunity.pdf Read More