Skip to main content

Data quality

Data quality is a perception or an assessment of data's fitness to serve its purpose in a given context. The quality of data is determined by factors such as accuracy, completeness, reliability, relevance and how up to date it is. As data has become more intricately linked with the operations of organizations, the emphasis on data quality has gained greater attention.


An oft-cited estimate originating from IBM suggests the yearly cost of data quality issues in the U.S. during 2016 alone was about $3.1 trillion. Lack of trust by business managers in data quality is commonly cited among chief impediments to decision-making.

The problem of poor data quality was particularly common in the early days of corporate computing, when most data was entered manually. Even as more automation took hold, data quality issues rose in prominence. For a number of years, the image of deficient data quality was represented in stories of meetings at which department heads sorted through differing spreadsheet numbers that ostensibly described the same activity.

Determining data quality

Aspects, or dimensions, important to data quality include accuracy, conformity and consistency. As a first step toward improving data quality, organizations typically perform data asset inventories in which the relative value, uniqueness and validity of data can undergo baseline studies. Established baseline ratings for known good data sets are then used for comparison against data in the organization going forward.

Methodologies for such data quality projects include the Data Quality Assessment Framework (DQAF), which was created by the International Monetary Fund (IMF) to provide a common method for assessing data quality. The DQAF provides guidelines for measuring data dimensions that include timeliness, in which actual times of data delivery are compared to anticipated data delivery schedules.

Managing data quality

Software tools specialized for data quality management match records, delete duplicates, establish remediation policies and identify personally identifiable data. Management consoles for data quality support creation of rules for data handling to maintain data integrity, discovering data relationships and automated data transforms that may be part of quality control efforts.

Collaborative views and workflow enablement tools have become more common, giving data stewards, who are charged with maintaining data quality, views into corporate data repositories. These tools and related processes are often closely linked with master data management (MDM) systems that have become part of many data governance efforts.

Data quality management tools include IBM InfoSphere Information Server for Data Quality, Informatica Data Quality, Oracle Enterprise Data Quality, Pitney Bowes Spectrum Technology Platform, SAP Data Quality Management and SAS DataFlux.


Comments

Popular posts from this blog

Computer Operating Systems

When you turn on the power to a computer, the first program that runs is usually a set of instructions kept in the computer's  read-only memory  (ROM). This code examines the system hardware to make sure everything is functioning properly. This  power-on self-test  (POST) checks the CPU,  memory , and  basic input-output systems  (BIOS) for errors and stores the result in a special memory location. Once the POST has successfully completed, the software loaded in ROM (sometimes called the BIOS or firmware ) will begin to activate the computer's disk drives. In most modern computers, when the computer activates the  hard disk drive, it finds the first piece of the operating system: the  bootstrap loader . The bootstrap loader is a small program that has a single function: It loads the operating system into memory and allows it to begin operation. In the most basic form, the bootstrap loader sets up the small driver programs that interface ...

Shaping the Digital Economy: From Infrastructure Thinking to Ecosystem Orchestration

Digital transformation is often discussed in terms of systems, platforms, and technologies. But at a national level, the real challenge is not building infrastructure, it is ensuring that infrastructure becomes economy-shaping capability . As digital ecosystems mature, the focus is shifting from isolated innovation to system-wide orchestration . The question is no longer “What systems do we build?” It is becoming “How do we ensure everything works together as one economy?” Beyond Infrastructure: The Limits of Building Alone Most digital economies begin with infrastructure development: Payment switches Mobile money platforms Core banking systems Digital identity frameworks Connectivity layers These are necessary foundations, but they do not automatically create impact. Without alignment across institutions, infrastructure risks becoming: Technically functional but underutilized Interoperable on paper but fragmented in practice Innovative in design but slow in ad...

Artificial intelligence for IT operations (AIOps)

Artificial intelligence for IT operations (AIOps) is the use of deep learning and big data analytics to automate routine administrative tasks, including deployment, root cause analysis and problem resolution, for an information technology (IT) system. Ideally, an AIOps platform brings three important capabilities to the enterprise: The ability to recognize abnormal system behavior faster and with greater accuracy than humanly possible. The ability to use IFTTT business rules to automate routine tasks. The ability to streamline communication among stakeholders. How AIOps works AIOps tools gather information from the IT tools and devices already in place and apply detailed analytics and machine learning to that information in order to identify potential issues and correct them. Typically, AIOps data comes from network log files, cloud monitoring tools and helpdesk ticketing systems. Big data technologies aggregate and organize all of the systems' outpu...