Data analysis is a diagnostic tool, not a cure for bad data

Within the portfolio of research consulting services that I provide, data analysis is by far the most requested, wherein clients send me already collected data for statistical and/or thematic analysis. One of the things that I insist on in this regard is that clients must send me a “complete, clean, and labelled” dataset. Of course, I could opt to do the “cleaning” myself and earn myself a few more cents in the process. But the “hidden” purpose for this requirement is to ensure that while my clients may not have a full picture of the trends within the raw data, they must at least be familiar enough with it’s raw contents to decide upfront if they feel the data is fit for purpose vis-à-vis their business or study objectives. 

Data analysis is a diagnostic tool, not a cure for bad data. Analysis acts like an X-ray or a mirror. It can accurately show you where the bone is broken or where the flaws are, but looking at the X-ray doesn't heal the fracture. Non-technical clients often think that by hiring a brilliant data scientist the insights will automatically be clean and valid. But the reality is that even sophisticated models run on poor quality data will simply give highly accurate insights into how bad the data is. While data analysts can implement complex models to "fix" the data, such as imputing missing fields, in doing so they are potentially introducing bias that ultimately leads to flawed conclusions and decisions. The classic "garbage in, garbage out" rule applies.

To fix data quality issues, users must move away from the results dashboard and closer to the data sources. Data ought to be treated as a product and therefore created or collected in accordance with strictly defined  standards of what "good data" looks like before it ever enters a database. Bad data must be prevented ab initio.

In the case of primary data collection, bad data is prevented through correct study design. This entails having a clear framework to finding the answers to the specified research problem, and requires clear research objectives and the identification of appropriate methods for sampling and data collection. In the case of secondary or archival data it entails prior examination of the credibility of the data sources, purpose and context for which the data was collected, methodology used to collect the data, and completeness of the data, among other things.

Long story short, while data analysis is an essential tool for uncovering insights, it is fundamentally diagnostic rather than corrective. The value of analysis hinges on the quality of the underlying data. Preventing bad data through rigorous study design and careful evaluation of data sources is critical to producing valid and reliable results.