About the Guest:

Jim Gavigan brings 30 years of experience in industrial manufacturing, from vibration analysis and control systems to working at Rockwell Automation and OSIsoft. He founded Industrial Insight in 2016 to help companies maximize the value of their time series data.

Key Topics:

• What is Datatude and why it matters
• The five dimensions: Data, Technology, People, Priorities, and Culture
• Why companies struggle to build sophisticated analytics on poor foundations
• The importance of starting small with concrete, achievable projects
• Common pitfalls: prioritizing technology over people and process
• How to scale data initiatives across multiple plants
• Building the right team and culture for data success

Key Takeaway:

Stop trying to implement advanced AI and analytics on crappy data. Focus on getting the basics right first: clean data, proper documentation, the right people, and a culture that supports data-driven decisions.

Connect with Jim:

• LinkedIn
• Website

Exactly one month ago I published the first episode of my Watts in Your Data Podcast. In my opinion, a topic of critical importance that is all too often overlooked, especially with all the buzz around AI.

In the most recent episode, I had the pleasure of inviting guest speaker Thomas Dhollander, co-founder of Timeseer.AI. Together we explored critical challenges in industrial time series data reliability and observability.

Key takeaways:

• Data quality is not just a technical issue — it’s a people and process problem, deeply tied to governance and ownership.
• Data management at many companies is still reactive — fixing issues only after models break or KPIs look suspicious. When companies scale their data-driven operations, they need to turn to proactive data management to avoid ending up in firefighting mode.
• Data maturity varies by company and by industry — utilities and pharma often lead, some other industries may still view data as a byproduct.
• Data should be treated like a product — with quality checks, documentation, and accountability — especially as you scale analytics. This is also true for OT data.
• AI needs data quality — ML and AI depend on quality inputs and sensor drift or misconfigured tags can quietly corrupt your entire model output. Interestingly, AI is also a key enabler in scaling data quality.
• Moving data to the cloud introduces new risks — missing context, inconsistent pipelines, and ownership confusion.

Welcome to my podcast! In this very first episode I introduce the topics of this podcast and explain my background in data.

• Follow the show
• About Denis Gontcharov

Soda Core is the open-source Python package developed by Soda. It can be compared to Great Expectations, but is much simpler in my opinion. I enjoy using Soda in my professional projects and will continue exploring this framework.

• Check out the complete code on GitHub.
• Browse the GX Data Doc on Azure Blob Storage.

Use Case

Last week I explored Soda as a data quality testing framework for my large enterprise client. This week I’m exploring a more mature alternative called Great Expectations or GX in short.

GX generates neat HTML reports called Data Docs that give an overview of your data quality test results. The client wants to share these reports with the team - but not with the world! As the client is already using Azure, hosting the report files on Azure Blob Storage seems like a good solution.

Why Azure Blob Storage?

1. Easy Implementation

Installing new solutions at enterprises is notoriously difficult. There’s often a long procurement process and many budget-approval hoops to jump through. Because the client is already using Azure, it’s only a small step to provision an additional Blob Container.

2. Familiar Access Control

As the Blob Container becomes part of the client’s Azure ecosystem, the existing IT-team can easily manage access to the Data Docs using RBAC¹. There are no new security-measures to implement.

Solution

This solution is a direct implementation of the GX documentation about how to host and share Data Docs on Azure Blob Storage.

Sample Data

The code defines two expectations for the following simple Pandas dataframe:

1. The NumericColumn may only have values between 0 and 90.
2. The TextColumn may only have values from “Item 1” to “Item 10”.

   NumericColumn TextColumn
0             10     Item 1
1             20     Item 2
2             30     Item 3
3             40     Item 4
4             50     Item 5
5             60     Item 6
6             70     Item 7
7             80     Item 8
8             90     Item 9
9            100    Item 10

Code

I won’t go into the same steps in detail here. Rather, I’ll highlight a couple of important points:

• I configured the following azure_blob_storage site definition in my great_expectations.yml file.

  azure_blob_storage:  # this is a user-selected name - you can select your own
    class_name: SiteBuilder
    store_backend:
       class_name: TupleAzureBlobStoreBackend
       container: \$web
       connection_string: ${AZURE_STORAGE_CONNECTION_STRING}
    site_index_builder:
      class_name: DefaultSiteIndexBuilder

• If you are running the [setup_gx.py] file for the first time, don’t forget to set do_config = True and update the path in CONTEXT_DIR to your system.

• Don’t forget to set your Azure Blob Storage connection string in the connection_string option. As this value is secret, it’s not included in my repository.

Result

I think the result looks pretty neat. The index.html along with the other files are created in a Blob Container $web:

The final result can be accessed by anyone on the internet here. We see that five GX runs have been made, resulting in five Validation Results.

Note how the Expectation Suites tab gives more information about the Expectation Suites, in this case panda_expectations. This feature gives business users clear information about how the date they’re using has been tested.

Conclusion

Overall, I quite like the Great Expectations framework so far. Comparing it to Soda Core there’s a couple of points I prefer about GX:

1. GX has more open-source features, e.g. Data Docs.
2. Although GX is more convoluted, the organization of Expectations into Suites allows to maintain order as the project grows.
3. The community behind GX seems sufficiently active.
4. No date is shared outside of the company to access certain features, as opposed to Soda Cloud.

For my current engagement I’m tasked with developing an automated data quality framework for a large industrial enterprise in the renewable energy sector. The client has over a hundred independent SCADA systems from various vendors gathering energy production data. All this data has to flow in one central repository to be analyzed with Databricks. The client is obligated to ensure high data quality for contractual reporting to external parties. Failure to deliver incurs high financial penalties.

Soda

Soda built their product offering Soda cloud on top of the open-source Python package Soda Core¹. In a nutshell, their solution allows you to:

1. Test your data pipelines
2. Monitor data quality
3. Create reports

Image source: Exploring the Soda Data Quality Platform

Simple Workflow

To test the framework, I defined a simple workflow that performs a number of data quality checks on two large datasets.

Installation

First I install the framework on my Databricks notebook:

pip install -i https://pypi.cloud.soda.io soda-spark-df

Next, I import and instantiate the Scan class, create a data definition name, and set the data source to be tested to a Spark dataframe:

from soda.scan import Scan

# Create a scan object
scan = Scan()

# Set a scan definition
# Use a scan definition to configure which data to scan,
# and when and how to execute the scan.
scan.set_scan_definition_name("Data Completeness")
scan.set_data_source_name("spark_df")

# Attach a Spark session
scan.add_spark_session(spark)

Configuration

I like the simplicity of Soda’s YAML configuration that is stored in my Databricks workspace.

The connection to Scoda Cloud’s dashboard and datasources is configured in a sonda_conf.yml file:

soda_cloud:
  host: cloud.soda.io
  api_key_id: 2bcda34c-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  api_key_secret: zuNLl1k55YM_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The actual data quality checks are defined in a checks.yml file:

checks for table_one:
  - missing_count(site_id) = 0:
      name: Ensure there are no null values in the site ID column
  - duplicate_count(site_id) = 0:
      name: Ensure there are no duplicate site ID's

checks for table_two:
  - missing_count(device_id) = 0:
      name: Ensure there are no null values in the site ID column
  - num_devices > 50:
      num_devices query: |
        SELECT COUNT(DISTINCT device_id) FROM table_two

Note how I’m defining a custom data quality check with SQL code.

The configuration is then added to the scan instance:

scan.add_sodacl_yaml_file("/Workspace/Users/Denis/Soda/soda_settings/checks.yml")
scan.add_configuration_yaml_file("/Workspace/Users/Denis/Soda/soda_settings/soda_conf.yml")

Result

Executing the checks and viewing the logs is straightforward:

scan.execute()
print(scan.get_logs_text())

This is an extract of the output:

INFO   | Scan summary:
INFO   | 2/4 checks PASSED:
INFO   |     table_one in spark_df
INFO   |       Ensure there are no null values in the site_id column [PASSED]
INFO   |     table_one in spark_df
INFO   |       Ensure there are no null values in the site_id column [PASSED]
INFO   | 2/4 checks FAILED:
INFO   |     table_two in spark_df
INFO   |       Ensure there are no duplicate site_id's [FAILED]
INFO   |         check_value: 1
INFO   |     table_two
INFO   |       num_devices > 50 [FAILED]
INFO   |         check_value: 41.0
INFO   | Oops! 2 failures. 0 warnings. 0 errors. 2 pass.
INFO   | Sending results to Soda Cloud

We see that only two of the four checks pass successfully:

• The site_id values are not unique for every row.
• There are less than 50 distinct device_id values.

Reporting

What I like about Soda Cloud is the out-of-the-box visualization of my test results. At a glance, I can instantly see how my data quality varies from day-to-day, troubleshoot issues, view data, and even send alarms.

Screenshot of my Soda Dashboard

Verdict

Soda looks promising, but it’s too early to tell whether it fits my current use cases.

What I like:

• Minimalistic design
• Intuitive configuration
• Convenient dashboard

What I don’t like:

• Data is sent off-premises
• Financial cost²
• Not very mature product yet³

Overall, I’m excited to continue exploring Soda as a potential alternative to Great Expectations!