Resources Check out the complete code on GitHub. Browse the GX Data Doc on Azure Blob Storage. Use Case Last week I explored Soda as a data quality testing framework for my large enterprise client. This week I’m exploring a more mature alternative called Great Expectations or GX in short.
GX generates neat HTML reports called Data Docs that give an overview of your data quality test results. The client wants to share these reports with the team - but not with the world! As the client is already using Azure, hosting the report files on Azure Blob Storage seems like a good solution.
...
Use Case For my current engagement I’m tasked with developing an automated data quality framework for a large industrial enterprise in the renewable energy sector. The client has over a hundred independent SCADA systems from various vendors gathering energy production data. All this data has to flow in one central repository to be analyzed with Databricks. The client is obligated to ensure high data quality for contractual reporting to external parties. Failure to deliver incurs high financial penalties.
...
I often work from WeWork, which is a public co-working space with a shared WIFI-network.
Securing my connection For security and privacy reasons, I therefore work behind a VPN. I purchased an AirPVN subscription and configured it as an OpenVPN connection in NetworkManager1:
NetworkManager’s terminal user interface nmtui
This way, even if I’m working from Prague, my location will always be shown as Belgium2. This was important because my current enterprise client’s IT-security team kept receiving alarms that my location was changing all over Europe.
...