In this new video I share how to overcome Azure CPU quota limits with Databricks Asset Bundles, a common roadblock many Databricks practitioners face when deploying Databricks Asset Bundles on Azure for the first time. Problem If you’re playing around with Databricks projects, Azure’s default CPU quota limits often fall short of what Databricks Asset Bundle Python template jobs and pipelines actually need to run. ...
In this video I explain how to read data from an Airtable into a PySpark dataframe on Databricks. Airtable is a popular spreadsheet tool used at many enterprises. It offers additional features compared to other tools such as Microsoft Excel and Google Sheets
In this video I demonstrate how to perform data quality checks on a Delta table in Databricks using Soda Core. Soda Core is the open-source Python package developed by Soda. It can be compared to Great Expectations, but is much simpler in my opinion. I enjoy using Soda in my professional projects and will continue exploring this framework. ...
This video was recorded by me for the United Manufacturing Hub In this video, we’ll walk through connecting HighByte, a data processing engine, with the United Manufacturing Hub (UMH). HighByte is an OT-friendly alternative to Node-RED to handle operational technology data, making it a useful tool alongside the UMH. We will start by demonstrating how to download and set up HighByte within Kubernetes. Next, we’ll show you how to configure it to use data from the UMH, model this data, and send it back to UMH. From there on, the modeled data in the UMH will then be used in other applications and stored automatically in the UMH Historian. ...
This video was recorded by me for the United Manufacturing Hub In this first part of the UNS Basics mini series, we introduce the Automation Pyramid as a system to control the plant. We highlight the challenges related to data analytics and present the Unified Namespace as the solution. We also show how the Automation Pyramid connects to the Unified Namespace, according to the United Manufacturing Hub’s philosophy. Finally, we briefly discuss where historical data from the Unified Namespace is saved. ...