In this new video I share how to overcome Azure CPU quota limits with Databricks Asset Bundles, a common roadblock many Databricks practitioners face when deploying Databricks Asset Bundles on Azure for the first time. Problem If you’re playing around with Databricks projects, Azure’s default CPU quota limits often fall short of what Databricks Asset Bundle Python template jobs and pipelines actually need to run. ...
In this video I explain how to read data from an Airtable into a PySpark dataframe on Databricks. Airtable is a popular spreadsheet tool used at many enterprises. It offers additional features compared to other tools such as Microsoft Excel and Google Sheets
In this video I demonstrate how to perform data quality checks on a Delta table in Databricks using Soda Core. Soda Core is the open-source Python package developed by Soda. It can be compared to Great Expectations, but is much simpler in my opinion. I enjoy using Soda in my professional projects and will continue exploring this framework. ...