What’s the difference between Databricks’s Overwatch and System Tables tools?
What’s the difference between Databricks’s Overwatch and System Tables tools?
Databricks recently released System Tables, to help users understand and monitor their usage. They also had the older and more skunk works project called Overwatch, which also provides usage and infrastructure information. So what’s the difference between the two? When should I use either one? Let’s dive in! Introduction Databricks as a whole is a
Jeffrey Chou
02 Oct 2023
![]()
Blog
The Easy and Comprehensive Guide To Understanding Databricks Pricing: How It Works and How To Reduce Your Cost
The Easy and Comprehensive Guide To Understanding Databricks Pricing: How It Works and How To Reduce Your Cost
Databricks is a popular unified analytics platform and a go-to solution for many organizations looking to harness the power of big data. Its collaborative workspaces have become the industry standard for data engineering and data science teams and an ideal environment for building, training and deploying machine learning and AI models at scale. However—as with
Jeffrey Chou
28 Sep 2023
![]()
Blog
Why Your Data Pipelines Need Closed-Loop Feedback Control
Why Your Data Pipelines Need Closed-Loop Feedback Control
Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale
Jeffrey Chou
10 Sep 2023
![]()
Blog
Are Databricks clusters with Photon and Graviton instances worth it?
Are Databricks clusters with Photon and Graviton instances worth it?
Configuring Databricks clusters can seem more like art than science. We’ve reported in the past about ways to optimize worker and driver nodes, and how the proper selection of instances impacts a job’s cost and performance. We’ve also discussed how autoscaling performs, and how it’s not always the most efficient choice for static jobs. In
Jeffrey Chou
17 Aug 2023
![]()
Blog
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
Introduction: The Gradient Command Line Interface (CLI) is a powerful yet easy utility to automate the optimization of your Spark jobs from your terminal, command prompt, or automation scripts. Whether you are a Data Engineer, SysDevOps administrator, or just an Apache Spark enthusiast, knowing how to use the Gradient CLI can be incredibly beneficial as
Pete Tamisin
11 Jul 2023
Blog, Case Study
Introducing: Gradient for Databricks
Introducing: Gradient for Databricks
Wow the day is finally here! It’s been a long journey, but we’re so excited to announce our newest product: Gradient for Databricks. Checkout our promo video here! The quick pitch Gradient is a new tool to help data engineers know when and how to optimize and lower their Databricks costs – without sacrificing performance.
Jeffrey Chou
19 Jun 2023
![]()
Blog
