Databricks Workspace Health SQL Toolkit
Databricks Workspace Health SQL Toolkit
As data engineers, understanding the intricacies of your Databricks environment is important. You can’t optimize performance, budget or ensure efficient resource allocations without it. Thankfully, Databricks gives you a behind-the-scenes look at how your workspace is running in system tables. Everything from query performance to job execution and cluster activity is in those tables. But

Noa Shavit
05 Feb 2025
Blog, News
Optimizing EC2 costs on Databricks
Optimizing EC2 costs on Databricks
The global data landscape is experiencing remarkable growth, with unprecedented increases in data generation and substantial investments in analytics and infrastructure. According to data from sources like Network World and, G2 the global datasphere is projected to expand from 33 zettabytes in 2018 to an astounding 175 zettabytes by 2025, reflecting a compound annual growth

Noa Shavit
27 Jan 2025
Blog
Adding an AI agent to your data infrastructure in 2025
Adding an AI agent to your data infrastructure in 2025
Imagine a world where you could simply tell your data infrastructure what you want it to achieve, rather than meticulously configuring every detail. This is precisely what Jeff Chou, Co-founder and CEO of Sync, discussed in the latest daily.dev webinar. This innovative concept is being made real through Gradient, the AI agent for data infrastructure

Noa Shavit
16 Jan 2025
Blog, Videos
Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
According to Wavestone’s 2024 Data and AI Leadership Executive Survey, about 82.2% of data and AI leaders report that their organizations are increasing investments in data and analytics. As companies increasingly rely on big data, the significance of efficient data processing solutions and optimal configuration of clusters become even more crucial. Choosing the correct cluster

Noa Shavit
17 Dec 2024
Blog
Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks is a quickly evolving platform with several compute options available for users, leaving many with a difficult choice. In this blog post, we look at three popular options for scheduled jobs using Databricks own’ TPC-DI benchmark suite. By the way, kudos to the Databricks team for creating such a fantastic test package. We highly

Jeffrey Chou
10 Dec 2024
Blog
Integrating Gradient with Terraform
Integrating Gradient with Terraform
Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, with Terraform emerging as the leading tool for this approach. The ability to define, version, and automate infrastructure through code has transformed operations from manual, error-prone processes into systematic, repeatable workflows. However, cluster configurations defined in Terraform are typically over-provisioned to guarantee performance and

Kartik Nagappa
09 Dec 2024
Blog, Releases