New Gradient quick-start notebooks- Optimize your Databricks jobs in minutes
New Gradient quick-start notebooks- Optimize your Databricks jobs in minutes
Since we launched Gradient to help control and optimize Databricks Jobs, one piece of feedback from users was crystal clear to us: “It’s hard to set up” And we totally agreed. By nature, Gradient is a pretty deep infrastructure product that needs not only your Spark eventlogs but also cluster information from your cloud provider.
Pete Tamisin
22 Sep 2023
![]()
Blog
Why Your Data Pipelines Need Closed-Loop Feedback Control
Why Your Data Pipelines Need Closed-Loop Feedback Control
Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale
Jeffrey Chou
10 Sep 2023
![]()
Blog
Are Databricks clusters with Photon and Graviton instances worth it?
Are Databricks clusters with Photon and Graviton instances worth it?
Configuring Databricks clusters can seem more like art than science. We’ve reported in the past about ways to optimize worker and driver nodes, and how the proper selection of instances impacts a job’s cost and performance. We’ve also discussed how autoscaling performs, and how it’s not always the most efficient choice for static jobs. In
Jeffrey Chou
17 Aug 2023
![]()
Blog
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
Introduction: The Gradient Command Line Interface (CLI) is a powerful yet easy utility to automate the optimization of your Spark jobs from your terminal, command prompt, or automation scripts. Whether you are a Data Engineer, SysDevOps administrator, or just an Apache Spark enthusiast, knowing how to use the Gradient CLI can be incredibly beneficial as
Pete Tamisin
11 Jul 2023
Blog, Case Study
Integrating Gradient into Apache Airflow
Integrating Gradient into Apache Airflow
Summary In this blog post, we’ll explore how you can integrate Sync’s Gradient with Airflow. We’ll walk through the steps to create a DAG that will submit a run to Databricks, and then make a call through Sync’s library to generate a recommendation for an optimized cluster for that task. This DAG example can be
Brandon Kaplan
27 Jun 2023
![]()
Blog
