How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
Introduction: The Gradient Command Line Interface (CLI) is a powerful yet easy utility to automate the optimization of your Spark jobs from your terminal, command prompt, or automation scripts. Whether you are a Data Engineer, SysDevOps administrator, or just an Apache Spark enthusiast, knowing how to use the Gradient CLI can be incredibly beneficial as
Pete Tamisin
11 Jul 2023
Blog, Case Study
Integrating Gradient into Apache Airflow
Integrating Gradient into Apache Airflow
Summary In this blog post, we’ll explore how you can integrate Sync’s Gradient with Airflow. We’ll walk through the steps to create a DAG that will submit a run to Databricks, and then make a call through Sync’s library to generate a recommendation for an optimized cluster for that task. This DAG example can be
Brandon Kaplan
27 Jun 2023
![]()
Blog
Developing Gradient Part II
Developing Gradient Part II
Introduction: Using Gradient in a Workflow Gradient, the latest product release from Sync Computing, helps customers manage the infrastructure behind their recurring Apache Spark applications. Gradient gives infrastructure recommendations for each job to lower the cost of their Production jobs while hitting their target SLA’s. We’ve been hard at work on this project for a
Sean Gorsky
19 Jun 2023
![]()
Blog
Developing Gradient Part I
Developing Gradient Part I
Introduction Sync recently introduced Gradient, a tool that helps data engineers manage and optimize their compute infrastructure. The primary facet of Gradient is a Project which groups a sequence of runs of a Databricks job. After each run, the Spark eventlog and cluster information is sent to Sync. That accumulated project data is then fed
Sean Gorsky
![]()
Blog
Introducing: Gradient for Databricks
Introducing: Gradient for Databricks
Wow the day is finally here! It’s been a long journey, but we’re so excited to announce our newest product: Gradient for Databricks. Checkout our promo video here! The quick pitch Gradient is a new tool to help data engineers know when and how to optimize and lower their Databricks costs – without sacrificing performance.
Jeffrey Chou
![]()
Blog
