Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Did you hear? There's a new SQL AI Developer certification (DP-800). Start preparing now and be one of the first to get certified. Register now

guyhay1
‎12-05-2024 03:11 PM

Announcing the public preview of the REST API for Livy for Data Engineering.

The Fabric Livy endpoint lets users submit and execute their Spark code on the Spark compute within a designated Fabric workspace, eliminating the need to create any Notebook or Spark Job Definition artifacts. This integration with a specific Lakehouse artifact ensures straightforward access to data stored on OneLake. Additionally, Livy API offers the ability to customize the execution environment through its integration with the Environment artifact.

When a request is sent to the Fabric Livy endpoint, the user-submitted code can be executed in two different modes:

Session Job:

  • A Livy session job entails establishing a Spark session that remains active throughout the interaction with the Livy API. This is particularly useful for interactive and iterative workloads.
  • A Spark session starts when a job is submitted and lasts until the user ends it or the system terminates it after 20 minutes of inactivity. Throughout the session, multiple jobs can run, sharing state and cached data between runs.

Batch Job:

  • A Livy batch job entails submitting a Spark application for a single execution. In contrast to a Livy session job, a batch job does not sustain an ongoing Spark session.
  • With Livy batch jobs each job initiates a new Spark session, which ends when the job finishes. This approach works well for tasks that don't rely on previous computations or require maintaining state between jobs.

The endpoint of the session job API would look like:

https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/ batches

The endpoint of the session job API would look like: https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/sessions

ws_id: this is the workspace in which the hosting Lakehouse artifact belongs to. lakehouse_id: this is the artifact id of the hosting Lakehouse. All the capacity consumption history from this batch API call will be associated with this artifact.

To access the Livy API endpoint, you need to create a Lakehouse artifact. After it's set up, you'll find the Livy API endpoint in the settings panel.

Announcing_the_public_preview_of_the_REST_API_for_Livy_for_Data_EngineeringAnnouncing_the_public_preview_of_the_REST_API_for_Livy_for_Data_Engineering

Lakehouse settings showing the Livy endpoint

Our Livy API documentation https://learn.microsoft.com/fabric/data-engineering/api-livy-overview provides more details, and shows you how to create your first Livy batch or session job.

0