Azure DataBricks Job activity in DataPipeline doesn't change status from pending to failure when the

Hi alejandroferre 

To make ADF recognize "Succeeded with failures" as a failure, you can add logic in your ADF pipeline to check the Databricks job run’s detailed status after the activity completes. If the status is "Succeeded with failures," you can explicitly fail the pipeline.

Modify Databricks Job to Fail on Task Failures

 

1. Add the Python code to your Databricks job (e.g., last notebook/task) to check for task failures and fail the job: 
   from databricks.sdk import WorkspaceClient
   w = WorkspaceClient()
   run_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().runId().get()
   run = w.jobs.get_run(run_id=run_id)
   has_failures = any(task.state.result_state == "FAILED" for task in run.tasks)
   if has_failures:
   raise Exception("Job succeeded with failures.")
2. Keep your existing Databricks activity (e.g., RunDatabricksJob) in the ADF pipeline unchanged.
3. Run the ADF pipeline.
   • If the Databricks job has task failures, it will throw an exception and fail.
   • ADF will mark the activity as "Failed," and the pipeline will fail.

If the above information is helpful, please give us Kudos and mark the response as Accepted as solution.
Best Regards,
Community Support Team _ C Srikanth.