Databricks Connection Fails Only in Fabric Pipeline

Question

Hello all!&nbsp;I am trying to create a Pipeline to get data from my Azure Databricks environment. I have an existing connection that I have used with Dataflows and Semantic Models before that works without any issue, but when I select the connection in a Copy Data activity in my pipeline I get the below error message.&nbsp;Error Message:ErrorCode=FailedToConnectToDatabricksWorkspace,Failed to connect to Databricks workspace. Error Cluster XXXXX does not exist. Cluster XXXXX does not exist Processed HTTP request failed.&nbsp;It is important to note the Server hostname and HTTP path are for a serverless SQL Warehouse in databricks, the Warehouse is running, I am authenticating with a PAT, the connection shows as "Online" in the "Manage connections and gateways" screen, and that the connection works for Dataflows and Semantic Models.&nbsp;Thank you in advance for the help!

v-priyankata · Answer

Hi&nbsp;AKemper&nbsp;
Thank you for reaching out to the Microsoft Fabric Forum Community.
BeaBF&nbsp;Thank you so much for your inputs.
I hope the information provided by users was helpful. If you still have questions, please don't hesitate to reach out to the community.
&nbsp;

beabf · Answer

AKemper&nbsp;
ODBC in ADF only works if:

You install the Databricks ODBC driver, and

You run the pipeline on a self-hosted IR where that driver is installed.

Here the steps:
&nbsp;

Set up a self-hosted Integration Runtime (SHIR)

Install it on a VM (e.g. in Azure or on-prem).

Docs: Install SHIR

Install the Databricks ODBC driver on that VM

Download from Databricks → Databricks ODBC driver.

Install it like any other ODBC driver.

Create a System DSN (Windows ODBC Data Source Administrator)

Configure it with:

Host(s) = your Databricks SQL hostname (e.g. adb-12345.6.azuredatabricks.net)

Port = 443

HTTP Path = your SQL warehouse path (from Databricks)

Authentication = Token

Token = your PAT

Configure your Linked Service in ADF

Type = ODBC

Runtime = your self-hosted IR (not auto-resolve / default)

Connection string = reference the DSN you created, e.g.:

&nbsp;

Driver={Simba Spark ODBC Driver}; Host=&lt;your-databricks-sql-hostname&gt;; Port=443; HTTPPath=&lt;your-sql-http-path&gt;; AuthMech=3; UID=token; PWD=&lt;your-pat&gt;; SSL=1; ThriftTransport=2;

(note: depending on driver version, some params may be optional)

Test connection

Should succeed now, because the driver is present on the SHIR.

if you cannot use JDBC and you cannot install a self-hosted IR (so ODBC won’t work, because Microsoft-hosted IR has no Databricks ODBC driver), then unfortunately you’ve hit a real limitation in ADF/Synapse.
The best workaround is to use Dataflows instead of Copy Data

You already said your existing connection works fine in Dataflows.

If your main goal is just to move data, you could orchestrate with pipelines but perform the copy with Mapping Dataflows instead of Copy Data.

Dataflows do support Databricks SQL directly.

&nbsp;
BBF
💡 Did I answer your question? Mark my post as a solution! 👍 Kudos are appreciated 🔥 Proud to be a Super User!  
&nbsp;

beabf · Answer

AKemper&nbsp; Hi!
Try to use the Azure Databricks (SQL) connector instead of the regular Databricks connector

In Copy Data, when selecting the source, choose:

Source type → Databricks (SQL endpoint)

Provide the server hostname, HTTP path, and PAT.

This treats it like a SQL source rather than a cluster-based notebook source.

This should work exactly like Dataflows do

BBF
💡 Did I answer your question? Mark my post as a solution! 👍 Kudos are appreciated 🔥 Proud to be a Super User!

akemper · Answer

BeaBF&nbsp;Thank you for the quick reply!&nbsp;In Copy Data when selecting a source the only Databricks option I have is Azure Databricks which is what is giving me the error.&nbsp;&nbsp;Thank you!

beabf · Answer

AKemper&nbsp;ok, in Copy Data, Databricks SQL endpoint is not exposed as a first-class source in some versions of ADF/Synapse.
&nbsp;
Try to use a&nbsp;JDBC/ODBC connector to query the SQL endpoint

Copy Data activity can use generic JDBC instead of the Azure Databricks connector.

Example JDBC URL for a SQL endpoint:

jdbc:spark://&lt;ServerHostname&gt;:443/default;
transportMode=http;
ssl=1;
httpPath=&lt;HTTPPath&gt;;
AuthMech=3;
UID=token;
PWD=&lt;PAT&gt;&nbsp;
&nbsp;
BBF
💡 Did I answer your question? Mark my post as a solution! 👍 Kudos are appreciated 🔥 Proud to be a Super User!

Forum Discussion

Databricks Connection Fails Only in Fabric Pipeline

10 Replies

Try to use the Azure Databricks (SQL) connector instead of the regular Databricks connector

You’re no longer hitting Databricks-side auth issues; you’re now hitting a driver problem on the ADF/Synapse side.
Use the JDBC connector instead of ODBC

Related content

Connection failed using a Folder connection in a pipeline

Databricks connection's in Dataflow Gen 2 vs Data Pipeline

Parameterized connections in Data pipelines

Connecting power bi to databricks

Connect warehouse lakehouse to Fabric pipeline.

Forum Discussion

Databricks Connection Fails Only in Fabric Pipeline

Try to use the Azure Databricks (SQL) connector instead of the regular Databricks connector

You’re no longer hitting Databricks-side auth issues; you’re now hitting a driver problem on the ADF/Synapse side.Use the JDBC connector instead of ODBC

Connection failed using a Folder connection in a pipeline

Databricks connection's in Dataflow Gen 2 vs Data Pipeline

Parameterized connections in Data pipelines

Connecting power bi to databricks

Connect warehouse lakehouse to Fabric pipeline.

You’re no longer hitting Databricks-side auth issues; you’re now hitting a driver problem on the ADF/Synapse side.
Use the JDBC connector instead of ODBC