Databricks Connection Fails Only in Fabric Pipeline

AKemper 

ODBC in ADF only works if:

1. You install the Databricks ODBC driver, and

2. You run the pipeline on a self-hosted IR where that driver is installed.

Here the steps:

• Set up a self-hosted Integration Runtime (SHIR)

  • Install it on a VM (e.g. in Azure or on-prem).

  • Docs: Install SHIR

• Install the Databricks ODBC driver on that VM

  • Download from Databricks → Databricks ODBC driver.

  • Install it like any other ODBC driver.

• Create a System DSN (Windows ODBC Data Source Administrator)

  • Configure it with:

    • Host(s) = your Databricks SQL hostname (e.g. adb-12345.6.azuredatabricks.net)

    • Port = 443

    • HTTP Path = your SQL warehouse path (from Databricks)

    • Authentication = Token

    • Token = your PAT

• Configure your Linked Service in ADF

  • Type = ODBC

  • Runtime = your self-hosted IR (not auto-resolve / default)

  • Connection string = reference the DSN you created, e.g.:

     

    Driver={Simba Spark ODBC Driver}; Host=<your-databricks-sql-hostname>; Port=443; HTTPPath=<your-sql-http-path>; AuthMech=3; UID=token; PWD=<your-pat>; SSL=1; ThriftTransport=2;

    (note: depending on driver version, some params may be optional)

• Test connection

  • Should succeed now, because the driver is present on the SHIR.

if you cannot use JDBC and you cannot install a self-hosted IR (so ODBC won’t work, because Microsoft-hosted IR has no Databricks ODBC driver), then unfortunately you’ve hit a real limitation in ADF/Synapse.

The best workaround is to use Dataflows instead of Copy Data

• You already said your existing connection works fine in Dataflows.

• If your main goal is just to move data, you could orchestrate with pipelines but perform the copy with Mapping Dataflows instead of Copy Data.

• Dataflows do support Databricks SQL directly.

BBF

💡 Did I answer your question? Mark my post as a solution!

👍 Kudos are appreciated

🔥 Proud to be a Super User!