Forum Discussion
AzureDatabricksWorkspace connector doesn't work in Pipeline with On-Prem Gateway?
Hello fellow Fabric users.
I'm trying to create a Fabric Pipeline step that would trigger a job in Azure Databricks workspace using the built in activity for that (image #3). Our ADB workspace is accessible only through On-Prem gateway connection, querying from Databricks SQL connector works well through the gateway, but I need to be able to also trigger things when needed from Fabric.
BUT, for some reason the activity settings do not "see" the gateway connection I created beforehand (image #1) and also when I try to create a brand new connection using built-in dialog, it completely ignores any gateways when I type in ADB workspace URL (image #2).
Please advise on how to solve this 🤔 thank you!
Hi zllll ,
Thank you for reaching out to fabric community.
At present, the Azure Databricks activity in Fabric Pipelines does not appear to support using an On-premises Data Gateway connection.
The Azure Databricks connector overview lists the supported pipeline capabilities as Copy activity and Lookup activity. It does not mention support for the Azure Databricks Job activity or the use of gateway-backed connections with that activity.
Documentation: https://learn.microsoft.com/en-us/fabric/data-factory/connector-azure-databricks-overview
As a workaround, you can:
- Ensure the Fabric Pipeline can reach the Azure Databricks workspace directly through a supported public or private networking configuration.
- Trigger the Databricks job indirectly by using a Web activity to call the Databricks Jobs REST API, or by using an intermediary service (such as an Azure Function or Logic App) that has network connectivity to the Databricks workspace.
Based on the current documentation and your screenshots, this appears to be an unsupported scenario rather than a configuration issue. If your scenario requires the Azure Databricks activity to support On-premises Data Gateway connections, you may consider submitting a feature request through the Microsoft Fabric Ideas portal( https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas )
Thanks!!
1 Reply
Hi zllll ,
Thank you for reaching out to fabric community.
At present, the Azure Databricks activity in Fabric Pipelines does not appear to support using an On-premises Data Gateway connection.
The Azure Databricks connector overview lists the supported pipeline capabilities as Copy activity and Lookup activity. It does not mention support for the Azure Databricks Job activity or the use of gateway-backed connections with that activity.
Documentation: https://learn.microsoft.com/en-us/fabric/data-factory/connector-azure-databricks-overview
As a workaround, you can:
- Ensure the Fabric Pipeline can reach the Azure Databricks workspace directly through a supported public or private networking configuration.
- Trigger the Databricks job indirectly by using a Web activity to call the Databricks Jobs REST API, or by using an intermediary service (such as an Azure Function or Logic App) that has network connectivity to the Databricks workspace.
Based on the current documentation and your screenshots, this appears to be an unsupported scenario rather than a configuration issue. If your scenario requires the Azure Databricks activity to support On-premises Data Gateway connections, you may consider submitting a feature request through the Microsoft Fabric Ideas portal( https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas )
Thanks!!