topic Re: How to write data into fabric lakehouse from Databricks notebook in Pipelines

How to write data into fabric lakehouse from Databricks notebook

Anonymous — Mon, 11 Mar 2024 15:53:56 GMT

When trying to access one lake Microsoft fabric (lakehouse) from databricks notebook to read data froom adls and write into fabric lakehouse getting this error: path has invalid authority error while reading and writing files to lakehouse
abfss://onelake.dfs.fabric.microsoft.com/path(actual path of file from lakehouse i am giving) has invalid authority.

I have followed steps from : https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks

created cluster and enable Azure Data Lake Storage (ADLS) credential passthrough on your cluster in the Advanced Options, but i have databricks premium account for different microsoft account and fabric workspace on different microsoft account.

Is it because of that? do i need to create fabric and databricks account for same Microsoft ID, if i wanted to use different account then what is the way to do that? please help

Re: How to write data into fabric lakehouse from Databricks notebook

v-cboorla-msft — Tue, 12 Mar 2024 06:53:32 GMT

Hi Anonymous

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience.

You're correct that having your Databricks workspace and Fabric workspace on different Azure Active Directory (AAD) tenants can cause the "invalid authority" error you're encountering.

While Microsoft Fabric offers seamless integration with Azure Databricks within the same tenant, there are currently limitations when working across different tenants. This means you cannot directly access an ADLS Gen2 storage account managed by a separate AAD tenant from your Databricks workspace, even with a premium account.

If you’re using different Microsoft accounts for Databricks and the fabric workspace, you might need to set up a service principal with the necessary permissions on the ADLS resource and configure Databricks to use this service principal for authentication.

For more details please refer to the following documents.

Link1 : Connect to Azure Data Lake Storage Gen2 and Blob Storage.

Link2. Access Azure Data Lake Storage using Microsoft Entra ID (formerly Azure Active Directory) credential passthrough (legacy)

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

Re: How to write data into fabric lakehouse from Databricks notebook

v-cboorla-msft — Tue, 12 Mar 2024 08:41:31 GMT

Hi Anonymous

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.

Thanks.