topic Re: how to connect to fabric lakhouse using duckdb from databricks premium workspace? in Data Engineering

how to connect to fabric lakhouse using duckdb from databricks premium workspace?

sleepeatazure — Tue, 21 May 2024 09:12:26 GMT

how can I access data from fabric datalake from databricks using duckdb? I am using databricks premium workspace and can read and wite using spark. But when trying to connect through duck db I am getting the below error (I am not using any SPN or access key for duckdb client):

IOException: IO Error: AzureBlobStorageFileSystem could not open file: 'abfss://demo@onelake.dfs.fabric.microsoft.com/Lakehouse.Lakehouse/Files/20180402040501.CSV', unknown error occurred, this could mean the credentials used were wrong. Original error message: 'Fail to get a new connection for: https://demo@onelake.blob.fabric.microsoft.com. Problem with the SSL CA cert (path? access rights?)'

Re: how to connect to fabric lakhouse using duckdb from databricks premium workspace?

v-cboorla-msft — Tue, 21 May 2024 16:37:56 GMT

Hi @sleepeatazure

Thanks for using Microsoft Fabric Community.

Please refer to the documentation where you might get some insights on how to connect to fabric lakhouse using duckdb.

Document : Exploring Python Libraries for Data Engineering in MS Fabric: Pandas, Polars, DuckDB, and PySpark | by Mariusz Kujawski | May, 2024 | Medium

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

Re: how to connect to fabric lakhouse using duckdb from databricks premium workspace?

v-cboorla-msft — Thu, 23 May 2024 16:57:04 GMT

Hi @sleepeatazure

Thanks.

Re: how to connect to fabric lakhouse using duckdb from databricks premium workspace?

v-cboorla-msft — Mon, 27 May 2024 06:21:14 GMT

Hi @sleepeatazure

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

Re: how to connect to fabric lakhouse using duckdb from databricks premium workspace?

Faisalsyad — Thu, 05 Jun 2025 07:33:23 GMT

I have same issue with sleepeatazure, my code is like this:

import duckdb # Initialize DuckDB connection conn = duckdb.connect() print("Connect success") conn.execute("SET TimeZone = 'UTC'") print("Execute timezone ke UTC success") conn.sql("LOAD '/home/path/azure.duckdb_extension';") print("Load azure.duckdb_extention success") conn.sql(""" CREATE SECRET secret4 ( TYPE AZURE, PROVIDER CREDENTIAL_CHAIN, CHAIN 'cli;env', ACCOUNT_NAME 'onelake' ); """) print("create secret credential success") conn.sql("LOAD '/home/path/delta.duckdb_extension';") print("Load delta.duckdb_extension success") print("create query") # Query SELECT dari parquet di abfss query = """ SELECT * FROM delta_scan('abfss://demo@onelake.dfs.fabric.microsoft.com/lakehouse/Tables/mapping_object'); """ print("create query success") # I also used not only delta_scan, but 'read_parquet' print("proses query start") # run query dan save result to pandas dataframe df = conn.execute(query).fetchdf() print("proses query success") print(df.head())

Error:

unknown error occurred, this could mean the credentials used were wrong. Original error message: 'Fail to get a new connection for: https://onelake.blob.fabric.microsoft.com. Problem with the SSL CA cert (path? access rights?)'

Extra:

Package Version: - Python 3.7.16
----------------- -----------
certifi 2022.12.7
deltalake 0.13.0
duckdb 1.1.3
joblib 1.3.2
numpy 1.21.6
pandas 1.3.5
pickleshare 0.7.5
pip 22.3.1
pyarrow 12.0.1
python-dateutil 2.9.0.post0
pytz 2025.2
scikit-learn 0.22
scipy 1.7.3
setuptools 65.6.3
six 1.17.0
typing_extensions 4.7.1
wheel 0.38.4

- (It’s better not to change the version, because it is compatible with the modeling in my previous infrastructure)

- And I have 2 different file, azure.duckdb_extension and delta.duckdb_extension to enable support for Azure authentication and reading Delta tables, sometimes you need to install additional packages manually if they are not included during the pip install duckdb.

I have read the Medium article you shared, but I couldn’t find any information related to DuckDB accessing ABFSS/blob storage.
Could you please let us know if you have any information or guidance regarding this issue?

Thank you.