topic Re: Download Lakehouse Files in Data Engineering

Download Lakehouse Files

VIvMouret — Mon, 27 Jan 2025 09:24:36 GMT

Hi everyone!

From a notebook in the default Spark environment, I save a table in the same lakehouse as a file
I need to create an application that retrieves existing files from the Lakehouse files, in order to extract them, download them, etc...

To do this, I created and monitored the creation of an API for GraphQL, and the creation of a single-page application using React NodeJS.

Am I missing something to achieve my goal here, or do I have what I need?
While digging around in the topics, I came across these two potential solutions :

Azure Data Lake Storage Gen2
virtual network or local data gateways

What do you think I should use ? And if I'm wrong, what should I do instead ?
Thank you in advance !

Re: Download Lakehouse Files

nilendraFabric — Mon, 27 Jan 2025 11:55:27 GMT

Hi @VIvMouret

In Microsoft Fabric, data in your Lakehouse is automatically stored in OneLake (backed by Azure Data Lake Storage Gen2). Since you already created a GraphQL API layer, you should be able to query and download files through that endpoint. Your single-page React application can call the GraphQL API to list, extract, and download Lakehouse files.If your application and the API run within Microsoft Fabric or an environment that has direct access to Fabric resources, you do not need additional services.

https://learn.microsoft.com/en-us/fabric/data-engineering/connect-apps-api-graphql

https://community.fabric.microsoft.com/t5/Data-Science/How-to-get-lakehouse-files-into-Azure-Function/m-p/4073966

OneLake provides open access to Fabric items using ADLS Gen2-compatible APIs:

Use OneLake URIs to reference files in your Lakehouse, e.g., https://onelake.dfs.fabric.microsoft.com/<workspace>/<item>.lakehouse/<path>/<fileName>

Assign appropriate permissions to your application in Azure (e.g., "Storage Blob Data Reader" role for ADLS Gen2)

Thanks

Re: Download Lakehouse Files

nilendraFabric — Mon, 27 Jan 2025 10:45:34 GMT

@VIvMouret Please accept the solution if this resolves your query, as it will help community to find the answer quickly

Re: Download Lakehouse Files

VIvMouret — Mon, 27 Jan 2025 10:52:09 GMT

thank you for your quick reply!
I'm going to test and try with ADLS Gen-2 as I already have the GraphQL API ready.

Re: Download Lakehouse Files

nilendraFabric — Mon, 27 Jan 2025 10:54:44 GMT

Thanks @VIvMouret please keep me posted. This is intresting to learn about this usecase.

Re: Download Lakehouse Files

VIvMouret — Mon, 27 Jan 2025 11:21:18 GMT

I've just tested your code, but I can't access the "files" properties

When I search in "Get data", I don't have direct access to the Lakehouse files
Will I normally be able to view the files in the Lakehouse?

Because I can't see them..

I still need to check that I have read-only permissions for the API

Re: Download Lakehouse Files

nilendraFabric — Mon, 27 Jan 2025 11:44:16 GMT

Hi @VIvMouret

I have tried the Graphql query too, its not working. So you are correct here that it is not supported, I am trying few other things will share soon.

And it seems like you have only access to tables from GraphQL api , not to files. SO we have to figure out different approach to query files.

Thanks

Re: Download Lakehouse Files

nilendraFabric — Mon, 27 Jan 2025 11:52:16 GMT

@VIvMouret

Try this approach instead of Graphql :

Use the Microsoft Fabric REST API. This allows you to programmatically access and download files by authenticating your app via Microsoft Entra (Azure AD). The process involves:

App Registration: Register your application in Microsoft Entra ID, assign permissions (e.g., Lakehouse.Read.All), and obtain credentials (Client ID, Client Secret, Tenant ID).
Authentication: Use these credentials to obtain an access token for API requests.
File Access: Use the Fabric REST API to list files in the Lakehouse and construct URLs for downloading them.

https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-api

Something like this :

import requests

file_url = "https://onelake.dfs.fabric.microsoft.com/<workspace>/<lakehouse>.lakehouse/path/to/file"
headers = {"Authorization": f"Bearer {token}"}

response = requests.get(file_url, headers=headers)
if response.status_code == 200:
with open("downloaded_file", "wb") as file:
file.write(response.content)
print("File downloaded successfully!")
else:
print(f"Failed to download file: {response.status_code}")

See if this is working

Thanks

Re: Download Lakehouse Files

VIvMouret — Tue, 28 Jan 2025 08:39:22 GMT

@nilendraFabric

I've already followed your first paragraph as a POC

To give you an idea of my code, I made this documentation at the beginning:

https://learn.microsoft.com/en-us/fabric/data-engineering/connect-apps-api-graphql
Then I moved on to this one as I was asked along the way:
https://learn.microsoft.com/en-us/entra/identity-platform/tutorial-single-page-app-react-prepare-spa?tabs=visual-studio-code

The documentation link you sent me doesn't include a query for downloading a file from a Lakehouse,
And even less in Python, I'm in NodeJS...

Re: Download Lakehouse Files

VIvMouret — Tue, 28 Jan 2025 16:57:16 GMT

Update of my problem:
All the technologies I've seen above (unless I'm mistaken) only serve to update and send new data to a Lakehouse, and that's not what we're looking for here.

To answer my question, which is :
I need to make the files I've created available to colleagues

Well, to do this, I changed direction: OneLake

I had to assign specific and restricted roles, install OneLake on all the machines so that each PC then had access to the desired folder
By pushing the files into a new Lakehouse, a new workspace and managing the roles, I was able to export the data.

However, I'm only keeping all this for development, I wouldn't recommend it for production.

I have other ideas for putting the export of Lakehouse files into production, such as with Power Pages, but as I can't find any way of extracting the files at the moment, the idea has been put off.

If anyone comes up with an alternative method, don't hesitate !