topic Listing files using REST API in Lakehouse in Data Engineering

Listing files using REST API in Lakehouse

munindra — Mon, 20 Jan 2025 05:45:48 GMT

i am trying to list the file of lakehouse using reatapi but it is not working but when i am trying using tables it is propely working
and i am following this Items - List Lakehouses - REST API (Lakehouse) | Microsoft Learn documentation for listing tables

i am using this url for table:https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses/{lakehouseId}/Tables
and i am getting proper result

similarly when i am trying with https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses/{lakehouseId}/Files
for files

Re: Listing files using REST API in Lakehouse

nilendraFabric — Mon, 20 Jan 2025 07:30:11 GMT

Hi munindra

The issue you’re encountering is due to the differences in API endpoints for listing tables versus files in a Microsoft Fabric Lakehouse.
For listing files in a Lakehouse, you need to use a different API endpoint. The correct approach is to use the Azure Data Lake Storage Gen2 REST API, as the Lakehouse files are stored in OneLake, which is compatible with ADLS Gen2.

To list files, you should use an endpoint similar to this

https://onelake.dfs.fabric.microsoft.com/{workspaceId}/{lakehouseId}.lakehouse/Files?resource=filesystem&recursive=true

You’ll need to include the appropriate authentication headers, typically using a bearer token obtained for the ‘storage’ scope.

use this approach and see

import requests

workspace_id = "your_workspace_id"
lakehouse_id = "your_lakehouse_id"
token = mssparkutils.credentials.getToken("storage")

url = f"https://onelake.dfs.fabric.microsoft.com/{workspace_id}/{lakehouse_id}.lakehouse/Files?resource=filesystem&recursive=true"
headers = {"Authorization": f"Bearer {token}"}

response = requests.get(url, headers=headers)
files = response.json()

please accept the solution and give kudos if this is helpful.

Thanks

Nilendra

Re: Listing files using REST API in Lakehouse

nilendraFabric — Mon, 20 Jan 2025 07:30:49 GMT

https://onelake.dfs.fabric.microsoft.com/{workspaceId}/{lakehouseId}.lakehouse/Files?resource=filesystem&recursive=true

Re: Listing files using REST API in Lakehouse

Anonymous — Tue, 28 Jan 2025 06:44:32 GMT

Hi munindra
Thank you for reaching out microsoft fabric community forum.
I wanted to check if you had the opportunity to review the information provided by nilendraFabric . Please feel free to contact us if you have any further questions. If his response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Listing files using REST API in Lakehouse

Anonymous — Tue, 04 Feb 2025 09:36:26 GMT

Hi nilendraFabric
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If his response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Listing files using REST API in Lakehouse

Anonymous — Tue, 25 Feb 2025 12:01:11 GMT

Hi munindra ,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.