topic Re: Issue listing/Uploading/modifying files in lakehouse using python in Data Engineering

Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Mon, 29 Jul 2024 10:14:37 GMT

Summary –

I have created a Lakehouse in Power BI Service and are trying to test the connectivity to the Lakehouse via Python following the documentation here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-access-python.
To achieve this, I am using the below code. This should give the output of list of the files that exist in that specific Lake House.

Code:

# Install the correct packages first in the same folder as this file.

# pip install azure-storage-file-datalake azure-identity

from azure.storage.filedatalake import DataLakeServiceClient, FileSystemClient

from azure.identity import DefaultAzureCredential

# Set your account, workspace, and item path here

ACCOUNT_NAME = "onelake"

WORKSPACE_NAME = "FABRIC MVP 1 "

DATA_PATH = "/LH_MVP_1.lakehouse/Files/Processed"

def main():

# Create a service client using the default Azure credential

account_url = fhttps://{ACCOUNT_NAME}.dfs.fabric.microsoft.com

token_credential = DefaultAzureCredential()

service_client = DataLakeServiceClient(account_url, credential=token_credential)

# Create a file system client for the workspace

file_system_client = service_client.get_file_system_client(WORKSPACE_NAME)

# List a directory within the filesystem

paths = file_system_client.get_paths(path=DATA_PATH)

for path in paths:

print(path.name + '\n')

if __name__ == "__main__":

main()

When I run the code in VS Code its throwing below error –
I am using service principle, I have relevant client tenants and secrets which have been given and have also given access to the Lake house.

Anyone facing similar issue and suggestions to solve for this please?

Re: Issue listing/Uploading/modifying files in lakehouse using python

Anonymous — Tue, 30 Jul 2024 01:34:28 GMT

Hi tajjallarukshan ,

The error message "azure.core.exceptions.ResourceNotFoundError:Artifact'lakehouse' is not found in workspace 'MVP'

ErrorCode:ArtifactNotFound" indicates that the Lakehouse project cannot be found in the specified workspace.

Please check the following aspects:

Double-check the names of your workspace and Lakehouse. Make sure there are no typos or extra spaces.

Verify that the Lakehouse project actually exists in the specified workspace. It is possible that lakehouse is in another workspace.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Tue, 30 Jul 2024 07:33:50 GMT

Hi Yang,

I had updated the code posted here with different names to maintain integrity of the organization.

I have validated the names for any typos or additional space, but thats not the case.

Copying workspace name from the portal and ensuring to copy name of lakehouse as is has been my approach. So there is no chance of such misses.

However, Here is my detailed analysis upon figuring out where the code is creating a problem:

import os

from azure.identity import DefaultAzureCredential

from azure.storage.filedatalake import DataLakeServiceClient

# Set your account name and workspace information here

ACCOUNT_NAME = "onelake"

DATA_PATH = "LH_MVP_1.Lakehouse/Files/Processed/"

def list_file_systems(service_client😞

try:

# Listing file systems might be restricted. We can ensure to handle this error or validate this part

file_systems = service_client.list_file_systems()

for fs in file_systems:

print("File system name:", fs.name)

except Exception as e:

print("An error occurred while listing file systems:", str(e))

def list_paths(file_system_client😞

try:

# List paths within the specified folder

paths = file_system_client.get_paths(path=DATA_PATH)

print(f"Listing paths:")

for path in paths:

print("Path name:", path.name)

except Exception as e:

print(f"An error occurred while listing paths", str(e))

def main(file_system_name😞

# Create a service client using the default Azure credential

account_url = f"https://{ACCOUNT_NAME}.dfs.fabric.microsoft.com"

token_credential = DefaultAzureCredential()

service_client = DataLakeServiceClient(account_url, credential=token_credential)

# List file systems to test if the service client works fine

list_file_systems(service_client)

# Create a file system client

file_system_client = service_client.get_file_system_client(file_system_name)

# List file systems to test if the service client works fine

list_paths(file_system_client)

if __name__ == "__main__":

main("FABRIC MVP 1")

Output:

So this shows:

1. Authentication is successful

2. All workspaces are being listed as file systems successfully

3. But error appears while attempting to list paths inside a specific workspace

Do you know if the format in which the data path is passed is correct here "file_system_client.get_paths(path=DATA_PATH)"?

My use case is to upload files to the lakehouse via Python. I followed the approach in this link, and I was able to successfully list/ upload and update files in lakehouse 2 months ago. But the same code abruptly stopped working just after a week of its successful run. I have a ticket with Microsoft for the same, but its not heading in any direction to get this working.

Re: Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Wed, 31 Jul 2024 12:24:03 GMT

There is this post in this link - How to use service principal authentication to access Microsoft Fabric's OneLake (dataroots.io). Can you help me understand if there are these settings at the tenant level that need to be enabled for the service principles to interact appropriately with the OneLake?

Re: Issue listing/Uploading/modifying files in lakehouse using python

JDT — Thu, 29 Aug 2024 09:56:12 GMT

Hi,

have you found a solution to this problem? We experiencing exactly the same thing. Our Python Scripts suddenly stop working without changing anything on our site.

Re: Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Thu, 29 Aug 2024 10:59:13 GMT

Hello,

I have found the cause of it, and it has to do with the nomenclature you follow for naming your lakehouse, workspace and all subsequent objects. They should not have upper case letters or special characters. That solved the problem for us!

Re: Issue listing/Uploading/modifying files in lakehouse using python

Narayan212 — Wed, 25 Sep 2024 10:54:47 GMT

Hi tajjallarukshan, I'm having the same problem. Even after your comment I tried to create another lakehouse with lower case letters and not special characters and still unable to access or list the paths. I am using the client id, secret and tenant id for the authentication. I am able to print file system name as you are trying in the code and it is working fine. But once I try to upload any file or try to list path then I am getting the same error. Can you please help me out here

Re: Issue listing/Uploading/modifying files in lakehouse using python

JDT — Tue, 01 Oct 2024 11:07:09 GMT

Hi,

what I have found out is, that whitespaces,* and | are evil. Do not use them in the workspace and lakehouse name. I couldnt find a clear rule. Even some names that are compliant with ADLS2's naming reference broke for me - naming-and-referencing-containers--blobs--and-metadata.

Re: Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Tue, 01 Oct 2024 11:18:53 GMT

Hello,

My case involved capital letters and spaces in the nomenclature of lakehouse and also the workspace. When I converted everything to lower case without any spaces, it worked absolutely fine.

This must be with nomenclature only - check for spaces, capital letters and special characters and ensure you dont use these.

Re: Issue listing/Uploading/modifying files in lakehouse using python

tajjallarukshan — Tue, 01 Oct 2024 11:19:59 GMT

Try creating a new workspace and new lakehouse, add this service principle to the workspace with contributr rights and try again.