Forum Discussion

tajjallarukshan's avatar
tajjallarukshan
Frequent Visitor
2 years ago
Solved

Issue listing/Uploading/modifying files in lakehouse using python

Summary – I have created a Lakehouse in Power BI Service and are trying to test the connectivity to the Lakehouse via Python following the documentation here: https://learn.microsoft.com/en-us/fabr...
data engineering
lakehouse
notebook
  • tajjallarukshan's avatar
    tajjallarukshan
    2 years ago

    Hello,

     

    I have found the cause of it, and it has to do with the nomenclature you follow for naming your lakehouse, workspace and all subsequent objects. They should not have upper case letters or special characters. That solved the problem for us!

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi tajjallarukshan ,

 

The error message "azure.core.exceptions.ResourceNotFoundError:Artifact'lakehouse' is not found in workspace 'MVP'

ErrorCode:ArtifactNotFound" indicates that the Lakehouse project cannot be found in the specified workspace.

 

Please check the following aspects:

 

  • Double-check the names of your workspace and Lakehouse. Make sure there are no typos or extra spaces.

 

  • Verify that the Lakehouse project actually exists in the specified workspace. It is possible that lakehouse is in another workspace.

 

If you have any other questions please feel free to contact me.

 

Best Regards,
Yang
Community Support Team

 

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

  • tajjallarukshan's avatar
    tajjallarukshan
    Frequent Visitor
    2 years ago

    Hi Yang,

     

    I had updated the code posted here with different names to maintain integrity of the organization.

    I have validated the names for any typos or additional space, but thats not the case. 

     

    Copying workspace name from the portal and ensuring to copy name of lakehouse as is has been my approach. So there is no chance of such misses. 

     

     

    However, Here is my detailed analysis upon figuring out where the code is creating a problem:

     


    import os
    from azure.identity import DefaultAzureCredential
    from azure.storage.filedatalake import DataLakeServiceClient

    # Set your account name and workspace information here
    ACCOUNT_NAME = "onelake"
    DATA_PATH = "LH_MVP_1.Lakehouse/Files/Processed/"

    def list_file_systems(service_client😞
        try:
            # Listing file systems might be restricted. We can ensure to handle this error or validate this part
            file_systems = service_client.list_file_systems()
            for fs in file_systems:
                print("File system name:", fs.name)
        except Exception as e:
            print("An error occurred while listing file systems:", str(e))

    def list_paths(file_system_client😞
        try:
            # List paths within the specified folder
            paths = file_system_client.get_paths(path=DATA_PATH)
            print(f"Listing paths:")
            for path in paths:
                print("Path name:", path.name)
        except Exception as e:
            print(f"An error occurred while listing paths", str(e))

    def main(file_system_name😞
        # Create a service client using the default Azure credential
        account_url = f"https://{ACCOUNT_NAME}.dfs.fabric.microsoft.com"
        token_credential = DefaultAzureCredential()
        service_client = DataLakeServiceClient(account_url, credential=token_credential)
       
        # List file systems to test if the service client works fine
        list_file_systems(service_client)

        # Create a file system client
        file_system_client = service_client.get_file_system_client(file_system_name)
     
         # List file systems to test if the service client works fine
        list_paths(file_system_client)
       

    if __name__ == "__main__":
           main("FABRIC MVP 1")

    Output:
     

    So this shows:

    1. Authentication is successful

    2. All workspaces are being listed as file systems successfully

    3. But error appears while attempting to list paths inside a specific workspace

    Do you know if the format in which the data path is passed is correct here "file_system_client.get_paths(path=DATA_PATH)"?

    My use case is to upload files to the lakehouse via Python. I followed the approach in this link, and I was able to successfully list/ upload and update files in lakehouse 2 months ago. But the same code abruptly stopped working just after a week of its successful run. I have a ticket with Microsoft for the same, but its not heading in any direction to get this working.

    • JDT's avatar
      JDT
      Frequent Visitor
      2 years ago

      Hi, 

      have you found a solution to this problem? We experiencing exactly the same thing. Our Python Scripts suddenly stop working without changing anything on our site.

      • tajjallarukshan's avatar
        tajjallarukshan
        Frequent Visitor
        2 years ago

        Hello,

         

        I have found the cause of it, and it has to do with the nomenclature you follow for naming your lakehouse, workspace and all subsequent objects. They should not have upper case letters or special characters. That solved the problem for us!