Really strange issue - os.listdir

[JP8991]

Hey all,

Could you kindly look at the below, I have a really weird one where I am referencing a location within my files area of a Lakehouse however sporadically it does not work, I have no idea why either, as it is so simple.

[gregahren]

Hi all,

I am also getting some of the same issues as you are decribing here. 

I found out that when working with os module and absolute paths(eg: '/lakehouse/default/Files') like 'os.listdir' these exceptions usually occur after uploading some files/folder while running a session with default lakehouse. 

What I imagine the issue is that fabric filesystem runs on top of blobfuse2 and the change you do is not imediately reflected to the users mounted filesystem under /lakehouse/default/Files. 

The workaround is to use notebookutils.fs module to list dir since this module reflects the changes imediatelly somehow. You can also use this module to read the file content for example: notebookutils.fs.head('Files/filetoread.txt', maxBytes=10000)

or to list files under files directory: notebookutils.fs.ls('Files/')

Still I think this is a bug so the microsoft team should address it.

Hope it helps someone. 

[mjam0410]

Hi, 

I've got the same issue:

When listing "Files", the folder "rawdev" is shown correctly 

Some minutes later, listing the content of the specified folder works...

[JP8991]

This is really frustrating that no one at MS seems to be listening here.

My pipeline failed again this morning due to this problem and I have to manually go in and keep refreshing the particular notebook that is affected until it works.

[frithjof_v]

Hi,

I suggest creating a support ticket to raise the issue.

Have you tried Notebookutils File System Utilities btw?

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#file-system-utilities

I haven't tried it myself but perhaps that is an option.

[JP8991]

@Anonymous whilst this seems to be working now on my end I don't think any of these are the solution so it does not make sense to mark them as such.

Anyone looking at this in the future will not have the correct answer.

[frithjof_v]

Is Xplor a file or directory?

If Xplor is a file, not a directory, then I guess the error message you're seeing makes sense.

[JP8991]

Xplor is a directory.

[Brunovs79_cimen]

In this example, I use the notebookutils to loop through the files in a dir and get only the name without the extension.
from notebookutils.fs import ls

import os

application_path = 'abfss://#########@onelake.dfs.fabric.microsoft.com/000000000000/Files/folder/'

files_list = []

for file in ls(os.path.join(application_path,'excel_files_from_anp')):

        file_name = file.name

        if file_name.endswith('.xlsx'😞

            files_list.append(file_name.split('.')[0])

for some reason os.listdir doesn't work in a pipeline or with abfss path.
Also ls is not allowed to return or anything or become an object.

print the dir of file and you will see some properties like name, size...

[Anonymous]

Hi @JP8991 ,

When referencing files in the lakehouse Files folder, you do not need to prefix them with /lakehouse/dafault.

As shown below, I have a 2019.csv file in the orders directory under the Files folder. When I reference it in the notebook, the path is as follows:

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[JP8991]

For the use case I need it for, which is to list what files are listed in a directory, I do not think this is correct.

[Anonymous]

Hi @JP8991 ,

You can use this code to list the directories in the Files folder:

import os

# set folder path
folder_path = "/lakehouse/default/Files/"

# list directory
try:
    directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))]
    # print list
    for directory in directories:
        print(directory)
except FileNotFoundError:
    print(f"The directory {folder_path} does not exist.")

You can use this code to list the directories in the subfolder orders under the Files folder:

import os

# set folder path
folder_path = "/lakehouse/default/Files/orders/"

# list directory
try:
    directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))]
    # print list
    for directory in directories:
        print(directory)
except FileNotFoundError:
    print(f"The directory {folder_path} does not exist.")

They all work well.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[frankmroberts]

I am attempting the same thing and used the code above and am getting the same response "The directory **** does not exist."

[JP8991]

I genuinely think there is some sort of bug.

I ran your code and this was my outcome:

I then ran the code three minutes later and this was my outcome which is exactly as expected: