topic Really strange issue - os.listdir in Data Engineering

Really strange issue - os.listdir

JP8991 — Thu, 19 Sep 2024 01:38:23 GMT

Hey all,

Could you kindly look at the below, I have a really weird one where I am referencing a location within my files area of a Lakehouse however sporadically it does not work, I have no idea why either, as it is so simple.

Re: Really strange issue - os.listdir

Anonymous — Thu, 19 Sep 2024 02:36:16 GMT

Hi @JP8991 ,

When referencing files in the lakehouse Files folder, you do not need to prefix them with /lakehouse/dafault.

As shown below, I have a 2019.csv file in the orders directory under the Files folder. When I reference it in the notebook, the path is as follows:

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Really strange issue - os.listdir

JP8991 — Thu, 19 Sep 2024 02:48:54 GMT

For the use case I need it for, which is to list what files are listed in a directory, I do not think this is correct.

Re: Really strange issue - os.listdir

Anonymous — Fri, 20 Sep 2024 02:27:10 GMT

Hi @JP8991 ,

You can use this code to list the directories in the Files folder:

import os # set folder path folder_path = "/lakehouse/default/Files/" # list directory try: directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))] # print list for directory in directories: print(directory) except FileNotFoundError: print(f"The directory {folder_path} does not exist.")

You can use this code to list the directories in the subfolder orders under the Files folder:

import os # set folder path folder_path = "/lakehouse/default/Files/orders/" # list directory try: directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))] # print list for directory in directories: print(directory) except FileNotFoundError: print(f"The directory {folder_path} does not exist.")

They all work well.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

Re: Really strange issue - os.listdir

frithjof_v — Sun, 22 Sep 2024 00:48:22 GMT

Is Xplor a file or directory?

If Xplor is a file, not a directory, then I guess the error message you're seeing makes sense.

Re: Really strange issue - os.listdir

JP8991 — Sun, 22 Sep 2024 23:28:44 GMT

I genuinely think there is some sort of bug.

I ran your code and this was my outcome:

I then ran the code three minutes later and this was my outcome which is exactly as expected:

Re: Really strange issue - os.listdir

JP8991 — Sun, 22 Sep 2024 23:29:11 GMT

Xplor is a directory.

Re: Really strange issue - os.listdir

JP8991 — Mon, 07 Oct 2024 20:59:43 GMT

@Anonymous whilst this seems to be working now on my end I don't think any of these are the solution so it does not make sense to mark them as such.

Anyone looking at this in the future will not have the correct answer.

Re: Really strange issue - os.listdir

mjam0410 — Thu, 17 Oct 2024 09:51:17 GMT

Hi,

I've got the same issue:

When listing "Files", the folder "rawdev" is shown correctly

Some minutes later, listing the content of the specified folder works...

Re: Really strange issue - os.listdir

JP8991 — Sun, 20 Oct 2024 20:29:21 GMT

This is really frustrating that no one at MS seems to be listening here.

My pipeline failed again this morning due to this problem and I have to manually go in and keep refreshing the particular notebook that is affected until it works.

Re: Really strange issue - os.listdir

frithjof_v — Sun, 20 Oct 2024 23:16:21 GMT

Hi,

I suggest creating a support ticket to raise the issue.

Have you tried Notebookutils File System Utilities btw?

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#file-system-utilities

I haven't tried it myself but perhaps that is an option.

Re: Really strange issue - os.listdir

frankmroberts — Tue, 05 Nov 2024 22:56:07 GMT

I am attempting the same thing and used the code above and am getting the same response "The directory **** does not exist."

Re: Really strange issue - os.listdir

gregahren — Mon, 11 Nov 2024 21:42:53 GMT

Hi all,

I am also getting some of the same issues as you are decribing here.

I found out that when working with os module and absolute paths(eg: '/lakehouse/default/Files') like 'os.listdir' these exceptions usually occur after uploading some files/folder while running a session with default lakehouse.

What I imagine the issue is that fabric filesystem runs on top of blobfuse2 and the change you do is not imediately reflected to the users mounted filesystem under /lakehouse/default/Files.

The workaround is to use notebookutils.fs module to list dir since this module reflects the changes imediatelly somehow. You can also use this module to read the file content for example: notebookutils.fs.head('Files/filetoread.txt', maxBytes=10000)

or to list files under files directory: notebookutils.fs.ls('Files/')

Still I think this is a bug so the microsoft team should address it.

Hope it helps someone.

Re: Really strange issue - os.listdir

Brunovs79_cimen — Tue, 04 Feb 2025 15:46:18 GMT

In this example, I use the notebookutils to loop through the files in a dir and get only the name without the extension.
from notebookutils.fs import ls

import os

application_path = 'abfss://#########@onelake.dfs.fabric.microsoft.com/000000000000/Files/folder/'

files_list = []

for file in ls(os.path.join(application_path,'excel_files_from_anp')):

file_name = file.name

if file_name.endswith('.xlsx'😞

files_list.append(file_name.split('.')[0])

for some reason os.listdir doesn't work in a pipeline or with abfss path.
Also ls is not allowed to return or anything or become an object.

print the dir of file and you will see some properties like name, size...