Forum Discussion
Really strange issue - os.listdir
Hey all,
Could you kindly look at the below, I have a really weird one where I am referencing a location within my files area of a Lakehouse however sporadically it does not work, I have no idea why either, as it is so simple.
13 Replies
Anonymous whilst this seems to be working now on my end I don't think any of these are the solution so it does not make sense to mark them as such.
Anyone looking at this in the future will not have the correct answer.Hi JP8991 ,
When referencing files in the lakehouse Files folder, you do not need to prefix them with /lakehouse/dafault.
As shown below, I have a 2019.csv file in the orders directory under the Files folder. When I reference it in the notebook, the path is as follows:
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!For the use case I need it for, which is to list what files are listed in a directory, I do not think this is correct.
Hi JP8991 ,
You can use this code to list the directories in the Files folder:
import os # set folder path folder_path = "/lakehouse/default/Files/" # list directory try: directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))] # print list for directory in directories: print(directory) except FileNotFoundError: print(f"The directory {folder_path} does not exist.")You can use this code to list the directories in the subfolder orders under the Files folder:
import os # set folder path folder_path = "/lakehouse/default/Files/orders/" # list directory try: directories = [d for d in os.listdir(folder_path) if os.path.isdir(os.path.join(folder_path, d))] # print list for directory in directories: print(directory) except FileNotFoundError: print(f"The directory {folder_path} does not exist.")They all work well.
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Is Xplor a file or directory?
If Xplor is a file, not a directory, then I guess the error message you're seeing makes sense.
Xplor is a directory.
- In this example, I use the notebookutils to loop through the files in a dir and get only the name without the extension.
from notebookutils.fs import lsimport os
application_path = 'abfss://#########@onelake.dfs.fabric.microsoft.com/000000000000/Files/folder/'
files_list = []
for file in ls(os.path.join(application_path,'excel_files_from_anp')):
file_name = file.nameif file_name.endswith('.xlsx'😞files_list.append(file_name.split('.')[0])
for some reason os.listdir doesn't work in a pipeline or with abfss path.
Also ls is not allowed to return or anything or become an object.print the dir of file and you will see some properties like name, size...
Hi,
I've got the same issue:
When listing "Files", the folder "rawdev" is shown correctly
Some minutes later, listing the content of the specified folder works...
This is really frustrating that no one at MS seems to be listening here.
My pipeline failed again this morning due to this problem and I have to manually go in and keep refreshing the particular notebook that is affected until it works.Hi,
I suggest creating a support ticket to raise the issue.
Have you tried Notebookutils File System Utilities btw?
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#file-system-utilities
I haven't tried it myself but perhaps that is an option.
Hi all,
I am also getting some of the same issues as you are decribing here.
I found out that when working with os module and absolute paths(eg: '/lakehouse/default/Files') like 'os.listdir' these exceptions usually occur after uploading some files/folder while running a session with default lakehouse.
What I imagine the issue is that fabric filesystem runs on top of blobfuse2 and the change you do is not imediately reflected to the users mounted filesystem under /lakehouse/default/Files.
The workaround is to use notebookutils.fs module to list dir since this module reflects the changes imediatelly somehow. You can also use this module to read the file content for example: notebookutils.fs.head('Files/filetoread.txt', maxBytes=10000)
or to list files under files directory: notebookutils.fs.ls('Files/')
Still I think this is a bug so the microsoft team should address it.
Hope it helps someone.
