Forum Discussion
Solved
Run Script for all Lakehouse Files / Lakehouse files storage account key
I would like to use the Notebook to run the same script for all files stored in the Lakehouse files. How can this be done? If there´s a need to get the storage account key for lakehouse files, where ...
You can easily list all files in a Lakehouse folder from your notebook. Just make sure to connect the lakehouse in the left pane first. that will then become your default lakehouse in the notebookfile_path = f"/lakehouse/default/Files/yoursubfoldername_here"lst = os.listdir(file_path)then you can iterate through the files and apply the same logic to each file in a loop.for file in lst:--do stuffSo the trick is not to run the same notebook on each file, but to read out the files in the notebook and the run the same logic on each file.you could create a pipeline that reads out the list of files in a folder and then call a notebook and pass the file path as a parameter, but that just adds unnecessary complexity.if you're looking for the fully qualified path to where your files are in the lakehouse, then you can grab it from the properties on the Files folder
You can easily list all files in a Lakehouse folder from your notebook. Just make sure to connect the lakehouse in the left pane first. that will then become your default lakehouse in the notebook
file_path = f"/lakehouse/default/Files/yoursubfoldername_here"
lst = os.listdir(file_path)
then you can iterate through the files and apply the same logic to each file in a loop.
for file in lst:
--do stuff
So the trick is not to run the same notebook on each file, but to read out the files in the notebook and the run the same logic on each file.
you could create a pipeline that reads out the list of files in a folder and then call a notebook and pass the file path as a parameter, but that just adds unnecessary complexity.
if you're looking for the fully qualified path to where your files are in the lakehouse, then you can grab it from the properties on the Files folder
For me it only works when i use the /lakehouse/default/Files... Path.
However when i try to use it with the abfs path, do get the following Error:
FileNotFoundError: [Errno 2] No such file or directory: 'abfss://.../input'
file_path = f"abfss://.../input"
lst = os.listdir(file_path)
lst
Any idea what causes that issue?