topic Re: How do you send mssparkutils.fs.ls() output to a dataframe in Data Engineering

How do you send mssparkutils.fs.ls() output to a dataframe

PeteSpillane — Fri, 29 Mar 2024 21:38:51 GMT

I can loop through the output using

files = mssparkutils.fs.ls('Files/orders/') for file in files: print(file.name, file.isDir, file.isFile, file.path, file.size)

But how do I send the output to a dataframe instead?

Re: How do you send mssparkutils.fs.ls() output to a dataframe

kriscoupe — Mon, 01 Apr 2024 00:14:32 GMT

Hi PeteSpillane ,

You can do this with the following code

from notebookutils import mssparkutils # Initialise variables data = [] columns = ["File Name", "Is Dir", "Is File", "File Path", "File Size"] files = mssparkutils.fs.ls('Files/orders/') # Add rows to lists for file in files: data.append([file.name, file.isDir, file.isFile, file.path, file.size]) # Create a dataframe dataframe = spark.createDataFrame(data, columns) # Show data frame dataframe.show()

Tested my side in Fabric notebook and all seemed to work okay.

Hope it helps,

Kris

Re: How do you send mssparkutils.fs.ls() output to a dataframe

PeteSpillane — Mon, 01 Apr 2024 19:08:33 GMT

Works perfectly. Thanks Kris!