Forum Discussion

PeteSpillane's avatar
PeteSpillane
Frequent Visitor
2 years ago
Solved

How do you send mssparkutils.fs.ls() output to a dataframe

I can loop through the output using 

 

 

files = mssparkutils.fs.ls('Files/orders/')
for file in files:
    print(file.name, file.isDir, file.isFile, file.path, file.size)

 

 

But how do I send the output to a dataframe instead?

data engineering
  • kriscoupe's avatar
    kriscoupe
    2 years ago

    Hi PeteSpillane ,

     

    You can do this with the following code

     

    from notebookutils import mssparkutils

# Initialise variables
data = []
columns = ["File Name", "Is Dir", "Is File", "File Path", "File Size"]
files = mssparkutils.fs.ls('Files/orders/')

# Add rows to lists
for file in files:
    data.append([file.name, file.isDir, file.isFile, file.path, file.size])

# Create a dataframe 
dataframe = spark.createDataFrame(data, columns) 
  
# Show data frame
dataframe.show()

     

    Tested my side in Fabric notebook and all seemed to work okay.

     

    Hope it helps,

    Kris

2 Replies

  • kriscoupe's avatar
    kriscoupe
    Solution Supplier
    2 years ago

    Hi PeteSpillane ,

     

    You can do this with the following code

     

    from notebookutils import mssparkutils

# Initialise variables
data = []
columns = ["File Name", "Is Dir", "Is File", "File Path", "File Size"]
files = mssparkutils.fs.ls('Files/orders/')

# Add rows to lists
for file in files:
    data.append([file.name, file.isDir, file.isFile, file.path, file.size])

# Create a dataframe 
dataframe = spark.createDataFrame(data, columns) 
  
# Show data frame
dataframe.show()

     

    Tested my side in Fabric notebook and all seemed to work okay.

     

    Hope it helps,

    Kris