Forum Discussion
Solved
Lakehouse - promote csv files to Tables using For Loop
I'm pretty new to Notebooks ... I trying to promote the 5 csv files in the Lakehouse file directory Lakehouse/Files/Imported to Tables in the Lakehouse Ideally I would like to use a For Loop (t...
Hi Theo86 ,
You can do like this:
#Read Files inside Directory files = mssparkutils.fs.ls("Files/Imported/") # Each item contains file name and path for file in files: # Skip folders inside if not file.isDir: table_name = file.name.replace(".csv","") df = spark.read.option("header", "true").option("inferSchema", "true").csv(file.path) df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(f"<schema>.{table_name}")You can add a few print statement to keep track of the flow while the code runs. Also you can add try, except block for error-handling.
stoic-harsh
Super User
Super UserHi Theo86 ,
You can do like this:
#Read Files inside Directory
files = mssparkutils.fs.ls("Files/Imported/")
# Each item contains file name and path
for file in files:
# Skip folders inside
if not file.isDir:
table_name = file.name.replace(".csv","")
df = spark.read.option("header", "true").option("inferSchema", "true").csv(file.path)
df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(f"<schema>.{table_name}")You can add a few print statement to keep track of the flow while the code runs. Also you can add try, except block for error-handling.