Forum Discussion
Load txt pole (|) separated file to delta table
fabric is giving error when loading txt file with pole separated to delta table. any idea how to load such files?
Hello akkibuddy7
You cannot use load to table option with | delimited files. That option is for csv
you have to use spark notebook to load to tables
df = spark.read.format("csv") \
.option("header", "true") \ # Set to 'false' if no header row exists
.option("delimiter", "|") \
.load("/path/to/your/file.txt")df.write.format("delta").mode("overwrite").saveAsTable("table_name")
If this is helpful please accept the answer and give kudos
3 Replies
Hello akkibuddy7
You cannot use load to table option with | delimited files. That option is for csv
you have to use spark notebook to load to tables
df = spark.read.format("csv") \
.option("header", "true") \ # Set to 'false' if no header row exists
.option("delimiter", "|") \
.load("/path/to/your/file.txt")df.write.format("delta").mode("overwrite").saveAsTable("table_name")
If this is helpful please accept the answer and give kudos
Realized few min back the same appoch ! but your code solves the issue Thanks again!
is there any approch for incremental data? if S3 txt gets updated , will have to again run the dataframe to pull updated table and that's bit tedious job.
