topic Re: Load txt pole (|) separated file to delta table in Data Engineering

Load txt pole (|) separated file to delta table

akkibuddy7 — Wed, 05 Feb 2025 06:44:45 GMT

fabric is giving error when loading txt file with pole separated to delta table. any idea how to load such files?

Re: Load txt pole (|) separated file to delta table

nilendraFabric — Wed, 05 Feb 2025 08:24:50 GMT

You cannot use load to table option with | delimited files. That option is for csv

you have to use spark notebook to load to tables

df = spark.read.format("csv") \
.option("header", "true") \ # Set to 'false' if no header row exists
.option("delimiter", "|") \
.load("/path/to/your/file.txt")

df.write.format("delta").mode("overwrite").saveAsTable("table_name")

If this is helpful please accept the answer and give kudos

Re: Load txt pole (|) separated file to delta table

akkibuddy7 — Wed, 05 Feb 2025 09:20:12 GMT

Realized few min back the same appoch ! but your code solves the issue Thanks again!

Re: Load txt pole (|) separated file to delta table

akkibuddy7 — Thu, 06 Feb 2025 06:57:29 GMT

is there any approch for incremental data? if S3 txt gets updated , will have to again run the dataframe to pull updated table and that's bit tedious job.