Forum Discussion
Helper ILoad txt pole (|) separated file to delta table
Hello akkibuddy7
You cannot use load to table option with | delimited files. That option is for csv
you have to use spark notebook to load to tables
df = spark.read.format("csv") \
.option("header", "true") \ # Set to 'false' if no header row exists
.option("delimiter", "|") \
.load("/path/to/your/file.txt")df.write.format("delta").mode("overwrite").saveAsTable("table_name")
If this is helpful please accept the answer and give kudos
Super UserHello akkibuddy7
You cannot use load to table option with | delimited files. That option is for csv
you have to use spark notebook to load to tables
df = spark.read.format("csv") \
.option("header", "true") \ # Set to 'false' if no header row exists
.option("delimiter", "|") \
.load("/path/to/your/file.txt")
df.write.format("delta").mode("overwrite").saveAsTable("table_name")
If this is helpful please accept the answer and give kudos
Helper Iis there any approch for incremental data? if S3 txt gets updated , will have to again run the dataframe to pull updated table and that's bit tedious job.