topic Re: Export entire delta table from lakehouse in Data Engineering

Export entire delta table from lakehouse

Anonymous — Tue, 03 Sep 2024 11:10:57 GMT

Is there any way to get a .csv onto my computer from a delta table in a lakehouse? I haven't found any viable approach so far despite trying.

Re: Export entire delta table from lakehouse

lbendlin — Tue, 03 Sep 2024 12:12:47 GMT

The Delta tables are already in Parquet format. it would be counterproductive to convert that to CSV.

"the entire table" - careful with that. Parquet supports versioning so you risk exporting all the prior versions too.

Re: Export entire delta table from lakehouse

FabianSchut — Tue, 03 Sep 2024 12:17:42 GMT

One way to achieve this is to write a small Python script in a notebook that reads the delta table from your lakehouse and exports it to the Files section of your lakehouse. It may be convinient to make a pandas dataframe after you've read the delta table with spark and use pandas 'to_csv' functionality.
With the OneLake file explorer: https://www.microsoft.com/en-us/download/details.aspx?id=105222, you could view and copy this csv to your local directory on your computer.

Re: Export entire delta table from lakehouse

Anonymous — Tue, 03 Sep 2024 12:20:44 GMT

There is only one version.

Re: Export entire delta table from lakehouse

Anonymous — Tue, 03 Sep 2024 12:36:40 GMT

Brilliant, it works. FYI, if I use

df.write.csv("Files/my_csv.csv")

to write the file, it keeps the delta table file structure, i.e. it makes a folder named "my_csv.csv". However, the file(s) are in csv, so for my purposes it just works.

Re: Export entire delta table from lakehouse

lbendlin — Tue, 03 Sep 2024 12:47:57 GMT

But seriously, converting a Delta file back to CSV is a travesty. Can your application not work with Parquet?

Re: Export entire delta table from lakehouse

Anonymous — Tue, 03 Sep 2024 12:58:15 GMT

My goals are beyond your understanding.

¯\_(ツ)_/¯

Re: Export entire delta table from lakehouse

frithjof_v — Tue, 03 Sep 2024 15:57:57 GMT

If you use Pandas to_csv(), instead of PySpark write.csv(), you will get a single CSV file.

https://learn.microsoft.com/en-us/fabric/data-science/read-write-pandas

Pandas cannot work with such large data volumes as Spark.

https://www.reddit.com/r/MicrosoftFabric/s/7EloDpd8wI

If your amount of data is not too large volume, then I guess I would use Pandas for writing to a single csv.