topic Re: save data to .csv in Dataflow

save data to .csv

Anonymous — Fri, 31 May 2024 05:49:48 GMT

hello,

in fabric dataflow, after some transformations, I have got the data.

Now I would like to save this data as csv.

How do I do this in notebook?

thanks

Re: save data to .csv

Anonymous — Fri, 31 May 2024 06:11:14 GMT

Hi again Anonymous
Thanks for using Fabric Community.
Unfortunately, Fabric Dataflow (Gen2) itself doesn't have a built-in functionality to directly save the transformed data as an CSV file. However, you can achieve this by integrating Dataflow with Notebooks:

1) Give the destination as lakehouse in Dataflow Gen2 and publish the Dataflow . This will create a new table in your lakehouse.

2) Create a new notebook and run the below code:

df = spark.sql("SELECT * FROM lakehouse_1.customers_1000 ")

df.write.option("header",True).csv("Files/customersnew.csv")

3) A file will be created in the lakehouse files section:

Hope this helps. Please let me know if you have any further questions.

Re: save data to .csv

NandanHegde — Fri, 31 May 2024 13:48:25 GMT

Adding on top of what Anonymous stated, you can even use data pipelines to copy the data from lakehouse and save this data as csv in the sink supported by data pipeline if need be.

Any specific reason why you want to integrate notebook?

Re: save data to .csv

Anonymous — Fri, 31 May 2024 13:54:07 GMT

please do let me know how to save the data as .csv in the pipeline

thank you

Re: save data to .csv

NandanHegde — Fri, 31 May 2024 14:15:58 GMT

you can use Copy activity for the same post dataflow trigger

where destination can be any sink supported by data pipelines.
In my example I have taken lakehouse only

Re: save data to .csv

Marcel_A — Mon, 12 Aug 2024 16:18:28 GMT

By any chance do you if the Notebook code can be updated to dynamically append the field "File_Instance" to the output name?

Re: save data to .csv

frithjof_v — Wed, 14 Aug 2024 07:02:09 GMT

If I understand correctly, this approach will create a folder with multiple files.

To create a single file from a Notebook, please refer to these threads:

https://community.fabric.microsoft.com/t5/Data-Engineering/How-do-I-just-write-a-CSV-file-to-a-lakehouse-with-a-notebook/m-p/4064426/highlight/true#M3311

https://community.fabric.microsoft.com/t5/Data-Engineering/Progrmatically-write-files-in-delta/m-p/4093622#M3528

I also think NandanHegde's solution is a good solution which doesn't involve Notebook.