topic Exporting DataFrame with More Than 70K Rows from Workbook in Data Engineering

Exporting DataFrame with More Than 70K Rows from Workbook

Melodyyy — Thu, 23 May 2024 18:10:45 GMT

Hi everyone,

I have a Python workbook where I run code to generate a DataFrame with over 70,000 rows. When I try to visualize the DataFrame in the workbook, it only shows the first 10,000 rows due to the limit of the DataFrame visualizer.

I'm looking for a solution to export all the rows from the DataFrame. I'm open to either saving all 70K rows as a table in the Lakehouse or directly exporting them as a CSV file. Any suggestions or insights would be appreciated 🙂

Cheers,

Re: Exporting DataFrame with More Than 70K Rows from Workbook

Anonymous — Fri, 24 May 2024 10:04:53 GMT

Hi @Melodyyy ,

Thanks for using Fabric Community. Can you try creating a table in Lakehouse?

df.write.mode("overwrite").format("delta").saveAsTable("abc5")

You should be able to create a table and can study the data using Lakehouse SQL Endpoint or Spark SQL in Notebooks.

Hope this is helpful. Do let me know incase of further queries.

Re: Exporting DataFrame with More Than 70K Rows from Workbook

Anonymous — Mon, 27 May 2024 09:29:36 GMT

Hi @Melodyyy ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Exporting DataFrame with More Than 70K Rows from Workbook

Anonymous — Wed, 29 May 2024 05:47:42 GMT