topic Re: Can only see first 1000 records while running query in fabric notebook in Data Engineering

Can only see first 1000 records while running query in fabric notebook

priyanksingh — Thu, 19 Oct 2023 05:02:41 GMT

Hi All,

While running any query on a table or running a pyspark code , I am able to see first 1000 records and also not able to export more records in csv.

Is there any way to export more than 1000 records in csv from fabric notebook ?

Re: Can only see first 1000 records while running query in fabric notebook

v-cboorla-msft — Thu, 19 Oct 2023 18:11:41 GMT

Hi @priyanksingh

Welcome to Microsoft Fabric Community and thanks for posting your question here.

As I understand that you want to export more than 1000 records in csv from Fabric Notebook.

When you run a PySpark query in Fabric Notebook, the results are limited to the first 1000 records by default. However, you can export more than 1000 records in csv fromat. Please follow the below mentioned workaround in order to achieve your resolution.

I have published 5000 records into my Lakehouse. Run the below mentioned code to save the file into your Lakehouse.

df = spark.sql("SELECT * FROM ShekarLH.5000_Records") df.write.csv("abfss://42965892-5819-4760-ad68-8983467a9df3@msit-onelake.dfs.fabric.microsoft.com/5023fdc9-fd65-4ef6-bb9f-bfdb82bd646f/Files/Destination/SH5000")

After running the above code, the csv file gets created in the respective path.

To access this file in your local desktop, please download Onelake Explorer using this link Download Onelake. Make sure to sync with your respective workspace.

Now you will able to download the csv file which is having all the 5000 records.

Hope this helps, let us know in case of any further queries.

Re: Can only see first 1000 records while running query in fabric notebook

v-cboorla-msft — Mon, 23 Oct 2023 05:26:20 GMT

Hi @priyanksingh

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Re: Can only see first 1000 records while running query in fabric notebook

priyanksingh — Tue, 24 Oct 2023 12:57:59 GMT

Hi,

Thank you for the suggestion. Yes i had tried this method but i was looking for some setting where we can change the default value. This setting is present in Databricks

Re: Can only see first 1000 records while running query in fabric notebook

HimanshuS-msft — Tue, 24 Oct 2023 17:36:45 GMT

Hello @priyanksingh
I am not sure if it is in place in Databricks , as per this thread i think they are still working on that , apoloziges if you are able to see that and may be they did not updated the thread .
I do not think we can have that set in Fabric at this time , but then you can use the .head() and that should do the trick .

df.head(10)

Thanks
Himanshu

Re: Can only see first 1000 records while running query in fabric notebook

v-cboorla-msft — Fri, 27 Oct 2023 07:08:19 GMT

Hi @priyanksingh

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .

Thanks