Forum Discussion
Spark Session configuration
Hi lsabetta ,
The issue happens because your Lakehouse table is stored in Delta format, not plain Parquet.
When you overwrite the table, new Parquet files are created, but old ones remain in the folder for versioning.
If you read the folder directly as Parquet, it loads both the old and new files - that’s why you see duplicate or outdated records.To fix this, make sure you read the table as a Delta table instead of as raw Parquet.
This ensures that only the latest valid version of the data is returned, without mixing older files.Thank you.
Hi lsabetta ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you pallavi_r for the prompt response.
The slow startup isn’t from your code or data size - it’s Spark cluster spin-up, which always takes a few minutes.
There’s no Spark config that makes session creation instant.
Below are few Options:
If tables are small, skip Spark and load them directly into Pandas/SQL (faster, no cluster).Keep the Spark session alive instead of restarting often.Ask your admin if a smaller/faster Spark pool is available.Use Spark only when you need distributed compute; otherwise stick with Python. you can’t make Spark spin up faster, but you can avoid Spark altogether or keep the session warm.
HI v-venuppu ,
Thanks for your answer.
My notebooks are written in Pandas because my tables are small and transformations are simple. I could use Python instead of Pyspark but the thing is that I need to read tables from my lakehouse.
Is there any way to read tables from a lakehouse without creating a sparksession?