Forum Discussion
Spark in Notebook taking more time to process the data.
Hi all, I'm working on a diagnostic log ingestion engine built with PySpark and Delta Lake on Microsoft Fabric. My setup parses incoming ZIP logs from a server, transforms signal data per file in...
Whatever we do, the park is taking more time to store data directly to the Lakehouse tables. Its faster when its storing as files in lakehouse. So we went ahead and stored as files and then moved the data to warehouse from files. However I will check more and mark as solution.