Forum Discussion

NagaRK's avatar
NagaRK
Icon for Advocate I rankAdvocate I
1 year ago

Spark in Notebook taking more time to process the data.

Hi all,   I'm working on a diagnostic log ingestion engine built with PySpark and Delta Lake on Microsoft Fabric. My setup parses incoming ZIP logs from a server, transforms signal data per file in...
data engineering
notebook
Srisakthi's avatar
Srisakthi
Icon for Super User rankSuper User
1 year ago

Hi NagaRK ,

 

The performance is based on your spark setting,the capacity that you are using.

Make sure these settings are available in your spark environment and which is attached to your notebook

1. Native Execution Engine - uses vectorized engine and helps improves query performance

2. Apache Spark latest runtime

3. Leverage Autotune properties - helps in speed up workload execution and performance

4. Leverage Spark Resource Profiles - Microsoft fabric provides flexibility to utilise these resource profiles. By default all the workspaces is atatched to writeHeavy profile. 

5. Make sure to enable v-order

 

You can refer this article on threadpool

https://learn-it-all.medium.com/parallelism-in-spark-notebook-execution-in-microsoft-fabric-8fb6ac3f79be

 

Regards,

Srisakthi