Forum Discussion
T-SQL Notebook vs. PySpark Notebook - Pipeline Performance Comparison
Hi [Recipient's Name],
Thank you for the update. I’m glad to hear that upgrading to Spark Runtime 1.3 has enhanced the consistency of the Pipeline refreshes.
Given that the issue was resolved following the upgrade, it’s likely that the previous runtime version experienced execution delays or metadata commit inconsistencies affecting the T-SQL operations. In contrast, PySpark may have managed metadata updates more efficiently, which could explain why it performed without any issues.
To ensure continued stability, I recommend the following actions:
- Monitor Execution Logs: Leverage Fabric’s monitoring tools to verify that all runs are complete successfully and to detect any hidden errors or delays.
- Validate Metadata Commit Times: If feasible, introduce a WAITFOR DELAY '00:00:05' statement after table creation to assess whether metadata commit timing was a factor.
- Maintain Up-to-Date Runtime Versions: Since the upgrade had a positive impact, staying on the latest stable version is advisable to mitigate the risk of similar issues in the future.
If this resolves your issue, kindly consider accepting your response as the solution. Doing so will help other community members facing similar challenges.
Thank you.
Community SupportHi WDixon2025,
Welcome to the Microsoft Fabric forum and thank you for your thorough explanation of the issue! It's great to see that you've conducted some testing and found that PySpark performs well within the same Pipeline. This is valuable information.
- When using DROP TABLE IF EXISTS and CREATE TABLE, make sure no other processes are accessing the table at the same time.
- Although you have tried using delay activities, consider configuring a Retry Policy in your Pipeline. Go to your Pipeline activity → Settings → Retry → Adjust the number of retry attempts and interval. This can help in case of transient issues.
- Ensure your Warehouse has adequate resources during Pipeline runs. Check for any throttling or capacity issues using the monitoring tools available in Fabric.
- After executing T-SQL Notebooks, sometimes data may not reflect immediately due to caching.
If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.
Thank you.