Forum Discussion
T-SQL Notebook vs. PySpark Notebook - Pipeline Performance Comparison
Hello Everyone,
I am new to this forum, so please feel free to direct my question elsewhere, if it is more appropriate.
BACKGROUND: My team created a Pipeline triggering our T-SQL Notebook to run using DROP TABLE IF EXISTS and CREATE TABLE statements to overwrite/update our table in the designated Warehouse.
ISSUE: When we run the T-SQL Notebook manually it works perfectly and refreshs with the latest data on the first attempt. However, when we use the Pipeline, it doesn't work the first time. Sometimes when we run the Pipeline 3 times, we finally get refreshed data results. We have tried putting delay activities between the steps in the Pipeline. However, that doesn't work consistently either.
TESTING: We have performed a comparsion using T-SQL Notebook vs. PySpark Notebook. The PySpark Notebook works perfectly with the Pipeline and has no problems refreshing the data on the first attempt. However, as stated in the ISSUE section above, the T-SQL Notebook does not work on the first Pipeline refresh attempt.
QUESTION: Has anyone ran into this issue using T-SQL Notebooks? And if so, what solution(s) have you figured out to assure your Pipeline refreshes and works successfully on the first attempt?
THANK YOU so much for your time, effort, and guidance,
WDixon2025
Hi [Recipient's Name],
Thank you for the update. I’m glad to hear that upgrading to Spark Runtime 1.3 has enhanced the consistency of the Pipeline refreshes.
Given that the issue was resolved following the upgrade, it’s likely that the previous runtime version experienced execution delays or metadata commit inconsistencies affecting the T-SQL operations. In contrast, PySpark may have managed metadata updates more efficiently, which could explain why it performed without any issues.
To ensure continued stability, I recommend the following actions:
- Monitor Execution Logs: Leverage Fabric’s monitoring tools to verify that all runs are complete successfully and to detect any hidden errors or delays.
- Validate Metadata Commit Times: If feasible, introduce a WAITFOR DELAY '00:00:05' statement after table creation to assess whether metadata commit timing was a factor.
- Maintain Up-to-Date Runtime Versions: Since the upgrade had a positive impact, staying on the latest stable version is advisable to mitigate the risk of similar issues in the future.
If this resolves your issue, kindly consider accepting your response as the solution. Doing so will help other community members facing similar challenges.
Thank you.
