User Profile
Marco117
Frequent Visitor
Joined 2 years ago
User Widgets
Contributions
fabric data engineering extension for vscode dont show the spark notebooks of my workspace
I followed the step-by-step tutorial and verified that I can run code using PySpark with the local environments: fabric-synapse-runtime-1-1 → C:\ProgramData\anaconda\envs\fabric-synapse-runtime-1-1 fabric-synapse-runtime-1-2 → C:\ProgramData\anaconda\envs\fabric-synapse-runtime-1-2 However, I can't see the notebooks in my workspace. Do you know what could be causing this? Am I missing an additional component? PS: I already tried reinstalling VS Code from scratch, but I'm still experiencing the same issue. VS Code extension overview - Microsoft Fabric | Microsoft LearnMs Fabric spark notebook with poor performance
I have a notebook that was running on a Databricks cluster with only 4 executor cores and 4 driver cores, and Autoscale and Dynamically allocate executors were disabled. Here, the notebook executed in approximately 3 minutes. Now, in Fabric, the same notebook with the same inputs that I read from Databricks takes 14 minutes to execute (attached image, the notebook is invoked from a pipeline). What I see different in Fabric is that the workspace has Autoscale and Dynamically allocate executors enabled, and there are moments when the notebook starts using 72 cores when this is really not necessary. What could I do to improve this?azure fabric spark session not starting september 4th 2024
Does anyone know why since yesterday the spark notebooks are not working properly? I see on the reditt forum that users are reporting problems since yesterday, I'm using spark 1.2 But on the Azure status page no problems are reported, what's going on? Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook. LivySessionId: 186ce75d-a25f-4778-b38f-215a4738e299Notebook: Notebook_46078b9c-eb0e-4665-9b71-61d1741c280a.' :Re: Performance issues running pipelines and notebooks
I have the same problem, when I test from my spark notebook that transforms and stores a considerable amount of records in the lakehouse, the execution time is 5 min, with this time I am fine! When I do the same execution calling the notebook from a pipeline the execution time increases to 30 min, the execution time increases by 600% 😓 , why does this happen? does the execution of the notebook from a fabric pipeline has less resources?
