data pipeline
515 Topics
SessionStateError
Hi all, During the pipeline run on Sunday, the notebook failed with the following error: SessionStateError: Livy session has failed. Session state: Dead. Session yields "Uncertain" result. The same pipeline completed successfully on Monday without any manual changes or rerun. What typically causes a Livy session to enter the Dead state? Since Monday's run completed successfully, can we confirm that the failed Sunday load was automatically recovered? How to overcome this issue? please guide me i want a solution28Views0likes2Comments
Safest approach for handling schema changes
What is the safest approach for handling schema changes in source tables when downstream Fabric pipelines, Lakehouse tables, and Power BI semantic models depend on them? For example, if a source system adds a new column, renames an existing column, changes a data type, or removes a column, what is the recommended way to manage these changes without breaking downstream pipelines and reports? Would you recommend using schema validation, a staging layer, versioned schemas, or some other approach in Microsoft Fabric?Solved37Views1like4Comments
Fabric administrator’s perspective
Hi, I am the only person responsible for managing Microsoft Fabric in my company, including administration, data engineering, workspace management, security, governance, and reporting. What initial setup steps and preventive measures should I complete from a Fabric administrator’s perspective? For example, I would like to enable item recovery and configure other important settings that protect the environment from accidental deletion, data loss, security issues, and operational failures. Could you provide a practical checklist covering tenant settings, workspace governance, access control, backup and recovery, monitoring, auditing, capacity management, deployment practices, and business continuity?Solved45Views1like4Comments
how to set failure notification on the pipeline level
I have created a pipeline running multiple steps, if one step failed, i would like to move to next step till it finished the whole pipeline. I don't want to set a failure notification per step because that could end up sending out too many failure emails. I tried to create an activator for the pipeline job, the activator captured the failure and was able to send out a testing email, but it failed to send out the alert email in real environment. I wonder if there is another way to set up alert for the whole pipeline? Thnx.Solved152Views1like8Comments
Scaling up to down fabric capacity giving utilisation error.
Hello, I have upscale my fabric capacity to F32 from F8 because I want to process 400 gb of data so now I have completed my work on few minutes and then I have down scale fabric capacity from F32 to F8 but after scaling down it is giving error like your organization has used entire capacity no capacity available. How to resolve this i don't want to increase capacity to resolve this issue.Solved73Views2likes7CommentsData pipeline is remaning into InProgress state
Hello, I have created data pipeline and in that i have only 1 notebook, but it is taking around 15-25 min average to run my pipeline and i am using F4 capacity. and i feel F4 is more then enough for it still why i am facing this issue Need help.Solved166Views0likes4CommentsFabric Workspace Architecture
I'm a Data Analytics Engineer at a logistics startup, and I'm the only person handling the data space. I'm looking for insights on how to effectively manage and monitor ETL processes using metadata-driven pipelines for a medallion architecture. I have 10 applications with about 2k tables total, though NetSuite accounts for 40% of the data. I'm considering 3 approaches for the Fabric workspace architecture: Approach 1: Create a dedicated NetSuite workspace with bronze, silver, and gold pipelines, plus 3 lakehouses for the medallion layers and schemas for individual departments. Then create similar workspaces for other applications. Approach 2: Organize by layer instead—separate workspaces for raw-to-bronze, bronze-to-silver, and silver-to-gold, with each workspace handling all applications at that layer. Approach 3: Use a single workspace to manage all medallion layers for all applications, but I'm not sure how to easily manage and monitor all the pipelines in this setup. Can you give me the pros and cons for each approach and recommend which one is best for a solo data engineer?Solved213Views2likes9CommentsDeployed notebooks cannot start a session, but a copy of the same notebook works fine
Hi all, I'm running into a strange issue with Microsoft Fabric Deployment Pipelines and was wondering if anyone else has experienced something similar. Scenario Source workspace: DEV Target workspace: PROD Deployment method: Fabric Deployment Pipeline The deployment completes successfully and the notebook appears in the PROD workspace. Problem When I open the deployed notebook in PROD and try to start a session, I get: "Your notebook session cannot be started. Please try running the notebook again. If the problem persists, check your network settings or contact support." The notebook never reaches the first code cell. No code is executed at all. What I've tested Creating a brand new notebook in PROD works. Creating a copy of the deployed notebook in the same PROD workspace works immediately. The copied notebook can start a Spark session and execute code normally. The original notebook deployed by the pipeline cannot start a session. The notebook copy has the same owner as the deployed notebook.Solved167Views0likes5CommentsUnable to Delete Data Pipelines and Notebooks
I have some items in my Fabric workspace that I cannot delete. When I try to delete them, Fabric tells me that they have dependent items, but this is not true. Two of my pipelines are empty (no activities) and my notebook has had the lakehouse reference removed. I have no idea why I cannot delete them. Any suggestions?Solved411Views0likes8Comments
Automating Fabric feature workspace setup with OneLake shortcuts
A few weeks ago I ran into an annoying problem with Microsoft Fabric. Every time I created a new feature workspace and branched it from my main Git branch, all the metadata came across perfectly (Lakehouse, Warehouse, schemas, tables, notebooks, pipelines, etc.), but... no data. That meant every developer had to manually: Create a shortcut to the PROD Warehouse Copy some production data Delete the shortcut Finally start developing So I automated it. Now a single notebook: Detects the current workspace automatically Creates a temporary shortcut from DEV Lakehouse to PROD Warehouse storage Discovers all schemas and tables (no hardcoding) Reads a configurable number of records from each PROD table Copies a configurable number of rows into the DEV Warehouse using Spark + synapsesql() Deletes the shortcut when it's done The important detail is that the DEV workspace already contains the complete metadata structure because it was created from the PROD Git branch. The notebook only seeds the data. The workflow is now: Create new workspace | v Create Git branch from main | v Run bootstrap notebook | v DEV workspace ready for development How does it work? The notebook uses the OneLake DFS endpoint called by the Fabric UI when creating and deleting shortcuts. The API calls used are: Create shortcut: POST https://onelake.dfs.fabric.microsoft.com/v2.0/workspaces/{workspaceId}/artifacts/{lakehouseId}/shortcuts/batchCreate Delete shortcut: DELETE https://onelake.dfs.fabric.microsoft.com/v2.0/workspaces/{workspaceId}/artifacts/{lakehouseId}/shortcuts/Tables/{shortcutName} The authentication is handled using the Fabric notebook identity: notebookutils.credentials.getToken( "https://storage.azure.com/" ) The shortcut is only temporary. It is created to expose the PROD Warehouse storage layer, Spark reads the underlying Delta files, the data is written into the DEV Warehouse, and then the shortcut is removed. And yes, I considered keeping permanent shortcuts, but that does not fit my workflow. I need the freedom to play around with data without depending on PROD. The shortcut is only a temporary bridge to seed DEV data and is removed afterward. So, the final result: PROD Warehouse | | OneLake DFS shortcut endpoint v Temporary shortcut in DEV Lakehouse | | Spark reads underlying Delta files v Existing DEV Warehouse tables | v Shortcut removed The notebook does not create new tables. It relies on the metadata already synchronized through Git and only inserts sample data into the matching DEV tables. A few things to consider: The amount of copied data is configurable. The user running the notebook needs access to both environments. Data governance rules should still be applied because this process moves PROD data into DEV. The biggest benefit for me is that a new feature workspace can now go from "empty metadata" to "ready for development" with a single notebook execution. I'm curious how other Fabric teams handle this today. Do you: copy data using pipelines? create the shortcut manually and keep them permanently? use another workspace provisioning approach?Solved339Views5likes7Comments
