data factory | pipelines
557 TopicsIn Data Pipelines, Schedules should not be part of the definition
The request is to ensure Schedules are not included in the definition of a Data Pipeline for purposes of Deployment Pipelines. Today, when we deploy content across environments, the item frequently appears as “different” because the schedule configuration is included in the comparison. This creates constant noise and makes it harder to identify meaningful differences. More importantly, a schedule configured in DEV or TEST can be promoted into PROD as part of a deployment. This creates a real operational risk and is unacceptable — PROD scheduling must be managed independently and explicitly, not carried over from lower environments.818Views26likes3CommentsPipeline activity Refresh SQL Endpoint (preview) does not accept dynamic content
I am testing out the new Pipeline activity for Refresh SQL Endpoint, currently in preview. Sure, I get it that this is in preview, but it should at least have basic minimum functionality. It does not. The Workspace ID property does not even allow Dynamic Content like a Copy Data activity Destination tab. The Lakehouse Endpoint ID will accept Dynamic Content, but after saving and re-opening the pipeline, that Dynamic Content is empty. Please fix these basic issues.597Views8likes6CommentsAllow customization of email notifications for failed pipeline activities
Currently, Microsoft Fabric can automatically send an email notification when an activity or item in a pipeline fails. However, the content and structure of this email follow a fixed template. It would be useful to have the option to customize this notification email, for example: Customize the subject line Add or remove information from the email body Include pipeline parameters or other contextual information Defina a custom message for the recipients This would make failure notification much more actionable. Insteaad of only notifying users that something failed, the email could immediately provide the relevant context and instructions needed to investigate or resolve the issue.10Views3likes0CommentsCron based schedules
So here is another MS Fabric frustration that I'd like to share with the community: In schedules for notebooks or pipelines I really miss the possibility to actually use a CRON schedule. This is something I miss and not necessarily the origin of my frustration. My frustration is that today I was experimenting with the Fabric CLI and when through a moment of relief: I noticed that it is possible to set a CRON type schedule (https://microsoft.github.io/fabric-cli/examples/job_examples/#create-cron-based-schedule) However, this frustration was short lived since the functionality of this Microsoft CRON type schedule is not the same as the CRON type schedules I am used to (https://crontab.guru/) I would like to create a schedule that runs on weekdays between 6AM and 7PM, every 15 minutes. This would result in a crontab like this: */15 6-19 * * 1-5. I still did not find a way to create this type of schedule in MS Fabric due to the constraints on scheduling. If I am wrong, please enlighten me. If I am not wrong, here is a kind request for the MS employees to just implement this type of crontab syntax. Kind regards, A tensed MS Fabric Developer.13Views1like0CommentsAllow dynamic content in Activity Name (and other fields on general tab)
We copy a large number of tables by iterating through a ForEach activity and calling the copy activity for each. When trying to answer "which tables take a long time to copy", it's hard because all of the copy activities just have the generic activity name. If I were able to include dynamic content in the Activity name field, I could use code like @concat('Copy activity - ', Item().TableName) and then in the output we could tell which table was being copied without having to dive into the .json input and output. This would also be helpful for other fields on the general tab - like Timeout settings and the like. Then timeout durations could be calculated offline, and fed into the activity vs. hardcoding. That way if the job starts running slower over time, the timeout setting could be gradually increased. p.s. Unable to use the new "Copy Job" activity because it doesn't support dynamic partitioning, needed for good performance. Thanks, Scott3.6KViews19likes3CommentsFabric Workspace Operator Role for Pipeline Monitoring and Rerun Operations
Many organizations have dedicated production support and operations teams responsible for monitoring Microsoft Fabric pipelines and responding to failures. Currently, the available workspace roles are Admin, Member, Contributor, and Viewer. While Viewers can monitor content, they cannot rerun failed pipelines. Contributors can rerun pipelines, but they also have permissions to modify pipeline definitions and other workspace artifacts. This creates a governance gap for enterprises that require segregation of duties between development and operations teams. Proposed Solution Introduce a new built-in Operator workspace role with permissions such as: Allowed View pipeline definitions Monitor pipeline runs and run history View execution logs and error details Manually trigger pipeline execution Rerun failed pipeline runs Cancel running pipeline executions Access workspace monitoring capabilities Not Allowed Create, edit, or delete pipelines Create, edit, or delete Fabric artifacts Modify notebooks, lakehouses, warehouses, or dataflows Manage workspace access or permissions Change workspace settings Connect/disconnect Git repositories Business Value This role would: Support enterprise segregation of duties requirements. Enable production support teams to resolve operational issues without granting development permissions. Reduce the risk of accidental changes in production workspaces. Improve governance and compliance. Align Microsoft Fabric with operational models commonly used in enterprise data platforms. Many organizations need a role that sits between Viewer and Contributor, providing operational control without development privileges. An Operator role would significantly improve security, governance, and supportability for Fabric Data Factory workloads. Use Case: Production support team monitors Fabric pipelines, investigates failures, and reruns failed executions, but should not be able to modify pipeline definitions or other workspace assets.50Views3likes0CommentsEnvironment Copy/Duplicate
Why is there still no single tool to copy/duplicate an environment. Its seems that an MCP nor Deployment Pipelines can do all the necessary tasks to copy the whole of an env in its entirety, ie all artefacts and warehouses/lakehouses etc such for either 1) creation of a new environment like Pre Prod, Staging or to quickly replicate an environment. I know that using a combo of tools it can be done. But surely Fabric has matured enough for this to be available by now or have i missed something ?15Views0likes0Commentswrong filter for CCI
Hi team that sets up the report, I extract report for EMEA and CEE. Some weeks now I see many files marked as review for CCI but 90 percent of them are wrongly tagged. Actually, wrong CCI is only below 80 and over 90 for our market. Kindly do set correct filter criteria. Thank you for your help. Zuzana Balazova8Views0likes0CommentsPipeline CI/CD Improvement, Get Item ID activity from workspace ID and item name
Currently when following any branch strategy that requires a separate ephemeral feature (or other) workspace running a notebook from the same workspace requires too much ground work To find the ID of the notebook you need a way to call the APIs to GetItems in the workspace to find the correct ID of the notebook (or other item like the lakehouse), this is overly convoluted requiring API calls through web activity or separate notebook in a stand alone workspace with fixed ID that returns the item IDs in the pipeline's workspace Idea: add a pipeline activity that retrieves the item ID of an item using a workspace ID and item name or a function in pipeline expressions that allows the same to be done so this can be inlined without a dedicated pipeline activity.22Views0likes0Comments