Fabric Meta Data Driven Pipelines
Join us for an engaging user group session focused on metadata-driven pipeline development in Microsoft Fabric. As organizations scale their data integration efforts, maintaining flexibility and governance becomes critical. This session will dive into best practices for building reusable, dynamic, and easily maintainable pipelines using metadata stored in centralized repositories. Whether you're a data engineer, BI developer, or architect, this session will offer practical insights into scaling your Fabric solutions with metadata-driven patterns. Come ready to learn, share, and connect with peers in the Fabric community.
Is it possible to write Copy activity logging to Fabric Lakehouse?
Greetings, community. Have a requirement where we need to log what files get transferred/fail from a Copy activity. I want to use the Logging feature, but this currently only supports ADLS connections. I remember reading somewhere that it is possible to access Lakehouse as ADLS, but I cannot get that working in Fabric. Anyone have ideas?
Semantic Model Refresh failing in Pipeline but not manually
Hello everyone, we have a Fabric F8 environment with a pipeline that triggers multiple pipelines to load data into bronze, silver and gold layer. After the Gold pipeline is succeeded, a Semantic Model Refresh activity should refresh a Semantic Model in another workspace. After it had worked for a few days, the pipeline now fails at the Semantic Model Refresh step. Restarting the pipeline from that step or triggering the Semantic Model refresh from a another pipeline or manually via the Refresh Button all succeed. The error message states, that a column is missing in the rowset. The column is created immediately before the semantic model refresh starts. What I have tried so far: - adding a 15 minutes waiting time to solve any metadata refresh activities - rebuild the whole pipeline from scratch - adding an invoke pipeline step, that starts a pipeline which only contains the Semantic Model Refresh activity None of that has solved the issue. Any suggestion would be highly welcome. Error message: Semantic model refresh execution failed, error message received from semantic model refresh operation - '{"errorCode":"ModelRefresh_ShortMessage_ProcessingError","errorDescription":"Retry attempts for failures while executing the refresh exceeded the retry limit set on the request.\n0xC11200B0: The '<oii>is_anonym<\/oii>' column does not exist in the rowset.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC11C0006: The current operation was cancelled because another operation in the transaction failed.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC14700F0: {\"RootActivityId\":\"46081B1B-9B40-4967-9402-297B0D45D710\"}"}'
Passing the Parameter while calling the Data Pipeline using REST API
Hello Readers, I am trying to trigger the REST API to trigger an on-demand job on my datapipeline: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/dataPipelines/{pipelineId}/jobs/execute/instances Body/Payload: { "parameters": [ { "name": "filed_based_run", "value": true, "type": "Boolean" }, { "name": "fail_whole_load", "value": true, "type": "Boolean" }, { "name": "config_file", "value": "on_demand", "type": "Text" } ] } Error: Status Code: 403 Forbidden Error Message: { "requestId": "cc5e44f7-cda1-4adc-9e18-ef19513b15d5", "errorCode": "FeatureNotAvailable", "message": "Parameter is not allowed for this item type or this item job type", "isRetriable": false } My question : Why we can't pass the parameter while calling the datapipeline and is there another I can do this like using Power Automate or Logic apps?
Invoke Pipeline "Rerun from Failed" re-executes entire child pipeline instead of skipping succeeded
Description: I have a parent-child pipeline setup in Fabric Data Factory, where the parent pipeline calls a child pipeline using the Invoke Pipeline activity. When one activity inside the child pipeline fails and the rest succeed, using "Rerun from failed" on the parent pipeline causes the entire child pipeline to re-execute from the beginning — including activities that already succeeded in the previous run. This is different from Azure Data Factory, where rerunning a parent pipeline from a failed activity correctly propagates the skip logic into nested Execute Pipeline calls — only the activities that actually failed (and anything downstream of them) get re-executed inside the child, while previously succeeded activities are marked Skipped. Steps to reproduce: Create a parent pipeline with an Invoke Pipeline activity calling a child pipeline. In the child pipeline, have multiple sequential/parallel activities where one fails and the rest succeed. Let the parent pipeline fail (due to the child failure). Use "Rerun from failed" on the parent pipeline run. Drill into the child pipeline's rerun — observe all activities (including previously succeeded ones) show fresh execution timestamps rather than being skipped. Impact: Non-idempotent activities (e.g., inserts without preceding deletes, counters, notifications, API calls) risk running twice, causing duplicate data or side effects. Increases pipeline runtime and cost unnecessarily on every rerun, since successful long-running steps are always redone. Breaks the expected "resume from point of failure" behavior that exists in ADF, making migration from ADF to Fabric riskier for complex orchestration patterns. Question for the community/Microsoft team: Is this a known/expected limitation of the current Invoke Pipeline implementation, or a bug? Is there a roadmap item to bring Fabric's rerun-from-failed behavior to parity with ADF for nested pipeline scenarios? In the meantime, is there a supported way to resume only the failed child pipeline run in a way that's reflected back in the parent's run status?Materialized Lake View Refresh activity Pipeline bug
Hi all, When using the preview feature refresh mlv activity in the pipeline, is do have some major bugs. After saving, it goes back to default, the pipeline does not want to refresh after, and I cannot parametrize it. Is there any workarround, and is this a known bug? Cause this is not workable.
