How can AI agents improve Microsoft Fabric Data Factory pipeline automation?
I am exploring how AI agents can help make data pipeline development and monitoring more efficient in Microsoft Fabric Data Factory. Some potential use cases: Automatically creating pipeline workflows based on business requirements Detecting pipeline failures and suggesting fixes Optimizing data movement and transformation steps Monitoring data quality and identifying anomalies Triggering automated actions through APIs or external tools I would like to understand how teams are currently combining AI agents with Fabric Data Factory. Are you using: AI-assisted pipeline generation? Automated error analysis and recovery workflows? Fabric notebooks with AI models for data operations? Custom agents connected with Data Factory pipelines? Would appreciate insights, real-world examples, or architecture patterns from developers working with Fabric Data Factory automation.
Passing a bearer token to copy data action using GET
Hi All, Not a question, but something I ran into, as I am just picking up MS Fabric, pipelines etc. I needed to use web request, to get an access token that is valid for 30 minutes. Next I needed to use that token as a bearer token to do a follow up API call to get actual data , there didn't seem to be that much info around, when I tried to google it. Eventually I found a you tube video, that told me what i needed to know. (linked below). Things like apikeys and other credentials will of course vary on the API yo are trying to use. This is my flow. Step 1 Do your web request Step 2 link the Web and copy data action Step 3 In the copy data action set it up as follows: As you can see the value of the Authorization header is dynamic, with in my case, the following expression: Bearer @{activity('Get access token').output.access_token} The part ".access_token" depends on the json you get returned, in my case it was a very flat structure with no above array. { "token_type": "BearerToken", "issued_at": "xxx", "salesforce_id": "xxx", "access_token": "xxx", "application_name": "xxx", "scope": "refresh_token full", "expires_in": "xxx" } Above you can see where i got the parameter access_token from. If this array had been nested in for instance another array called "data" then the value would have been data.access_token in stead of just access_token. I hope this is usefull info to share here. The video I got this from you can find here . Cheers Hans
Invoke Pipeline Returns "Could not found the requested item"
I'm having issues with the Invoke Pipeline(Preview) activity where I am getting the error {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} To test this issue, I set up a: A pipeline with the only activity in it being Invoke Pipeline(Preview) No parameters - The original pipeline I set up had parameters and I was getting the same error code so I tried to remove it to see if it was a parameter issue A child pipeline with the only activity in it to Get Metadata for a table in a lakehouse No parameters This pipeline will successfully run on its own When I run #1 I get the error message {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} however I'm not sure why the Item would not be found since I was able to specify it. I am attaching my configuration settings as screenshots to this post. Did I miss something in the Invoke Pipeline settings? Parent Pipeline Child Pipeline
Evolving the ADF to Fabric migration experience
Hi team, When working with Azure Data Factory and moving into Microsoft Fabric, the new “migration” experience is a really interesting step forward. It helps bring both worlds closer and makes the transition smoother. After trying it out, I’ve seen that it opens up a lot of possibilities, and it’s great to see that we can also propose ideas to continue improving it. In this context, I’ve shared an idea that I believe could bring even more value. Right now, the experience allows you to connect and integrate existing ADF assets into Fabric, which is very helpful to start working within the ecosystem. As a next step, it could be interesting to move towards a deeper integration where: Assets are fully integrated into Fabric They are executed from Fabric And their consumption can be associated with Fabric capacity Additionally, it could be valuable to give customers the flexibility to choose where the cost is allocated (Fabric or Azure), enabling a more gradual adoption. This is a small evolution on top of what already exists, but it could make a big difference in how the experience aligns with different adoption scenarios. If this idea resonates with you, you can vote for it here: https://lnkd.in/ejT9QSRj And if you find it useful, I would really appreciate your support with a like on the comment and voting for the idea. This helps increase visibility and gives it a better chance to evolve.Dataflow Gen2: Variable Library references fail in Connection Settings
Hi everyone, I am working on variabilizing my Dataflow Gen2 for deployment pipelines, following the documentation on Variable references. I’ve noticed a discrepancy in how variables are handled, and I suspect it might be a bug or a UI limitation. The Issue: When I define parameters using a Blank Query and try to use them to replace values in the Source Connection settings (via the user interface), the connection fails or isn't recognized. And replace in the source: let Source = Lakehouse.Contents([]), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" That's not working. The Workaround (Manual M Code): However, if I bypass the UI and manually insert the Variable.ValueOrDefault syntax directly into the Source Step using the Advanced Editor, it works perfectly. let Source = Lakehouse.Contents([]), // Retrieving variables manually works fine workspace_id = Variable.ValueOrDefault("$(/**/My Library/Workspace ID)", "default_value"), lakehouse_id = Variable.ValueOrDefault("$(/**/My Library/Lakehouse ID)", "default_value"), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" My Question: Has anyone else encountered this issue where the UI connection dialog doesn't seem to parse the variable reference, forcing us to hardcode M logic manually? Is there a specific syntax for the UI that differs from the Advanced Editor? Thanks for your help! References: - Dataflow Gen2 Variable Library Integration
