How can AI agents improve Microsoft Fabric Data Factory pipeline automation?
I am exploring how AI agents can help make data pipeline development and monitoring more efficient in Microsoft Fabric Data Factory. Some potential use cases: Automatically creating pipeline workflows based on business requirements Detecting pipeline failures and suggesting fixes Optimizing data movement and transformation steps Monitoring data quality and identifying anomalies Triggering automated actions through APIs or external tools I would like to understand how teams are currently combining AI agents with Fabric Data Factory. Are you using: AI-assisted pipeline generation? Automated error analysis and recovery workflows? Fabric notebooks with AI models for data operations? Custom agents connected with Data Factory pipelines? Would appreciate insights, real-world examples, or architecture patterns from developers working with Fabric Data Factory automation.
Passing a bearer token to copy data action using GET
Hi All, Not a question, but something I ran into, as I am just picking up MS Fabric, pipelines etc. I needed to use web request, to get an access token that is valid for 30 minutes. Next I needed to use that token as a bearer token to do a follow up API call to get actual data , there didn't seem to be that much info around, when I tried to google it. Eventually I found a you tube video, that told me what i needed to know. (linked below). Things like apikeys and other credentials will of course vary on the API yo are trying to use. This is my flow. Step 1 Do your web request Step 2 link the Web and copy data action Step 3 In the copy data action set it up as follows: As you can see the value of the Authorization header is dynamic, with in my case, the following expression: Bearer @{activity('Get access token').output.access_token} The part ".access_token" depends on the json you get returned, in my case it was a very flat structure with no above array. { "token_type": "BearerToken", "issued_at": "xxx", "salesforce_id": "xxx", "access_token": "xxx", "application_name": "xxx", "scope": "refresh_token full", "expires_in": "xxx" } Above you can see where i got the parameter access_token from. If this array had been nested in for instance another array called "data" then the value would have been data.access_token in stead of just access_token. I hope this is usefull info to share here. The video I got this from you can find here . Cheers Hans
Invoke Pipeline Returns "Could not found the requested item"
I'm having issues with the Invoke Pipeline(Preview) activity where I am getting the error {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} To test this issue, I set up a: A pipeline with the only activity in it being Invoke Pipeline(Preview) No parameters - The original pipeline I set up had parameters and I was getting the same error code so I tried to remove it to see if it was a parameter issue A child pipeline with the only activity in it to Get Metadata for a table in a lakehouse No parameters This pipeline will successfully run on its own When I run #1 I get the error message {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} however I'm not sure why the Item would not be found since I was able to specify it. I am attaching my configuration settings as screenshots to this post. Did I miss something in the Invoke Pipeline settings? Parent Pipeline Child Pipeline
Evolving the ADF to Fabric migration experience
Hi team, When working with Azure Data Factory and moving into Microsoft Fabric, the new “migration” experience is a really interesting step forward. It helps bring both worlds closer and makes the transition smoother. After trying it out, I’ve seen that it opens up a lot of possibilities, and it’s great to see that we can also propose ideas to continue improving it. In this context, I’ve shared an idea that I believe could bring even more value. Right now, the experience allows you to connect and integrate existing ADF assets into Fabric, which is very helpful to start working within the ecosystem. As a next step, it could be interesting to move towards a deeper integration where: Assets are fully integrated into Fabric They are executed from Fabric And their consumption can be associated with Fabric capacity Additionally, it could be valuable to give customers the flexibility to choose where the cost is allocated (Fabric or Azure), enabling a more gradual adoption. This is a small evolution on top of what already exists, but it could make a big difference in how the experience aligns with different adoption scenarios. If this idea resonates with you, you can vote for it here: https://lnkd.in/ejT9QSRj And if you find it useful, I would really appreciate your support with a like on the comment and voting for the idea. This helps increase visibility and gives it a better chance to evolve.
Is the pricing of Fabric pipeline data copy activities incorrect?
Hi all, here are some tests and questions regarding the pricing of data copy activities within Fabric pipelines. tldr; I'd like to know your opinion about the pricing of data copy activities. In my opinion, the consumption of pipeline data copy tasks is not billed correctly. Almost all operations are measured with the same number of CUs, independent of the duration and optimization. This leads into the conclusion that optimizing the data copy activities does not have a real impact to billing but reducing the number of elements absolutely has. What did I test? I did some investigations on a Fabric pipeline with a copy data task loading 12 tables from a test database to parquet. The copy data task is executed within a ForEach loop. The target was to investigate how we can optimize the CU usage of copy data tasks. What does Microsoft say? According to the pricing page of fabric pipelines, the following statement is written for “Data Movement” tasks (copy data activity): Data Movement service for Copy activity runs. You are charged based on the Capacity Units consumed during the Copy activity execution duration. In the price breakdown on how the “Data movement” is charged, Microsoft states that the metrics are… Based on Copy activity run duration in hours and the used intelligent optimization throughput resources Source: Pricing for data pipelines - Microsoft Fabric | Microsoft Learn, 18.12.2024 But, what is “intelligent optimization”? According to Microsoft’s “Copy activity performance and scalability guide” there are several things to consider like the parallel copy for partitioned sources or the intelligent throughput optimization. Source: Copy activity performance and scalability guide - Microsoft Fabric | Microsoft Learn So I made three tests with different settings, changing the intelligent throughput optimization setting (use Max vs Auto) and comparing it against setting the batch count in the ForEach loop to 6. The batch count has a lot of impact to the duration, but the “ITO” setting does not: Let’s have a look at the consumed CUs in the Fabric Metrics App: All pipelines are charged the same. If we go deeper into the detail, we can check how many CUs are used by the single activities. According to Microsoft’s pricing calculation, the duration of the operations is important for the calculation of the costs: https://learn.microsoft.com/en-us/fabric/data-factory/pricing-pipelines That means that the duration should have direct impact to the CU calculation and costs. If we have a look at the single operations, they all have 360 CUs, independent of the runtime: Not what I expected. Taking the next statement out of a blog post here, this is also what I assume for the calculation: In my eyes: 1.5 CU per hour gives 0,0004166 CU per second. Say 30 s duration. 30 * 0,0004166 = 0,0125. Now how many intelligent optimization throughput resources are used? Was set to auto, so unclear. But even assuming a maximum of 256, we only get 256 * 0,0125 = 3,2 CU (s). Far from listed 360! Source: Solved: Minimum CU (s) billing per copy? Or am I just bad ... - Microsoft Fabric Community Let’s have a look at a real-life scenario up and running at a customer. If we check the correlation between duration of the operation and CUs, we actually see that almost all data movement operations have 360 CUs! Actually, 99% of the operations at the customer result in 360 CUs. If I look at the duration at least the operations with higher CUs are also “long-running” ones: Here we see another thing: It seems that the CUs are calculated in 360-steps (maybe this is linked to a time calculation in seconds somehow ((60*60)/10)? Questions Is this calulation of consumption expected and intended (by Microsoft)? Am I doing the calculation right, or do I oversee something? What could a possible best-practice be if you have lot of tables to copy with a rather small amount of data? Other sources Confused by CU-seconds calculation - Microsoft Fabric Community
