Forum Discussion

Shreya_Barhate's avatar
Shreya_Barhate
Resolver II
4 months ago
Solved

Errors like 403,404 while running FUAM pipelines

Hi,

I’m running a Fabric FUAM orchestration pipeline Load_FUAM_Data_E2E that triggers multiple FUAM child pipelines (capacities/workspaces/items/activities/capacity metrics/refreshables). When I run the E2E pipeline, multiple child pipelines fail with 403/404 errors on admin API calls. However, when I run the same child pipelines individually, they complete successfully except Get Items.
Here are the two main errors I’m seeing:

1. Get Items – 404 NotFound(This error is very consistent):

Operation on target Get Items failed: Failure happened on 'Source' side.
ErrorCode=RestCallFailedWithClientError
Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException
Message=Rest call failed with client error, status code 404 NotFound, please check your activity settings.
Request URL: https://api.fabric.microsoft.com/v1/admin/items?<suppressed 1 parameter(s)>.
Response: {"requestId":"2828ba19-7693-41b9-84b5-dc01735eb543","errorCode":"EntityNotFound","message":"The requested resource could not be found","relatedResource":{"resourceId":"b5aed419-c02e-4bfc-9cb0-9928349153c3","resourceType":"Item"}}
Source=Microsoft.DataTransfer.ClientLibrary


2. Get Capacities – 403 Forbidden:

Operation on target Get Capacities failed: Failure happened on 'Source' side.
ErrorCode=RestCallFailedWithClientError
Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException
Message=Rest call failed with client error, status code 403 Forbidden, please check your activity settings.
Source=Microsoft.DataTransfer.ClientLibrary

Run behavior:

Run 1: 6 child pipelines failed

Run 2: only 2 child pipelines failed


Individually: all child pipelines run successfully except the Get Items pipeline.


My questions:

1. What could cause these admin API calls to fail only when triggered via the E2E parent pipeline, but succeed when running the child pipelines individually?
2. Since failures vary by run (6 failed → 2 failed), could this be related to parallel execution/throttling/transient errors or race conditions (upstream outputs not ready causing downstream 404 “EntityNotFound”)?
3. What are the recommended best practices to stabilize FUAM ingestion pipelines—sequential orchestration, retry/backoff, or specific tenant/admin settings for Fabric/Power BI admin APIs?




Thanks,
Shreya

data engineering
  • deborshi_nag's avatar
    deborshi_nag
    4 months ago

    Hello Shreya_Barhate 

     

    Since the child pipelines run successfully when you run them interactively, it suggests to me that it is related to API authorization on the identity runnig it. When you run a pipeline directly, the REST APIs execute under your identity context, and you may have all the required authorization - hence they succeed! 

     

    When the E2E pipeline orchestrates it, it is using the connection identity, which is probably lacking the Admin API access in tenant settings. 

     

    There could also be a case of concurrency, since you mention that not all errors are consistent. This is probably because the Admin APIs are throttled, and you're hitting those limits. 

     

    I suggest you use the following:

     

     - a service principal enabled with admin API setting for running the pipelines 

     - try running child pipelines sequentially or with a low batchcount, with retries

     

5 Replies

  • deborshi_nag's avatar
    deborshi_nag
    Super User
    4 months ago

    Hello Shreya_Barhate 

     

    Since the child pipelines run successfully when you run them interactively, it suggests to me that it is related to API authorization on the identity runnig it. When you run a pipeline directly, the REST APIs execute under your identity context, and you may have all the required authorization - hence they succeed! 

     

    When the E2E pipeline orchestrates it, it is using the connection identity, which is probably lacking the Admin API access in tenant settings. 

     

    There could also be a case of concurrency, since you mention that not all errors are consistent. This is probably because the Admin APIs are throttled, and you're hitting those limits. 

     

    I suggest you use the following:

     

     - a service principal enabled with admin API setting for running the pipelines 

     - try running child pipelines sequentially or with a low batchcount, with retries

     

  • v-shchada-msft's avatar
    v-shchada-msft
    Community Support
    4 months ago

    Hi Shreya_Barhate,
     

    Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to the deborshi_nag  for sharing valuable insights. 
     

    Could you please confirm if your issue has been resolved using the suggested approach? This will help other community members facing similar scenarios. 
     

    Thank you.

     

    • Shreya_Barhate's avatar
      Shreya_Barhate
      Resolver II
      4 months ago

      hi v-shchada-msft , deborshi_nag ,

      Thanks for your response. Interestingly, the pipeline completed successfully later without any changes, so I’m trying to understand what caused the inconsistency in the earlier run(s).

      I’m running the pipelines via a service principal with the necessary Admin API settings enabled. Are there known intermittent issues with this approach, or could you suggest which telemetry/logs/settings I should validate to narrow this down?
      But for now issue is solved.


      Thanks,
      Shreya

      • deborshi_nag's avatar
        deborshi_nag
        Super User
        4 months ago

        Glad to know the pipeline completed successfully. Try to reduce the concurrency gradually and see if that reduces the occurances of these failures.