Forum Discussion

__maca__'s avatar
__maca__
Advocate I
2 years ago

Pipeline | REST API | High CU % Usage

Hi,   I have set up a simple process to query an external API and retrieve data using Copy Data Activity. The issue I'm facing is that the CU % Usage keeps increasing linearly and eventually reache...
data engineering
Data Pipeline
Lakehouse
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi __maca__ 

 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

 

Thank you.

__maca__'s avatar
__maca__
Advocate I
2 years ago

Hi v-cboorla-msft,

Thanks for the detailed answer and sorry for the late reply. I’ve been busy trying out all the ideas you suggested. Unfortunately, I haven’t seen any improvement yet. The API we’re using limits us to 15 IDs at a time, which makes bulk retrieval somewhat impossible. I’ve set up my pipeline so that all the ForEach loops are triggered one after another and tried using higher batch sizes (the max allowed by Fabric is 50, so I couldn’t go beyond that). Do you think setting the ForEach activities to sequential within the loops could solve the problem? It might take forever, though...

 

There must be another way to make this work. I don’t see how switching to an even higher capacity makes sense for such a basic operation. I’d really appreciate it if you could provide me with other ideas or solutions.

 

Thanks!

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi __maca__ 

     

    Apologies for the inconvenience and the delay in response.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi __maca__ 

       

      We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

       

      Thank you.