Connecting to a Rest API source - pipeline vs dataflow gen2

[pmscorca]

Hi,

I need to connect to a Dynamics 365 BC source by a Rest API.

I'm trying to implement and test a pipeline and a dataflow gen2.

I've noticed that a dataflow gen2 is more performant than a pipeline, but also significantly more expensive in CU: why?

Thinking to take advantage of dataflow gen2, does it exist any tricks to reduce the relative CU consumption?

Thanks

[v-lgarikapat]

@tayloramy   Thanks for your prompt responses.

@pmscorca  Thabnks for reaching out to fabric  community 

Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further

Thanks,

Lakshmi

[v-lgarikapat]

@pmscorca Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further

Thanks,

Lakshmi

[tayloramy · Accepted solution]

Hi @pmscorca, 

In general Dataflow Gen 2 is just more expensive. 

You can see the pricing calculations here: https://learn.microsoft.com/en-us/fabric/data-factory/pricing-dataflows-gen2

Compared to the Pipeline CU costs: https://learn.microsoft.com/en-us/fabric/data-factory/pricing-pipelines

If you want the most cost effective approach, that would probably be to use a Python notebook to pull the data out of the REST API.