CopyJob - add to Invoke Pipeline?

[fb200]

I want orchestrate all my pipelines (pl1 ->pl2 -> cj - pl4)> with invoke pipelines. I have not found a way to add copyjob. Anyone has a solution?

[fb200]

Hi Yang,

thanks for your feedback and thanks for the workaround.

I use copyjob with incremental load. And I like the no-code capability to append the tables.

I believe its a big benefit to add new data to exisiting tables. 

Is there any plan on the roadmap to add this feature to the invoke pipelines?

[v-huijiey-msft · Accepted solution]

Hi @fb200 ,

The roadmap does not see a rollout plan for the feature to use copy jobs (preview) in the data pipeline at the moment:

What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

You can give feedback on Home.

Data Factory Dataflow Gen2 also offers incremental importing. This feature allows you to load only new or updated data, thus increasing the efficiency of data loading. This is similar to the incremental import feature of Copy Job.

For more information and examples, please refer to the official documentation below:

Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[fb200]

Hi Yang, 

I really appreciate your support. I tried the OData connector in Gen2, but the authentication options were only anonymous and basic. It seems this has been updated.

I will give this a try again.

Thanks 

Florian

[v-huijiey-msft]

Hi @fb200 ,

There is no option to use the copy job(preview) function on the data pipeline canvas.

I created a copy job(preview) in workspace and wrote its data to a warehouse, which I think is a data extraction as well as usage process, and dataflow and pipeline can be used instead.

One small note: when using multiple pipelines in one pipeline please select the “invoke pipeline(legacy)” option instead of the “invoke pipeline(preview)” option, which is currently only supported with ADF & Synapse pipelines.

Here's a screenshot of my test running in pipeline with copy job(preview) using dataflow instead.

The preview feature may have some limitations, so use an alternative if one is available.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!