topic Re: Exploit the output of a data pipeline as with Azure Data Factory in Pipelines

Exploit the output of a data pipeline as with Azure Data Factory

vivien57 — Fri, 11 Oct 2024 13:39:04 GMT

Hello everyone,

Is it possible at the moment to use output like this in the Data Pipeline? My tests are not working.

https://stackoverflow.com/questions/78556016/how-to-get-row-count-of-the-result-from-exists-activity-in-azure-data-factory-da

If not, how do you recover the number of lines inserted by a Data Pipeline or Copy Data in the Data Pipeline ?

Thank you in advance for your help,

Have a nice day,

Vivien

Re: Exploit the output of a data pipeline as with Azure Data Factory

spencer_sa — Fri, 11 Oct 2024 15:21:16 GMT

You get a whole stack of data in the output array of the copy data activity including rows read and rows copied (see attached image)
You can then query this in later steps using syntax like;

@activity('Copy data1').output.rowsRead or @activity('Copy data1').output.rowsCopied

or even output as a pipeline return variable using a Set Variable activity.

Re: Exploit the output of a data pipeline as with Azure Data Factory

Anonymous — Mon, 14 Oct 2024 02:05:04 GMT

Hi vivien57 ,

Currently in Fabric Data Factory it is not possible to use @activity('Copy data1').output.rowsRead or @activity('Copy data1').output.rowsCopied like Azure Data Factory for dynamic content.

The only alternative is to process the data in Dataflow Gen2 and then transfer the data through Data Pipeline or Copy Data, you can check this official document for more details: Move and transform data with dataflow and data pipelines - Microsoft Fabric | Microsoft Learn

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Exploit the output of a data pipeline as with Azure Data Factory

vivien57 — Mon, 14 Oct 2024 05:33:38 GMT

Hello,

Thanks for your feedback.

It's weird because it's not available in autocomplete, but it works 🙂 with a different sequence of functions from ADF.

Thanks a lot.

Have a nice day,

Vivien

Re: Exploit the output of a data pipeline as with Azure Data Factory

vivien57 — Mon, 14 Oct 2024 05:34:44 GMT

Hello

As spencer_sa wrote, it works
Have a nice day,
Vivien