topic Re: Use iteration object in data copy mapping in Pipelines

Use iteration object in data copy mapping

PanuO — Wed, 11 Oct 2023 13:29:29 GMT

I have dataflow which makes REST call to get different courses from API. For each course I need to get more detailed content. I have dataflow that uses Data Copy action to get all the courses and then I have For Each action that iterates every item received from API. I am able to do an another REST API call to get detailed course information, but the API response does not have the course identifier. I have it in the iteration item, but I cannot use it in mapping section. If I try to put item().id or {item().id} it produces an error:

ErrorCode=UserErrorSchemaMappingCannotInferSinkColumnType,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Data type of column 'id' can't be inferred from 1st row of data, please specify its data type in mappings of copy activity or structure of DataSet.,Source=Microsoft.DataTransfer.Common,'

Is there a way to use iteration object in mapping / should I do my dataflow in some another way? I have used the item().id in relative path, so I know it works there.

Re: Use iteration object in data copy mapping

Anonymous — Thu, 12 Oct 2023 09:37:41 GMT

Hi PanuO - Thanks for using Fabric Community,

As I understand you are trying to fetch data using API in Fabric Data Factory with the help of pipelines.

Can you please confirm if your pipeline design is similar to below screen shot?

Can you please share some details along with screenshots regarding your approach and issue.
I would be able to help you better if you can provide these details.

Re: Use iteration object in data copy mapping

PanuO — Thu, 12 Oct 2023 09:47:09 GMT

That is quite close. I now have datalake as input, because it is faster to test that way (I have stored data in there so I don't want to fetch it all the time when I test this).

The foreach has following Items setting:

@activity('Get courses from datalake').output.value

This is the pipeline:

Re: Use iteration object in data copy mapping

Anonymous — Thu, 12 Oct 2023 10:26:16 GMT

Hi PanuO - Thanks for your response.

Can you please share a complete screenshot of for loop settings and copy data activity settings.

Eg:
For Loop -

Copy Data -

Re: Use iteration object in data copy mapping

PanuO — Thu, 12 Oct 2023 10:29:21 GMT

Loop:

Copy Data:

(The API has result objects inside array called results, so thats why I need to use the collection reference in mapping)

Re: Use iteration object in data copy mapping

Anonymous — Thu, 12 Oct 2023 11:17:36 GMT

Hi PanuO ,

Can you please add column type to int (or respective type) and try to rerun the pipeline, let me know if you are still facing the same issue.

Add on - Please refer these videos that might help you with your issue.

32. Copy data from REST API which sends response in Pages using Azure data factory - YouTube
21. Dynamic Column mapping in Copy Activity in Azure Data Factory - YouTube

Thank you