topic Re: Copy data - Mapping section is Collection reference option in Pipelines

Copy data - Mapping section is Collection reference option

ribba — Tue, 12 Dec 2023 00:04:48 GMT

Hi,

Trying to load data from a rest API using "Copy data". Got it working with pagination and everything, but can't access the real data as the "Collection reference" is missing in the gui...?

Data coming through the API looks like this. Would like to get the data inside `values`.

[ { "fullResultSize": 1383, "from": 0, "count": 1000, "versionDigest": "'If-None-Match' header not specified", "values": [ { "id": 100686570, "version": 6, "url": "example.com", "project": { "id": 123456, "url": "example.com/133833984" }, } ] } ]

Clicking "view source" on the object it is there but empty

{ "collectionReference": "", "typeConversion": true, "typeConversionSettings": { "allowDataTruncation": false, "treatBooleanAsNumber": false }, "columnFlattenSettings": { "treatArrayAsString": false, "treatStructAsString": false, "flattenColumnDelimiter": "." }

Any suggestions? Thanks!

Re: Copy data - Mapping section is Collection reference option

v-cboorla-msft — Tue, 12 Dec 2023 15:55:03 GMT

Hi ribba

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

Re: Copy data - Mapping section is Collection reference option

ribba — Wed, 13 Dec 2023 11:54:20 GMT

Thanks !

Re: Copy data - Mapping section is Collection reference option

ribba — Tue, 09 Jan 2024 14:24:41 GMT

v-cboorla-msft Any update on this?

Re: Copy data - Mapping section is Collection reference option

v-cboorla-msft — Thu, 11 Jan 2024 06:07:12 GMT

Hi ribba

Apologies for the issue that you are facing here.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thanks.

Re: Copy data - Mapping section is Collection reference option

noahadmcdonald — Mon, 15 Jan 2024 20:33:49 GMT

Any update on this? I'm finding a similar problem with the mapping. I think it's in part to the copy data activity not being able to recognize the array data type. I have never used Azure Data Factory but I see that Collection Reference and Advanced Editor exist in ADF but not in Microsoft Fabric. Is there a work around for this?

Re: Copy data - Mapping section is Collection reference option

PrakashTamil — Mon, 05 Feb 2024 19:14:20 GMT

I'm also facing similar issue 'collectionReference' property is not availble for copy data activity in Microsoft Fabric. Any plans to enable this property near future ?

Re: Copy data - Mapping section is Collection reference option

emcus — Wed, 13 Mar 2024 13:27:22 GMT

Hi,

This is stopping me as well. Any ideas on how to work around it?

Re: Copy data - Mapping section is Collection reference option

HenrikS2 — Thu, 04 Apr 2024 21:58:45 GMT

Hi! It's been a while since this thread was updated, any news on supporting "Collection reference" in Fabric?

Re: Copy data - Mapping section is Collection reference option

nbjdeunk — Fri, 05 Apr 2024 12:49:00 GMT

We're missing this option too.

We use this option a lot in pipelines in Azure Synapse. Currently we can't use Fabric because of the absence of this option.

Any updates on this? Or is there an Idea to vote for?

UPDATE

I found out that if you choose an 'External' sink first, the option will appear. You can set it at that time and change the sink afterwards to 'Workspace'. The setting 'Collection reference' is still set in the JSON definition.

Then just map like you normally would.

Re: Copy data - Mapping section is Collection reference option

Joshgibbons — Tue, 22 Oct 2024 19:06:38 GMT

I haven't seen a post on this, in a while. Did this somehow get solved?

Re: Copy data - Mapping section is Collection reference option

AndersF — Thu, 21 Nov 2024 13:07:57 GMT

So when just the source is selected the collection reference appears

But as soon as I select a destination, it disappears, regardless of the destination:

Re: Copy data - Mapping section is Collection reference option

AndersF — Thu, 21 Nov 2024 16:01:21 GMT

Found a workaround. Roaming around with the json for the pipeline i found that by changing the value of typeConversion from true to false enables the collation reference. Assuming this will be fixed in the future this pipeline may break, but it works for now.

Re: Copy data - Mapping section is Collection reference option

Joshgibbons — Fri, 22 Nov 2024 18:21:30 GMT

Thanks for posting this. I verified it works.
I spent quite a while in the Pipeline JSON, trying to figure out a way to do something just like this (and finally gave up and came here).