Copy data - Mapping section is Collection reference option

[ribba]

Hi,

Trying to load data from a rest API using "Copy data". Got it working with pagination and everything, but can't access the real data as the "Collection reference" is missing in the gui...?

Data coming through the API looks like this. Would like to get the data inside `values`.

[
  {
    "fullResultSize": 1383,
    "from": 0,
    "count": 1000,
    "versionDigest": "'If-None-Match' header not specified",
    "values": [
      {
        "id": 100686570,
        "version": 6,
        "url": "example.com",
        "project": {
          "id": 123456,
          "url": "example.com/133833984"
        },
      }
    ]
  }
]

Clicking "view source" on the object it is there but empty

{
"collectionReference": "",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": false,
"treatBooleanAsNumber": false
},
"columnFlattenSettings": {
"treatArrayAsString": false,
"treatStructAsString": false,
"flattenColumnDelimiter": "."
}

Any suggestions? Thanks!

[Joshgibbons]

I haven't seen a post on this, in a while. Did this somehow get solved?

[nbjdeunk]

We're missing this option too. 

We use this option a lot in pipelines in Azure Synapse. Currently we can't use Fabric because of the absence of this option.

Any updates on this? Or is there an Idea to vote for?

UPDATE

I found out that if you choose an 'External' sink first, the option will appear. You can set it at that time and change the sink afterwards to 'Workspace'. The setting 'Collection reference' is still set in the JSON definition. 

Then just map like you normally would. 

[emcus]

Hi,

This is stopping me as well. Any ideas on how to work around it?

[PrakashTamil]

I'm also facing similar issue 'collectionReference' property is not availble for copy data activity in Microsoft Fabric. Any plans to enable this property near future ?

[noahadmcdonald]

Any update on this? I'm finding a similar problem with the mapping. I think it's in part to the copy data activity not being able to recognize the array data type. I have never used Azure Data Factory but I see that Collection Reference and Advanced Editor exist in ADF but not in Microsoft Fabric. Is there a work around for this?

[ribba]

@v-cboorla-msft Any update on this?

[v-cboorla-msft]

Hi @ribba 

Apologies for the issue that you are facing here.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thanks.

[v-cboorla-msft]

Hi @ribba 

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

[AndersF]

So when just the source is selected the collection reference appears

But as soon as I select a destination, it disappears, regardless of the destination:

[AndersF]

Found a workaround. Roaming around with the json for the pipeline i found that by changing the value of typeConversion from true to false enables the collation reference. Assuming this will be fixed in the future this pipeline may break, but it works for now. 

[Joshgibbons]

Thanks for posting this. I verified it works.
I spent quite a while in the Pipeline JSON, trying to figure out a way to do something just like this (and finally gave up and came here).

[HenrikS2]

Hi! It's been a while since this thread was updated, any news on supporting "Collection reference" in Fabric?

[ribba]

Thanks !