Flattening nested JSON values in a dataflow with varying keys.

[Velegandla]

We are using Azrue DevOps REST API calls to return JSON files and storing them in blob. Then we perform a dataflow to transform the data. The issue is a portion of the JSON being stored in blob has varying keys.

When we specify the columns to map in a Select action, we are selecting specifically one of the varying keys from a list of options. But need to map ALL of these – we cannot manually specify these because the data source is so large.

We cannot implement a standard name for this section of the JSON. A wildcard for { } would work ideally but is not supported. We do not care what the keys are, just the contents (id, name).
 
Select Action:
Source Column:
resources.pipelines.{src-release}.pipeline.id
resources.pipelines.{src-release}.pipeline.name

resources.pipelines.{build }.pipeline.id

resources.pipelines.{build }.pipeline.name

Mapping Name as:
‘pipelineID’

‘pipelineName’


Below is a JSON snippet which highlights the key from the source JSON

Example of Select action mapping – each key shows as its own dropdown:

[Anonymous]

Hi @Velegandla 
Thanks for using Fabric Community.
Can you please confirm if you are trying the above scenario with ADF dataflow or Fabric Dataflows?
thanks

[Velegandla]

@Anonymous 

ADF dataflow

[Anonymous · Accepted solution]

Hi @Velegandla 

Apologies for the inconvenience, this community is specifically for discussions and questions related to Microsoft Fabric. For queries related to Azure Data Factory, we recommend using the dedicated Microsoft Q&A forum where they are ready to assist you better. Here's the link to post your query in Microsoft Q&A : Ask a question - Microsoft Q&A.

Thanks.