Flattening nested JSON values in a dataflow with varying keys.

We are using Azrue DevOps REST API calls to return JSON files and storing them in blob. Then we perform a dataflow to transform the data. The issue is a portion of the JSON being stored in blob has varying keys.

When we specify the columns to map in a Select action, we are selecting specifically one of the varying keys from a list of options. But need to map ALL of these – we cannot manually specify these because the data source is so large.

We cannot implement a standard name for this section of the JSON. A wildcard for { } would work ideally but is not supported. We do not care what the keys are, just the contents (id, name).
 
Select Action:
Source Column:
resources.pipelines.{src-release}.pipeline.id
resources.pipelines.{src-release}.pipeline.name

resources.pipelines.{build }.pipeline.id

resources.pipelines.{build }.pipeline.name

Mapping Name as:
‘pipelineID’

‘pipelineName’


Below is a JSON snippet which highlights the key from the source JSON

Example of Select action mapping – each key shows as its own dropdown: