When using metadata-driven pipeline frameworks, the parameterized pipelines show up in the Monitoring Hub under the same pipeline name. It would be useful to be able to surface a custom value in the Monitoring Hub, and pass a value into that from pipelines dynamically. For example, imagine a metadata-driven pipeline that performs a lookup on a list of tables to be ingested, and then kicks off many dynamic children pipelines, one for each table that needs to be ingested. Today, all of these children pipelines will appear in the Monitoring Hub under the same generic pipeline name. If we want to find the pipeline for a particular table, we have to click into each pipeline to see what the inputs were. This is very inefficient at scale. If there was a custom field that could be populated for each run of the dynamic pipeline and surfaced in the Monitoring Hub, we could pass in the table name, enabling the ability to filter for a particular table's ingestion pipeline instance in the Monitoring Hub.
... View more