Enable Dataflow Gen2 to Dynamically Adjust Schema Mapping & Resource Paths Using Pipeline Parameters

[ChunPo]

Please consider enabling Dataflow Gen2 to:

1. Dynamically adjust schema mapping at runtime based on the resolved parameter value.
2. Allow resource paths (such as Lakehouse table names) to be fully driven by public parameters from pipelines.
3. Support end‑to‑end dynamic data ingestion patterns commonly used in modern data engineering.

@https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters#considerations-and-limitations

[miguel]

Hi! 

Could you tell us a bit more about the scenario for #1 and #3?

For #2, and specifically for your case with Lakehouse destinations, changing the name of your table is completely supported for Dataflow public parameters. You can follow the tutorial below for more information on how to make that happen:

Parameterized Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

[ChunPo]

I have contacted the support team and here is their response: Please review the document below, which outlines that using public parameters is limited when parameters change the resource path for either the source or destination. Additionally, parameters cannot be used to modify schema mapping. @https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters#considerations-and-limitations

[miguel]

That is correct information. However, changing the LakehouseId, WorkspaceId or TableName when using the Lakehouse connector (point #2) do not impact the resource path. By resource path we are referring to the connection component as described in the Dataflow item definition from the link below:

Dataflow definition - Microsoft Fabric REST APIs | Microsoft Learn

If you could tell us a bit more about your scenario for #1 and #3, perhaps with an example, that would help tremendously.

[AMladenov]

@miguel I have the following use-case for #1/#3 - Metadata driven dataflow which ingest X number of excel files from a Sharepoint, the parameters (sourcepath, filename, column mapping, column datatypes, destination schema, destination tablename, filters on columns and etc.) are read from a JSON files with a ForEach loop and fed to the dataflow. Currently I cannot manage to parametrize the data destination table schema, it does not follow the set schema in the dataflow for the current ingested file (at runtime), the one set at the time of dataflow creation through the UI persists.

[miguel]

@AMladenov by table schema you mean what columns your table must have during an output OR the schema of your destination such as a Lakehouse schema as defined in the link below?

Lakehouse schemas - Microsoft Fabric | Microsoft Learn

[AMladenov]

@miguel the columns of the table during output. "Data destination" column mapping does not update on run even on "Use automatic setting", dataflow needs to be republished.

[miguel]

thanks @AMladenov !

I was able to find two ideas that you can upvote that are requesting that exact feature:

Dataflow Gen2: Append With Dynamic Schema - Microsoft Fabric Community

Dynamic Schema limitation in DataFlow Gen2 - Microsoft Fabric Community

Do feel free to post any new ideas if perhaps your scenario differs from any of those so our team can take it into consideration.