Forum Discussion
Question about Dataflow Gen2 runtime remapping across multiple tables
Hi Johanny_O ,
Thank you for reaching out to the Microsoft Community Forum.
As mentioned by GilbertQ , Please try Notebook/Pyspark instead of Dataflow Gen2. Dataflow Gen2 does not fully support runtime schema remapping across heterogeneous tables in a single reusable dataflow. It is schema-flexible within a query, but not schema-dynamic at execution level across different entities. It can be used when you have limited number of tables.
Notebook / spark is a true runtime schema inference, No pre-bound column mapping and can read any table dynamically, cast all columns to string and write dynamically with overwrite/append. Please refer below sample Pyspark code.df = spark.read.table(f"{p_source_schema}.{p_source_table}")
df_str = df.select([col(c).cast("string") for c in df.columns])
df_str.write.mode("overwrite").saveAsTable(f"{p_dest_schema}.{p_dest_table}")Note: Use Notebook / spark for Single reusable ingestion process across many tables with different schemas and for Bronze ingestion. Dataflow Gen2 for standardized transformations, fixed-schema pipelines and curated layers.
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Hello Johanny_O Dataflow Gen2 allows parameterisation, enabling you to execute the same process with different values each time within a pipeline. Please refer to the Microsoft documentation for details on how this works.
Parameterized Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Hello, thank you for your suggestion and for sharing the Microsoft documentation.
We followed the guidance in the Parameterized Dataflow Gen2 article and successfully implemented parameterized source and destination tables inside a pipeline.
Our tests confirmed that:
Runtime parameter override works correctly
A single Dataflow Gen2 can dynamically change the destination table name
The pipeline successfully passes parameters through a ForEach loop
However, we encountered a limitation when loading multiple tables with different schemas through the same parameterized Dataflow Gen2.
Although the destination table is created dynamically, the destination column mapping appears to remain tied to the schema originally configured and published in the Dataflow. As a result, the mapping is not automatically regenerated for different table schemas during runtime execution within the same ForEach loop.
In practice, our tests suggest that:
One parameterized Dataflow Gen2 can successfully change destination tables
But column mappings are not dynamically rebuilt for heterogeneous schemas at runtime
Our current understanding is that for large Bronze ingestion across multiple different schemas, the practical options may be:
One Dataflow Gen2 per table/schema
Or a metadata-driven Notebook/Spark approach
Could you please confirm whether this behavior is expected, or if there is an additional configuration step (for example around destination query editing or dynamic mapping) that enables fully dynamic schema mapping within a single parameterized Dataflow Gen2?
Thank you again for your help.