Question about Dataflow Gen2 runtime remapping across multiple tables

[Johanny_O]

In Microsoft Fabric Dataflow Gen2, is it supported to use a single parameterized Dataflow for fully metadata-driven Bronze ingestion across multiple tables with different schemas?

Scenario:

• Pipeline passes runtime parameters:

  • p_source_schema

  • p_source_table

  • p_dest_schema

  • p_dest_table

• Dataflow source navigation is dynamic using parameters

• Destination uses dynamic schema and automatic settings

• Goal is:

  • same DF2 reused for customers/orders/products/etc.

  • automatic runtime column mapping

  • all columns converted dynamically to string/text

Problem:

• The Dataflow still appears to retain metadata/mapping from the first previewed table (customers)

• Runtime refresh either:

  • creates the wrong destination table

  • or fails with generic EntityUserFailure / Mashup errors

Question:
Does DF2 truly support runtime automatic remapping for different table schemas inside a single reusable Dataflow, or is the recommended architecture:

• one DF2 per table

• or Notebook/Spark for metadata-driven Bronze ingestion?

[v-dineshya]

Hi @Johanny_O ,

Thank you for reaching out to the Microsoft Community Forum.

As mentioned by @GilbertQ , Please try Notebook/Pyspark  instead of Dataflow Gen2.  Dataflow Gen2 does not fully support runtime schema remapping across heterogeneous tables in a single reusable dataflow. It is schema-flexible within a query, but not schema-dynamic at execution level across different entities. It can be used when you have limited number of tables.

Notebook / spark is a true runtime schema inference, No pre-bound column mapping and can read any table dynamically, cast all columns to string and write dynamically with overwrite/append. Please refer below sample Pyspark code.

df = spark.read.table(f"{p_source_schema}.{p_source_table}")
df_str = df.select([col(c).cast("string") for c in df.columns])
df_str.write.mode("overwrite").saveAsTable(f"{p_dest_schema}.{p_dest_table}")

Note: Use Notebook / spark for Single reusable ingestion process across many tables with different schemas and for Bronze ingestion. Dataflow Gen2 for standardized transformations, fixed-schema pipelines and curated layers.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

[v-dineshya]

Hi @Johanny_O ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

[v-dineshya]

Hi @Johanny_O ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

[GilbertQ]

Hi @Johanny_O 

Fold this type of ingestion pattern. I would highly recommend using a Python notebook for your data ingestion. It is much more flexible and can do what you're looking for. Dataflow Gen 2 can work, but it does have some nuances as you have experienced.

[Johanny_O]

Thank you, this is very helpful and aligns with what we observed during testing.

We were able to parameterize source and destination tables successfully and execute the Dataflow through a pipeline and ForEach loop.

However, the challenge appears to be around destination column mapping when processing multiple tables with different schemas through the same Dataflow Gen2. The destination mapping seems to remain tied to the originally published schema rather than being rebuilt dynamically at runtime.

Your comment about DF Gen2 having "nuances" is very helpful.

Just to confirm our understanding:

For a fully metadata-driven Bronze ingestion pattern across heterogeneous schemas (customers, orders, products, etc.) using a single reusable process, would you consider Notebook/Python the recommended architecture rather than relying on one reusable Dataflow Gen2?

We would appreciate your confirmation on whether this limitation around runtime mapping behavior is expected in Dataflow Gen2.

Thank you again for your guidance.

[deborshi_nag]

Hello @Johanny_O Dataflow Gen2 allows parameterisation, enabling you to execute the same process with different values each time within a pipeline. Please refer to the Microsoft documentation for details on how this works. 

Parameterized Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

[Johanny_O]

Hello, thank you for your suggestion and for sharing the Microsoft documentation.

We followed the guidance in the Parameterized Dataflow Gen2 article and successfully implemented parameterized source and destination tables inside a pipeline.

Our tests confirmed that:

• Runtime parameter override works correctly

• A single Dataflow Gen2 can dynamically change the destination table name

• The pipeline successfully passes parameters through a ForEach loop

However, we encountered a limitation when loading multiple tables with different schemas through the same parameterized Dataflow Gen2.

Although the destination table is created dynamically, the destination column mapping appears to remain tied to the schema originally configured and published in the Dataflow. As a result, the mapping is not automatically regenerated for different table schemas during runtime execution within the same ForEach loop.

In practice, our tests suggest that:

• One parameterized Dataflow Gen2 can successfully change destination tables

• But column mappings are not dynamically rebuilt for heterogeneous schemas at runtime

Our current understanding is that for large Bronze ingestion across multiple different schemas, the practical options may be:

• One Dataflow Gen2 per table/schema

• Or a metadata-driven Notebook/Spark approach

Could you please confirm whether this behavior is expected, or if there is an additional configuration step (for example around destination query editing or dynamic mapping) that enables fully dynamic schema mapping within a single parameterized Dataflow Gen2?

Thank you again for your help.