Forum Discussion
Question about Dataflow Gen2 runtime remapping across multiple tables
Hi Johanny_O ,
Thank you for reaching out to the Microsoft Community Forum.
As mentioned by GilbertQ , Please try Notebook/Pyspark instead of Dataflow Gen2. Dataflow Gen2 does not fully support runtime schema remapping across heterogeneous tables in a single reusable dataflow. It is schema-flexible within a query, but not schema-dynamic at execution level across different entities. It can be used when you have limited number of tables.
Notebook / spark is a true runtime schema inference, No pre-bound column mapping and can read any table dynamically, cast all columns to string and write dynamically with overwrite/append. Please refer below sample Pyspark code.df = spark.read.table(f"{p_source_schema}.{p_source_table}")
df_str = df.select([col(c).cast("string") for c in df.columns])
df_str.write.mode("overwrite").saveAsTable(f"{p_dest_schema}.{p_dest_table}")Note: Use Notebook / spark for Single reusable ingestion process across many tables with different schemas and for Bronze ingestion. Dataflow Gen2 for standardized transformations, fixed-schema pipelines and curated layers.
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Hi Johanny_O
Fold this type of ingestion pattern. I would highly recommend using a Python notebook for your data ingestion. It is much more flexible and can do what you're looking for. Dataflow Gen 2 can work, but it does have some nuances as you have experienced.
Thank you, this is very helpful and aligns with what we observed during testing.
We were able to parameterize source and destination tables successfully and execute the Dataflow through a pipeline and ForEach loop.
However, the challenge appears to be around destination column mapping when processing multiple tables with different schemas through the same Dataflow Gen2. The destination mapping seems to remain tied to the originally published schema rather than being rebuilt dynamically at runtime.
Your comment about DF Gen2 having "nuances" is very helpful.
Just to confirm our understanding:
For a fully metadata-driven Bronze ingestion pattern across heterogeneous schemas (customers, orders, products, etc.) using a single reusable process, would you consider Notebook/Python the recommended architecture rather than relying on one reusable Dataflow Gen2?
We would appreciate your confirmation on whether this limitation around runtime mapping behavior is expected in Dataflow Gen2.
Thank you again for your guidance.
