Question about Dataflow Gen2 runtime remapping across multiple tables

Hi Johanny_O ,

Thank you for reaching out to the Microsoft Community Forum.

As mentioned by GilbertQ , Please try Notebook/Pyspark  instead of Dataflow Gen2.  Dataflow Gen2 does not fully support runtime schema remapping across heterogeneous tables in a single reusable dataflow. It is schema-flexible within a query, but not schema-dynamic at execution level across different entities. It can be used when you have limited number of tables.

Notebook / spark is a true runtime schema inference, No pre-bound column mapping and can read any table dynamically, cast all columns to string and write dynamically with overwrite/append. Please refer below sample Pyspark code.

df = spark.read.table(f"{p_source_schema}.{p_source_table}")
df_str = df.select([col(c).cast("string") for c in df.columns])
df_str.write.mode("overwrite").saveAsTable(f"{p_dest_schema}.{p_dest_table}")

Note: Use Notebook / spark for Single reusable ingestion process across many tables with different schemas and for Bronze ingestion. Dataflow Gen2 for standardized transformations, fixed-schema pipelines and curated layers.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh