Error refreshing the Dataflow Gen2 with multiple Lakehouse Schemas selected as the data destination

[Manojkumar29]

Issue:  

Error refreshing the Dataflow Gen2 with multiple Lakehouse Schemas selected as the data destination:

WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document. MashupException.Error: Expression.Error: We cannot convert the value null to type Table. Details: Reason = Expression.Error; ErrorCode = 10277; Microsoft.Data.Mashup.Error.Context = User". Error code: 104100. 

Investigation:  

Reproduction Steps: 

1. Create a Dataflow Gen2 and in the Data destination settings, choose the default Lakehouse destination and save it: 
   1. Proceed with the refresh. 
   2. The refresh completes successfully without any issues. 

2. Also, if we modify the Data destination to a Lakehouse and select the single schema for all destination tables and save it: 

   1. Trigger the refresh again. 
   2. The refresh succeeds as expected. 
3. But the issue is when we, update the Data destination to a Lakehouse, but this time map the tables to multiple different schemas instead of a single schema and save it:
   1. Initiate the refresh.
   2. The refresh fails and throws the below error: 
   3. 

Observed Behavior

• The failure occurs only when multiple schemas are configured as the destination.
• The issue is reproducible across refresh attempts.
• The Dataflow Gen2 UI allows selecting multiple schemas, but the refresh runtime does not handle this configuration correctly.

Additional Observation (UI Inconsistency)

When “Full hierarchy” is set to True in the Dataflow Gen2 data destination:

• The runtime effectively expects one schema for all tables.
• However, the UI still allows users to select different schemas per table.
• Saving this configuration is allowed, but refresh fails afterward.

This strongly suggests a design inconsistency rather than an intentional limitation.

Current Workaround & Impact

To work around this issue, we are currently:

• Creating multiple Dataflow Gen2 instances
• Each Dataflow writes to the same Lakehouse, but only one schema per Dataflow.

Due to this limitation, the number of Dataflows that need to be created and maintained has increased significantly.

Ask to Product Team

If multiple schemas are not supported, the UI should prevent users from configuring them.

If the UI allows this configuration (as it does today), Dataflow Gen2 should support writing to multiple schemas within the same Lakehouse without refresh failures.

We request the Product team to:

• Review this behavior as a potential defect or design gap
• Align the runtime behavior with the UI experience
• Consider enabling multi‑schema destinations in a single Dataflow Gen2, or clearly blocking it in the UI if unsupported

This enhancement would significantly improve usability, scalability, and customer experience when working with structured Lakehouse environments.