Dataflow destination schema does not match with source

[Dipesh123]

Hello,

I have build ETL pipeline where I have  created dataflow to copy tables from lakehouse to warehouse.

I also created  data model  in  warehouse and built report on top of it.

However  when i run pipeline for the second time it  failed  with following error:

"Dataflow refresh transaction failed with status CancelledInternalError"

while investigating the error , It is found that destination schema does not match with source schema and table related with source table   shows  in destination schema.

In mapping also  those  tables are visible in destination schema.

Can anyone help to resolve this issue?

Thank you.

[anros]

Hello Dipesh,

Thanks for your answer. After some investigation, the behavior you are facing is a bug and we are tracking it.

In the interim I would recommend avoid linking tables and views.

Thank you,

Antoine

[anros]

Hello Dipesh,

Not sure I fully understand the scenario from the customer.

1. Copy a table from LH to WH using a dataflow (so this is Gen2 with output destination). All complex types should be stripped from this step.
2. Publish and run the dataflow
3. Do you edit the resulting table in WH ? We can see the 4 rows of Record type here, how did you get to this point ?
4. Run the dataflow again

If you don't mind providing more information with reproducible steps it would help us provide you a better answer.

Could you also provide, from the refresh history of the failing dataflow, the following information:

• Request ID
• Session ID
• Dataflow ID
• Screenshot of the page shown when clicking on the activity starting with "WriteToDatabaseTable_"

Thank you,

Antoine

[Dipesh123]

Hello Antoine,

Thank youfor your reply.

Please find below details.

• Request ID:  af4803e8-31ed-404b-b65b-e1c7fa0a662b
• Session ID: c53e3e78-455f-45a9-be08-84e61bf896e6
• Dataflow ID: c3facea7-19e7-4df3-ba52-45cb91382d37

I have 3 tables to copy from Lakehouse to respective table in warehouse

1.DailyPlan_New==>DailyPlan

2.Data==>Data

3.ProjectPlan==>ProjectPlan

I created additional 3 Views in WH for data modeling purpose.

1.Week_view

2.Developer_view

3.QA_view

The Data table in WH is connected with ProjectPlan,Developer_View,Week_View and QA_View with one to many relationship in Warehouse.However , now all these tables visble in schema of Data table in WH. It results in failure of copy activity  as source and destination schema does not match.

Thank you,

Dipesh123