topic Dataflow destination schema does not match with source in Dataflow

Dataflow destination schema does not match with source

Anonymous — Wed, 19 Jul 2023 10:27:25 GMT

Hello,

I have build ETL pipeline where I have created dataflow to copy tables from lakehouse to warehouse.

I also created data model in warehouse and built report on top of it.

However when i run pipeline for the second time it failed with following error:

"Dataflow refresh transaction failed with status CancelledInternalError"

while investigating the error , It is found that destination schema does not match with source schema and table related with source table shows in destination schema.

In mapping also those tables are visible in destination schema.

Can anyone help to resolve this issue?

Thank you.

Re: Dataflow destination schema does not match with source

anros — Thu, 20 Jul 2023 14:12:33 GMT

Hello Dipesh,

Not sure I fully understand the scenario from the customer.

Copy a table from LH to WH using a dataflow (so this is Gen2 with output destination). All complex types should be stripped from this step.
Publish and run the dataflow
Do you edit the resulting table in WH ? We can see the 4 rows of Record type here, how did you get to this point ?
Run the dataflow again

If you don't mind providing more information with reproducible steps it would help us provide you a better answer.

Could you also provide, from the refresh history of the failing dataflow, the following information:

Request ID
Session ID
Dataflow ID
Screenshot of the page shown when clicking on the activity starting with "WriteToDatabaseTable_"

Thank you,

Antoine

Re: Dataflow destination schema does not match with source

Anonymous — Thu, 20 Jul 2023 19:21:57 GMT

Hello Antoine,

Thank youfor your reply.

Please find below details.

Request ID: af4803e8-31ed-404b-b65b-e1c7fa0a662b
Session ID: c53e3e78-455f-45a9-be08-84e61bf896e6
Dataflow ID: c3facea7-19e7-4df3-ba52-45cb91382d37

I have 3 tables to copy from Lakehouse to respective table in warehouse

1.DailyPlan_New==>DailyPlan

2.Data==>Data

3.ProjectPlan==>ProjectPlan

I created additional 3 Views in WH for data modeling purpose.

1.Week_view

2.Developer_view

3.QA_view

The Data table in WH is connected with ProjectPlan,Developer_View,Week_View and QA_View with one to many relationship in Warehouse.However , now all these tables visble in schema of Data table in WH. It results in failure of copy activity as source and destination schema does not match.

Thank you,

Dipesh123

Re: Dataflow destination schema does not match with source

anros — Mon, 24 Jul 2023 08:38:59 GMT

Hello Dipesh,

Thanks for your answer. After some investigation, the behavior you are facing is a bug and we are tracking it.

In the interim I would recommend avoid linking tables and views.

Thank you,

Antoine