Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Dipesh123
Regular Visitor

Dataflow destination schema does not match with source

Hello,

I have build ETL pipeline where I have  created dataflow to copy tables from lakehouse to warehouse.

I also created  data model  in  warehouse and built report on top of it.

However  when i run pipeline for the second time it  failed  with following error:

"Dataflow refresh transaction failed with status CancelledInternalError"

while investigating the error , It is found that destination schema does not match with source schema and table related with source table   shows  in destination schema.

Dipesh123_1-1689762149585.png

In mapping also  those  tables are visible in destination schema.

Dipesh123_3-1689762263761.png

 

Can anyone help to resolve this issue?

Thank you.

 

3 REPLIES 3
anros
Microsoft Employee
Microsoft Employee

Hello Dipesh,

 

Thanks for your answer. After some investigation, the behavior you are facing is a bug and we are tracking it.

In the interim I would recommend avoid linking tables and views.

 

Thank you,

Antoine

anros
Microsoft Employee
Microsoft Employee

Hello Dipesh,

 

Not sure I fully understand the scenario from the customer.

  1. Copy a table from LH to WH using a dataflow (so this is Gen2 with output destination). All complex types should be stripped from this step.
  2. Publish and run the dataflow
  3. Do you edit the resulting table in WH ? We can see the 4 rows of Record type here, how did you get to this point ?
  4. Run the dataflow again

If you don't mind providing more information with reproducible steps it would help us provide you a better answer.

Could you also provide, from the refresh history of the failing dataflow, the following information:

  • Request ID
  • Session ID
  • Dataflow ID
  • Screenshot of the page shown when clicking on the activity starting with "WriteToDatabaseTable_"

Thank you,

Antoine

Hello Antoine,

Thank youfor your reply.

Please find below details.

  • Request ID:  af4803e8-31ed-404b-b65b-e1c7fa0a662b
  • Session ID: c53e3e78-455f-45a9-be08-84e61bf896e6
  • Dataflow ID: c3facea7-19e7-4df3-ba52-45cb91382d37

I have 3 tables to copy from Lakehouse to respective table in warehouse

1.DailyPlan_New==>DailyPlan

2.Data==>Data

3.ProjectPlan==>ProjectPlan

I created additional 3 Views in WH for data modeling purpose.

1.Week_view

2.Developer_view

3.QA_view

The Data table in WH is connected with ProjectPlan,Developer_View,Week_View and QA_View with one to many relationship in Warehouse.However , now all these tables visble in schema of Data table in WH. It results in failure of copy activity  as source and destination schema does not match.

 

Thank you,

Dipesh123

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.