Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Dataflow destination schema does not match with source

Hello, I have build ETL pipeline where I have  created dataflow to copy tables from lakehouse to warehouse. I also created  data model  in  warehouse and built report on top of it. However  when i...
Dataflows
anros's avatar
anros
Microsoft Employee
3 years ago

Hello Dipesh,

 

Not sure I fully understand the scenario from the customer.

  1. Copy a table from LH to WH using a dataflow (so this is Gen2 with output destination). All complex types should be stripped from this step.
  2. Publish and run the dataflow
  3. Do you edit the resulting table in WH ? We can see the 4 rows of Record type here, how did you get to this point ?
  4. Run the dataflow again

If you don't mind providing more information with reproducible steps it would help us provide you a better answer.

Could you also provide, from the refresh history of the failing dataflow, the following information:

  • Request ID
  • Session ID
  • Dataflow ID
  • Screenshot of the page shown when clicking on the activity starting with "WriteToDatabaseTable_"

Thank you,

Antoine

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Hello Antoine,

Thank youfor your reply.

Please find below details.

  • Request ID:  af4803e8-31ed-404b-b65b-e1c7fa0a662b
  • Session ID: c53e3e78-455f-45a9-be08-84e61bf896e6
  • Dataflow ID: c3facea7-19e7-4df3-ba52-45cb91382d37

I have 3 tables to copy from Lakehouse to respective table in warehouse

1.DailyPlan_New==>DailyPlan

2.Data==>Data

3.ProjectPlan==>ProjectPlan

I created additional 3 Views in WH for data modeling purpose.

1.Week_view

2.Developer_view

3.QA_view

The Data table in WH is connected with ProjectPlan,Developer_View,Week_View and QA_View with one to many relationship in Warehouse.However , now all these tables visble in schema of Data table in WH. It results in failure of copy activity  as source and destination schema does not match.

 

Thank you,

Dipesh123