Forum Discussion

TGG360's avatar
TGG360
Frequent Visitor
7 months ago
Solved

Error Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2

Hello, I am encountering the following error when attempting to copy data from the Bronze Lakehouse to the Silver Lakehouse using Dataflow Gen2: Error: Extracted_data_WriteToDataDestination: There ...
Need Help
  • TGG360's avatar
    TGG360
    7 months ago

    I am using a pipeline as an alternative. I haven't figured out the root cause of the issue. Thanks

  • TGG360's avatar
    TGG360
    6 months ago

    The issue with the DF Gen2 integration has been resolved. The data source configuration should use the SQL connection string rather than the Lakehouse connector. After updating the data source to use the SQL connection string, all processes tested successfully.

Mauro89's avatar
Mauro89
Icon for Super User rankSuper User
7 months ago

Hi , 

it seems to me there might be an issue with the DeltaTable snapshot (as stated in the error message).

 

Try these:

 

In a notebook attached to the same lakehouse as the Dataflow source:

spark.sql("DESCRIBE HISTORY your_table_name").show()

If this fails → the Delta log is broken.

Then rebuild the table: delete the table first and then recreate it (via notebook or your dataflow).

 

Best regards!

PS: If you find this helpful, consider leaving kudos or mark it as solution.

TGG360

TGG360's avatar
TGG360
Frequent Visitor
7 months ago

Thank Mauro89  for your input. The code isnt failing. I was able to see the versions and the timestamps of the processes