Forum Discussion

pacman's avatar
pacman
New Member
7 months ago
Solved

Dataflow to Lakehouse WriteToDestination Failure

Hey guys, a bit puzzled as to why this might be happening and was wondering if any of y'all could help.   I have a table, ~80 col x ~ 2 mil rows with historical data. I am trying to run it via Data...
data engineering
data pipeline
lakehouse
  • pacman's avatar
    pacman
    7 months ago

    Hi v-ssriganesh 

    After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.

v-ssriganesh's avatar
v-ssriganesh
Community Support
7 months ago

Hi pacman,
Thank you for posting your query in the Microsoft Fabric Community Forum.

Since Microsoft Support has already recommended Fast Copy for Dataflow Gen2, that is the right next step to validate. Fast Copy can improve throughput by optimizing the data write path, which may help in scenarios where larger batches struggle to complete.

If Fast Copy does not resolve the issue for larger date ranges, the next action would be to continue working with Microsoft Support so they can review backend execution logs and advise on supported ingestion approaches for this workload.

Please feel free to update the thread with the outcome of the Fast Copy test.

Best regards,
Ganesh Singamshetty.