Forum Discussion
Dataflow to Lakehouse WriteToDestination Failure
Hi v-ssriganesh
After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.
Community SupportHi pacman,
Thank you for posting your query in the Microsoft Fabric Community Forum.
Since Microsoft Support has already recommended Fast Copy for Dataflow Gen2, that is the right next step to validate. Fast Copy can improve throughput by optimizing the data write path, which may help in scenarios where larger batches struggle to complete.
If Fast Copy does not resolve the issue for larger date ranges, the next action would be to continue working with Microsoft Support so they can review backend execution logs and advise on supported ingestion approaches for this workload.
Please feel free to update the thread with the outcome of the Fast Copy test.
Best regards,
Ganesh Singamshetty.
Hi v-ssriganesh,
The Fast Copy attempt did not work - it failed almost instantly. It appears Fast Copy is not supported via the Databricks connector. I have been working hand in hand with Microsoft's team to try and get this issue resolved, currently it seems a bit bigger than initially suggested.
- v-ssriganesh
Hello pacman,
Thank you for the update. If you have found a solution or workaround, please consider sharing it here, as it could help others facing the same issue.Thank you for your time and contribution.
Hi v-ssriganesh
After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.