Forum Discussion

pacman's avatar
pacman
New Member
7 months ago
Solved

Dataflow to Lakehouse WriteToDestination Failure

Hey guys, a bit puzzled as to why this might be happening and was wondering if any of y'all could help.   I have a table, ~80 col x ~ 2 mil rows with historical data. I am trying to run it via Data...
data engineering
data pipeline
lakehouse
  • pacman's avatar
    pacman
    7 months ago

    Hi v-ssriganesh 

    After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.

tayloramy's avatar
tayloramy
Super User
7 months ago

Hi pacman, 

 

Are there any really large text or binary columns in your table? 
I've seen dataflows (and pipelines) struggle with really large BLOB or text columns. 

 

 

  • pacman's avatar
    pacman
    New Member
    7 months ago

    Hello tayloramy ,

    Suprisingly no large text nor binary columns. Mostly consists of brief descriptionary text (age, gender, etc.) and identifying keys. Microsoft support suggested using Fast Copy for Dataflow Gen2, so that is on my list of things to try when I run again over night.