Forum Discussion
Dataflow to Lakehouse WriteToDestination Failure
Hi v-ssriganesh
After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.
Hi deborshi_nag,
1. Optimized write is enabled by default I believe. Of the tables I have loaded successfully (this one table being the exception), I checked their most recent json and VORDER is enabled.
2. I am partitioning based on my date column. The main table contains ~2 1/2 years worth of data, I am loading by 6 month frame, starting initially with only ~4 month timeframe, much much smaller than the table's entireity.
3. All the columns are quite important, and although this is an option, I would like to avoid it. Are you asking this because, of my options available, I could later rejoin the columns via union in the Lakehouse?
4. It is folding to one SQL query which checks out.
Also a bit more backstory to this table - it is pulling from Databricks (which the table itself flows upstream from a silver environment) - not sure if it affects anything but it is of note. In addition, could this shared capacity have any play as to why this table won't load? I've had 1 months worth of data load in < a minute, but yet the 4 months not loading whatsoever.
Super Userare you saying this table is produced by Databricks - if so, is this an external table or a managed Databricks table?
Hi, this table is in our Databricks environment, and is being pulled through the Dataflow via the Databricks connector