Forum Discussion
Dataflow to Lakehouse WriteToDestination Failure
Hi v-ssriganesh
After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.
Hi pacman
There are a couple of reasons why Power BI Import could be taking significantly less time as compared to Dataflows Gen2. The engines behind these two processes are different, and their write paths are also different.
Power Import uses VertiPaq engine to build the semantic model, whereas Dataflows Gen2 writes into many smaller Parquet files, and additionaly may also do schema drift checks, data type inference, null handling and automatically applied transformations. This increases the metadata and transaction overhead and may slow down the copy.
Can you check for a few things?
- Optimized Write is enabled?
- are you partitioning?
- can you reduce the columns?
- Check folding: Use “View Native Query” in Power Query Online.
Hope this helps - please appreciate leaving a Kudos or accepting as a Solution!
Hi deborshi_nag,
1. Optimized write is enabled by default I believe. Of the tables I have loaded successfully (this one table being the exception), I checked their most recent json and VORDER is enabled.
2. I am partitioning based on my date column. The main table contains ~2 1/2 years worth of data, I am loading by 6 month frame, starting initially with only ~4 month timeframe, much much smaller than the table's entireity.
3. All the columns are quite important, and although this is an option, I would like to avoid it. Are you asking this because, of my options available, I could later rejoin the columns via union in the Lakehouse?
4. It is folding to one SQL query which checks out.
Also a bit more backstory to this table - it is pulling from Databricks (which the table itself flows upstream from a silver environment) - not sure if it affects anything but it is of note. In addition, could this shared capacity have any play as to why this table won't load? I've had 1 months worth of data load in < a minute, but yet the 4 months not loading whatsoever.
are you saying this table is produced by Databricks - if so, is this an external table or a managed Databricks table?
Hi, this table is in our Databricks environment, and is being pulled through the Dataflow via the Databricks connector