Forum Discussion
Dataflow to Lakehouse WriteToDestination Failure
Hi v-ssriganesh
After much trial and error myself and a Microsoft employee were able to fix the issue. It is a bug dealing with the 2.0 connector - ADBC driver. Switching it to 1.0 and using the ODBC driver worked. Apparently this will be fixed in an update soon, but for now, I have swapped all Databricks connections to 1.0 and they work fine.
Super UserHi pacman
There are a couple of reasons why Power BI Import could be taking significantly less time as compared to Dataflows Gen2. The engines behind these two processes are different, and their write paths are also different.
Power Import uses VertiPaq engine to build the semantic model, whereas Dataflows Gen2 writes into many smaller Parquet files, and additionaly may also do schema drift checks, data type inference, null handling and automatically applied transformations. This increases the metadata and transaction overhead and may slow down the copy.
Can you check for a few things?
- Optimized Write is enabled?
- are you partitioning?
- can you reduce the columns?
- Check folding: Use “View Native Query” in Power Query Online.
Hope this helps - please appreciate leaving a Kudos or accepting as a Solution!