Forum Discussion
Direct Lake Mode error: "Class: 'ParquetException' Status: 'Unexpected end of stream'"
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...
Got this error as well when tesitng out fabric's new direct lake pbi connection mode.
I load a table form azure data lake gen 2 with shortcut to a lakehouse in microsoft fabric. I then create a dataset based on this lakehouse. The table I get trouble with is the largest in my model and has 75 000 000 rows.