Forum Discussion
Direct Lake Mode error: "Class: 'ParquetException' Status: 'Unexpected end of stream'"
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...
Here is the notebook code if you want to try to reproduce...the data is publically available. Let me know if there's a better way to share the notebook; I just copy/pasted each cell into the file. The bottom of the file also has some info about custom PBI dataset model.
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...