Forum Discussion
Direct Lake Mode error: "Class: 'ParquetException' Status: 'Unexpected end of stream'"
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...
I wouldn't have expected string columns to cause this - I think it was related to specific encodings. But it may be there there is a consequence to other columns that results from removing these two columns.
Note that this issue during an analysis phase, so it wouldn't matter which columns the query would access.
Here is the notebook code if you want to try to reproduce...the data is publically available. Let me know if there's a better way to share the notebook; I just copy/pasted each cell into the file. The bottom of the file also has some info about custom PBI dataset model.
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...