Forum Discussion
Direct Lake Mode error: "Class: 'ParquetException' Status: 'Unexpected end of stream'"
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...
There is a known bug for this "Unexpected end of stream" error - it can happen in some relatively uncommon Parquet column layouts. A fix for this should be rolling out around next week (hopefully by Sep 8th). Please give it a try after that...
Thanks for the info. I was able to resolve the issue today by removing two high cardinality comment columns. Neither are involved in the query that fails, but their removal solved the issue. They are both strings.
Do you think they meet the criteria that the bug will fix?I wouldn't have expected string columns to cause this - I think it was related to specific encodings. But it may be there there is a consequence to other columns that results from removing these two columns.
Note that this issue during an analysis phase, so it wouldn't matter which columns the query would access.
Here is the notebook code if you want to try to reproduce...the data is publically available. Let me know if there's a better way to share the notebook; I just copy/pasted each cell into the file. The bottom of the file also has some info about custom PBI dataset model.
