Forum Discussion
Direct Lake Mode error: "Class: 'ParquetException' Status: 'Unexpected end of stream'"
This issue has been resolved. My dim and fact tables had skewed data which was causing the issue (somehow). Running OPTIMIZE on each of the tables and then refreshing my PBI dataset allowed the measures to calculate successfully.
One thing that is unclear to me: I explicitly enabled OptimizeWrite (and it's enabled by default https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql#what-is-optimized-write) so why were explicit OPTIMIZE commands required as well? I assumed OptimizeWrite would result in...optimized writes...
I fixed the issue by changing the databricks runtime version to 11.3 TLS, with spark version 3.3.0, which is the same that fabric uses. After that I have deleted the folder with the delta table in the storage account. Finally, I recreated the delta table and the issue was resolved. Hopefully MS comes with a better solution.