Error when schedulling with incremental refresh - how to find the blocking file

[LGH]

Hi, 

I'm trying to schedule a model with incremental refresh. When refreshing the model locally (with a small date range) I get no errors and can build the visuals as expected. However, when I schedule the model, I get following error during the historical load: 

Fout in de gegevensbronWe're sorry, an error occurred during evaluation.;class parquet::ParquetException (message: 'Couldn't deserialize thrift: TProtocolException: Invalid data Deserializing page header failed. ') class parquet::ParquetException (message: 'Couldn't deserialize thrift: TProtocolException: Invalid data Deserializing page header failed. ') class parquet::ParquetException (message: 'Couldn't deserialize thrift: TProtocolException: Invalid data Deserializing page header failed. '). The exception was raised by the IDataReader interface. Please review the error message and provider documentation for further information and corrective action.
URI clusterWABI-NORTH-EUROPE-G-PRIMARY-redirect.analysis.windows.net
Activiteits-id9fcb3db5-cd0e-4f8d-8fd4-056209f2ff05
Aanvraag-id9637511c-ca3a-40c5-9c2c-ca883b9f17ec
Tijd2024-06-08 09:24:11Z

I'm guessing that there is one (or multiple) parquet files without data or missing columns, but since we're speaking about 2000+ files it's almost impossible to check this manually. Is there a way to search efficiently for the file(s) that cause the problem since the error doesn't give me a lot of information.

Thanks! 

[LGH]

@lbendlin well we don't know that's kinda the issue. Possibly yes, but we don't know for certain since the query gets 4 years of daily data and manually checking each file is a lot of work. 

[lbendlin]

So what are you planning to do?

[LGH]

Well that's why I came here and asked the question? This was my exact question? How to find the file that contains possible errors in the most efficient way since the error PBI provides us doesn't tell us anything. 

[lbendlin]

Check the first file and the last file.  If they have the same structure then the reason must be elsewhere. If not,  check the file in the middle. Keep narrowing it down as if you were doing a sort.

[lbendlin]

Did the meta data change over time? Columns added/removed/renamed?