Incremental refresh data source bug

[DMM]

Without incremental refresh, scheduled refresh works. As soon as I enable incremental refresh, the refresh fails because of "invalid credentials".

The data source is a folder, let's say it's called ChildFolder, and it is located at ServerName\ParentFolder\ChildFolder

The gateway has ServerName\ParentFolder added as a data source, which is why the refresh works in the first place. However, incremental refresh says the credentials for ChildFolder are invalid. This makes no sense because the access has been granted to its parent folder, and scheduled refresh works without incremental refersh.

[v-qiuyu-msft]

Hi @DMM, 

Did you get data from Excel file? 

I find a similar issue reported internally: CRI 178162795. Excel files does not support Query Folding that is required for Incremental Refresh:
https://docs.microsoft.com/en-us/power-query/power-query-folding#sources-that-support-folding

Best Regards,
Qiuyun Yu

[Anonymous]

Hi DMM,

We have been having the same issue for the last couple of weeks, the only way to get the refresh to succeed with incremental was to remove all our excel sources. It's been refreshing successfully with Incremental off.

Issue only started after we tried refreshing it after publishing from desktop. The dataset refreshed fine up until that point with incremental on.

The excel sources are all stored in a folder with permission given on the parent directory as you have. The incremental queries do not contain any data from excel sources and all query fold.

We have been in touch with Microsoft support, the latest we have is there's nothing to solve it currently and they are looking into a solution for mid april.

[DMM]

@Anonymous @v-qiuyu-msft I have pinpointed the issue, and it's not Excel files. I have successfully configured incremental refresh on a folder, where all files are Excel. I've done that in a brand new file, and the only loaded query was combined Excel files. I've also tested this on a folder with CSV files only successfully, so the file type is not an issue.

My actual file is a bit more complicated. After I combine files, I append another Excel file, and that's when I get my error that does not make sense -- it says credentials for that particular folder are invalid.

Just to be clear -- this happens with CSV files too.

I guess this kind of makes sense -- incremental refresh should only be implemented on a table based on a single source. If I combine files and then append another one, I might get duplicate data across different partitions. This issue can be closed I guess.

@Anonymous if you get the same error, and you have only one source in your query (folder), make sure your query returns the same schema even if there are no files. I had to be quite crafty to ensure there are no errors because of different schemas so I could rule it out.