Incremental Refresh Issue

[VISRAI]

Hi all, I recently applied incremental refresh in Power BI with SharePoint as the data source. After making these changes, the report refreshes successfully in Power BI Desktop. However, when I publish it to the server and try to refresh there, I get an error stating that the "ABC" column does not exist. I checked the tables and found that this column appears in some, but not all tables. I'm not sure what the actual issue is.

[Poojara_D12 · Accepted solution]

Hi @VISRAI 

This error pops up because of a mismatch between the schema of your historical data partitions and the fresh data coming in during a Service refresh. When you configure incremental refresh, Power BI splits your SharePoint dataset into separate, frozen time chunks based on your parameters. If a contributor added or removed the "ABC" column in a recent SharePoint file, your local Desktop preview might handle the discrepancy quietly, but the Power BI Service engine strictly enforces schema uniformity across all historical and new partitions during a scheduled refresh.

To fix this, you need to go back into Power Query Desktop and ensure that the "ABC" column is explicitly handled so every table output has an identical structure. Before the step where you combine your SharePoint files, add a step to explicitly select or remove columns, or use a custom M code step like Table.SelectColumns(Source, {"RequiredColumn1", "RequiredColumn2"}, MissingField.UseNull) on the individual tables. This forces Power BI to insert a column of null values for "ABC" in the older tables where it doesn't exist, creating a matching schema across all files that won't crash when the Service tries to append the data into the incremental partitions.

[v-veshwara-msft]

Hi @VISRAI ,

Thanks for reaching out to Microsoft Fabric Community.

Just checking in to see if your query is resolved and if any responses were helpful.
Otherwise, feel free to reach out for further assistance.

Thank you.

[Murtaza_Ghafoor · Accepted solution]

@VISRAI 

Why this is happening:

"ABC" column exists in some tables but not all tables.

This strongly indicates a schema inconsistency issue in share point.

When you apply Incremental refresh it creates partitions in the PBI Service, Power BI demands that:

All columns must have identical column structure.

• identical data types.
• identical transformation yields.
• across all partitions/files/folders returned from the SharePoint.

If one SharePoint file/table is missing the ABC or BAC column, PBI Service refresh can fail even if it works in Power BI Desktop.

You can create exception in your power BI code to save guard this by adding following

Power Query Code

if Table.HasColumns(#"Previous Step", "ABC")

then #"Previous Step"

else Table.AddColumn(#"Previous Step", "ABC", each null)

 If this helps, ✓ Mark as Kudos | Help Others

[Poojara_D12 · Accepted solution]

Hi @VISRAI 

This error pops up because of a mismatch between the schema of your historical data partitions and the fresh data coming in during a Service refresh. When you configure incremental refresh, Power BI splits your SharePoint dataset into separate, frozen time chunks based on your parameters. If a contributor added or removed the "ABC" column in a recent SharePoint file, your local Desktop preview might handle the discrepancy quietly, but the Power BI Service engine strictly enforces schema uniformity across all historical and new partitions during a scheduled refresh.

To fix this, you need to go back into Power Query Desktop and ensure that the "ABC" column is explicitly handled so every table output has an identical structure. Before the step where you combine your SharePoint files, add a step to explicitly select or remove columns, or use a custom M code step like Table.SelectColumns(Source, {"RequiredColumn1", "RequiredColumn2"}, MissingField.UseNull) on the individual tables. This forces Power BI to insert a column of null values for "ABC" in the older tables where it doesn't exist, creating a matching schema across all files that won't crash when the Service tries to append the data into the incremental partitions.

[Olufemi7]

Hello @VISRAI,
This usually happens because incremental refresh creates partitions in the Power BI Service, and one or more SharePoint files don’t contain the ABC column.
When Service refresh runs, it hits that missing column in a partition, Desktop works because it doesn’t fully simulate partition processing.
Fix: All files must have the same schema or handle it in power query: 

if Table.HasColumns(Source, "ABC")
then Source
else Table.AddColumn(Source, "ABC", each null)

Then republish and do a full refresh to rebuild partitions
Docs:
Configure incremental refresh and real-time data for Power BI semantic models
Table.HasColumns- Power Query M 

[GilbertQ]

Hi @VISRAI 

What do I'll then have to do is to build it into your power query process to enable their column to exist even if it is not in some of your Sharepoint data. To ensure that the refresh does not fail.

[lbendlin]

Can you show a sanitized version of the Power Query code?