Forum Discussion
Incremental refresh
- Anonymous1 year ago
Hi,miguel ,thanks for your concern about this issue.
Your answer is excellent!
And I would like to share some additional solutions below.
Hello,ImSh7m .I am glad to help you.
In fact, after reading the article provided by miguel, I ran the following test and determined that the CSV file can do what the article says, which is to refresh only the data that is in the refresh interval (incremental refreshing)
This is my test.(StartDate as datetime, EndDate as datetime) => let // Convert StartDate and EndDate to numbers and calculate the range of years start = Number.From(Date.From(StartDate)), end = Number.From(Date.From(EndDate)) - 1, Years = List.Distinct(List.Transform({start..end}, each Text.From(Date.Year(Date.From(_))))), // Get the list of files from the SharePoint site Source = SharePoint.Files("https://xxx123456789.sharepoint.com/sites/123SiteName", [ApiVersion = 15]), // Filter the file list to keep only files with folder paths containing the years #"Filtered Rows" = Table.SelectRows(Source, each List.Contains(List.Transform(Years, (x) => Text.Contains([Folder Path], x)), true)), // Add a custom column to load the file content as CSV #"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each Csv.Document([Content], [Delimiter=" ", Columns=2, Encoding=65001, QuoteStyle=QuoteStyle.None])), // Expand the custom column to display all columns #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column1", "Column2"}), // Combine all columns into one column using Tab as the delimiter #"Combined Columns" = Table.AddColumn(#"Expanded Custom", "Combined", each Text.Combine({Text.From([Column1]), Text.From([Column2])}, "#(tab)")), // Select and rename the required columns #"Selected Columns" = Table.SelectColumns(#"Combined Columns", {"Combined"}), // Define an empty table schema Schema = #table(type table [Combined = text], {}) in // Try to return the combined data, if it fails return the empty table schema try #"Selected Columns" otherwise SchemaAfter executing the function, process the imported csv data
Original data processing completed (here you need to process the original data according to your actual situation)
This incremental refresh connects csv files in multiple folders in sharepoint
The test data:Setting Incremental Refresh Query Parameters
In my testing, the system warns:
Cannot confirm whether M queries can be collapsed or not, and suggests not to use non-collapsible queries for incremental refresh.
Whether the data source supports query collapsing or not affects the efficiency of the whole refresh execution (the time required for the refresh), and has no impact on the final result of the incremental refresh execution (if the refresh is successful, only part of the data is refreshed)
This is also covered in the issue I share below.
Issues Link:Solved: Cannot setup incremental refresh parameters - Microsoft Fabric Community
Fixing Query Folding with SQL Server in Power BI for Incremental Refresh – bits2BI
After successfully uploading to the service, configure the data credentials & gateway connection, then refresh the semantic model, this csv file data, will be in the historical interval, but not in the refresh interval of the data for modification can not see whether to perform an incremental refresh, I modified the 2024 dataReplacement of csv file in Sharepoint online (to simulate a change in the data source data)
before refresh:
The refresh result:
I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.
Best Regards,
Carson Jian,
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi,miguel ,thanks for your concern about this issue.
Your answer is excellent!
And I would like to share some additional solutions below.
Hello,ImSh7m .I am glad to help you.
In fact, after reading the article provided by miguel, I ran the following test and determined that the CSV file can do what the article says, which is to refresh only the data that is in the refresh interval (incremental refreshing)
This is my test.
(StartDate as datetime, EndDate as datetime) =>
let
// Convert StartDate and EndDate to numbers and calculate the range of years
start = Number.From(Date.From(StartDate)),
end = Number.From(Date.From(EndDate)) - 1,
Years = List.Distinct(List.Transform({start..end}, each Text.From(Date.Year(Date.From(_))))),
// Get the list of files from the SharePoint site
Source = SharePoint.Files("https://xxx123456789.sharepoint.com/sites/123SiteName", [ApiVersion = 15]),
// Filter the file list to keep only files with folder paths containing the years
#"Filtered Rows" = Table.SelectRows(Source, each List.Contains(List.Transform(Years, (x) => Text.Contains([Folder Path], x)), true)),
// Add a custom column to load the file content as CSV
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each Csv.Document([Content], [Delimiter=" ", Columns=2, Encoding=65001, QuoteStyle=QuoteStyle.None])),
// Expand the custom column to display all columns
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column1", "Column2"}),
// Combine all columns into one column using Tab as the delimiter
#"Combined Columns" = Table.AddColumn(#"Expanded Custom", "Combined", each Text.Combine({Text.From([Column1]), Text.From([Column2])}, "#(tab)")),
// Select and rename the required columns
#"Selected Columns" = Table.SelectColumns(#"Combined Columns", {"Combined"}),
// Define an empty table schema
Schema = #table(type table [Combined = text], {})
in
// Try to return the combined data, if it fails return the empty table schema
try #"Selected Columns" otherwise Schema
After executing the function, process the imported csv data
Original data processing completed (here you need to process the original data according to your actual situation)
This incremental refresh connects csv files in multiple folders in sharepoint
The test data:
Setting Incremental Refresh Query Parameters
In my testing, the system warns:
Cannot confirm whether M queries can be collapsed or not, and suggests not to use non-collapsible queries for incremental refresh.
Whether the data source supports query collapsing or not affects the efficiency of the whole refresh execution (the time required for the refresh), and has no impact on the final result of the incremental refresh execution (if the refresh is successful, only part of the data is refreshed)
This is also covered in the issue I share below.
Issues Link:
Solved: Cannot setup incremental refresh parameters - Microsoft Fabric Community
Fixing Query Folding with SQL Server in Power BI for Incremental Refresh – bits2BI
After successfully uploading to the service, configure the data credentials & gateway connection, then refresh the semantic model, this csv file data, will be in the historical interval, but not in the refresh interval of the data for modification can not see whether to perform an incremental refresh, I modified the 2024 data
Replacement of csv file in Sharepoint online (to simulate a change in the data source data)
before refresh:
The refresh result:
I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.
Best Regards,
Carson Jian,
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.