Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hi,
I'm receiving daily excel files with a table in which a field shows the extraction date of the data.
I want to keep track of all data and store all historical values into a data model to make some basic performance comparisons.
Excel file 1 would look like this
Test Value ExtractionDate
A | 3 | 13/02/2023 |
B | 8 | 13/02/2023 |
C | 6 | 13/02/2023 |
D | 9 | 13/02/2023 |
Excel file 2 would look like this (Values & extraction date will change the following day)
Test Value ExtractionDate
A | 3 | 14/02/2023 |
B | 9 | 14/02/2023 |
C | 10 | 14/02/2023 |
D | 5 | 14/02/2023 |
I want to write a basic script to append Excel file 1 & 2 and 3 and 4 and...... N (each time a new file is posted on the Shared folder into a table so that I can have all data into one sole table and that the import process would be fluid.
Is there a simple solution to do this? Did try several methods including Table.InsertRows but I could not figure that out.
My script is (didn't work)
let
// Import Excel data from the source table
Source = Excel.Workbook(File.Contents("MyFile.xlsx"), null, true),
Table1 = Source{[Item="Table1",Kind="Table"]}[Data],
// Determine the maximum extraction date in the destination table
MaxExtractionDate = List.Max(Base[ExtractionDate]),
// Filter the source table based on the extraction date
FilteredTable1 = Table.SelectRows(Table1, each [ExtractionDate] > MaxExtractionDate),
// Append the filtered rows to the destination table
Import = Table.InsertRows(Base, 1, FilteredTable1)
in
Import
Can you please give me a hand?
Hi @debrauxjg - the question I have - how is Excel erasing and replacing the data each day? Ideally, Power BI/Power Query would connect to the original source rather than the Excel.
You could consider changing this daily process to take of copy the file/data before erasing and replacing so you have the daily backup for the history to import into Power Query while maintaining the current file process. Power Automate can help with this scenario - copy the excel file then start the Power BI refresh.
Note - Power BI incremental load features (in the dataset or dataflow) could also help to read the current file and append to the previous loads. While this is an option, I would hesitate to use because the Incremental Load will not be able access the history to reload the previous partitions should it break. I have tried something like this in the past, but it breaks. A good backup strategy is hard to implement.
Hi @debrauxjg - there are many YouTube videoes with example. Here is one Combining Multiple Files from a folder using Power Query in Excel or Power BI - YouTube
Many thanks @Daryl-Lynch-Bzy but it is slightly more complex than that. There is only one excel file that is fed constantly with daily data and then erased the following day and replaced by today's data and so on. If I retrieve data from this file I'd see day 1 data. The following day day 2 data. And so non. But how to consolidate day 1 & 2 in the same table even if the file is changing every day? I was hoping to do that with a PowerQuery script.
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.