Creating new Table and append

[EHell]

Hi,

I have a table loading into PowerQuery and is refreshed daily, overwriting the previous days data.

We would like to start storing prev data to create a timeline - ideally we would want to append data during every daily refresh, while adding on the last refresh date (current data doesn't contain latest date). 

However there are 1000s of records so after awhile the table would get too big. 

Would it be possible to group the columns we want to keep, summarize the numeric column,  add today's date and then load it in to a new table and append to that daily? 

New table would be like:
Select

Item,

Status,

Count of Records,

Today's Date (added in Power Query?)

Group By

Item,

Status,

Today's Date

>>> Next refresh appends latest data to table, refreshed daily <<<

Bit hard to explain, hope this make sense. Thank you in advance

[rohit1991]

Hi @EHell

Power BI Desktop/Power Query doesn’t keep history each refresh rebuilds the table. If you want a daily snapshot (grouped + today’s date), you have to write it outside the PBIX and then read it back.

Two workable paths:

• Dataflow Gen2 >> Lakehouse/Warehouse (Append): group by Item/Status, add SnapshotDate, set Update method = Append, schedule daily, and point your PBIX at that table.

• SQL / OneLake / SharePoint + a scheduled job: do the same grouping, then INSERT/append today’s rows each day; the PBIX reads that table/file.

[BA_Pete]

Hi @EHell ,

Power Query is not a data warehouse and is not designed to retain data beyond its immediate use.

The PQ refresh process works like this:

Wipe all data > Import available data from source > Perform transformations > Push to Data Model.

As such, there's no function to create and retain history/archives of changing source data.

The closest native function (within the Power BI universe) would be Incremental Refresh, which comes with its own significant caveats:

https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview 

For your scenario, I would recommend creating an incrementally refreshed Dataflow that performs the grouping at each daily refresh. This will then be available to any reports you want to use this summarised data for in future.

Beyond this, you will need to arrange some kind of data warehousing or storage capability at your end to keep the history that you can then query with PQ.

Pete

[v-kpoloju-msft · Accepted solution]

Hi @EHell,

Thank you for reaching out to the Microsoft fabric community forum. I understand you need a solution to store historical data and manage table size over time while adding new daily data. Here’s a step-by-step approach in Power Query to meet your requirements:

Create a New Table for Historical Data: Create a new query in Power Query to store the appended data. This new table will allow you to retain the previous day's data and append the latest daily data without overwriting anything.

Group and Summarize Data: Use the Group By operation to group the data by your selected columns (e.g., Item and Status). Aggregate the numeric column (e.g., Count of Records) using an appropriate function (such as Sum or Count).

Add Today's Date: You can add a column with today's date using the function DateTime.LocalNow(). This will ensure each entry is associated with the date when the refresh occurs. Append the Data: After processing the data, append the transformed data into your historical table. This will ensure that new data is continuously added to the table without overwriting previous data.

Limit Table Size (Optional): To prevent the table from growing too large, you can filter out older records (e.g., keep only the last 30 days of data). You can use Power Query’s filtering options based on the current date.

I recommend reviewing the following Microsoft documentation for more information:
How to GROUP BY or summarize rows - Power Query | Microsoft Learn
Add a custom column - Power Query | Microsoft Learn
Append queries - Power Query | Microsoft Learn

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you for using Microsoft Community Forum.

[v-kpoloju-msft]

Hi @EHell,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @EHell,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

[v-kpoloju-msft]

Hi @EHell,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.