Setting up data pipeline

[Anonymous]

For a dashboard that I am building on Power BI Desktop, I am currently pulling data from Excel, but I recently learned that the more sales data would be added every month and since Excel can only hold 1 Million rows, so keeping the data in Excel is not possible for a long time. I am trying to know that how should I keep the data so that it is added to the same single file each month and the dashboard stays connected to this single file and keeps updating, should I do csv or some database, any help would be appreciated.

Thanks.

[otravers]

Where is the data in your Excel file coming from? How much new data will you have each month? There are MANY ways to skin that cat, we need a little more info.

[Anonymous]

Hi otravers,

Thank you for responding, the data is coming from a third party source(I cannot specify the source), but the data is about sales, I can download the data in Excel file, and the file that I downloaded has 300K rows, but this data is for the last 3 years and we get new data just for a month, so I expect the data to be less than 300K in the future.

[otravers]

For that volume of data you don't really need a database and could do the following:

1. Ingest the historical data file in a dataflow

2. Set up a second dataflow that loads a folder where you'll save monthly xlsx or csv files

3. Load and append these two dataflows in your dataset