Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago

Setting up data pipeline

For a dashboard that I am building on Power BI Desktop, I am currently pulling data from Excel, but I recently learned that the more sales data would be added every month and since Excel can only hold 1 Million rows, so keeping the data in Excel is not possible for a long time. I am trying to know that how should I keep the data so that it is added to the same single file each month and the dashboard stays connected to this single file and keeps updating, should I do csv or some database, any help would be appreciated.

Thanks.

Need Help

3 Replies

  • otravers's avatar
    otravers
    Community Champion
    4 years ago

    Where is the data in your Excel file coming from? How much new data will you have each month? There are MANY ways to skin that cat, we need a little more info.

    • Anonymous's avatar
      Anonymous
      Not applicable
      4 years ago

      Hi otravers,

      Thank you for responding, the data is coming from a third party source(I cannot specify the source), but the data is about sales, I can download the data in Excel file, and the file that I downloaded has 300K rows, but this data is for the last 3 years and we get new data just for a month, so I expect the data to be less than 300K in the future.

      • otravers's avatar
        otravers
        Community Champion
        4 years ago

        For that volume of data you don't really need a database and could do the following:

         

        1. Ingest the historical data file in a dataflow

        2. Set up a second dataflow that loads a folder where you'll save monthly xlsx or csv files

        3. Load and append these two dataflows in your dataset