Forum Discussion
Power pivot table with more than 10 million records
10 million records is not a big model. Something else is wrong, it is just not clear what. How many columns in the table? What is the cardinality of the columns? Many columns and highly unique columns are your enemies.
If the table is defined so that I can go into detail in the product study and well from the Erp, what alternatives do I have?
- MattAllington5 years ago
Community Champion
I"m not 100% sure what you are asking. The table structure and needs of your source system are normally different to what is required for Power BI. Power Query is designed to help with that. Simply connect to the source, use Power Query to choose which columns you need to load, and only load those columns. If you just load the 4 columns you posted earlier, then I am sure it will be fine.
- Syndicate_Admin5 years ago
Administrator
Excuse me, I don't think I explained myself well.
The four columns I passed you were just one example for you to see the cardinality of the table, but as I mentioned earlier, the table has a total of 28 columns and all are necessary for the user to do a detailed analysis from power pivot.
At this point and seeing that in the table you can not remove columns because they are all necessary and therefore power pivot will continue to give me errors when downloading the 10 million records at once, what I need is an alternative solution, so I asked if it was possible in the power pivot data model , make a kind of "UNION" from two tables.
Suppose the 10 billion records are divided into 10 years or exercises.
You would create a first table in the power pivot model with records from the first 7 years. This table would be downloaded a first time and would never be updated, as these records belong to already closed exercises, so downloading them once would be sufficient and unnecessary to update them and prevent the data refresh from being long or fails.
Second, it would create a second table, with the last 3 years of records, which if they can vary, so this table would be updated from power pivot to user demand.
The part I need to know is: Is it possible from a dynamic table to make a "union" species of the two tables in order to have the data of the 10 years and therefore 10 million records?
In short, if only the stirrers of the last few years can change, why am I going to update the 10-year data?
This alternative is just an idea in my head that I don't know if it's possible to carry it out, so I threw the question. Maybe someone has had a problem that looks like or equals mine and solved it differently.
Thanks a lot.