Forum Discussion
Can Power Query handle this much data?
Jkaelinwhat are your data types in your facts? i really dont see an issue in combining them if they are most numerical, why doing you just try and see what happens? essentially its mostly up to your machine. and obviously because this is in memory technology you want to make sure you standarize your data as much as possible , the less unique values you have the better in order to get the most compression. Start only with what you need rather than everything.
Jkaelin one dimension and 8 facts? sounds like the issue might be with the modelling, although i might be wrong.
it really depends on a variety of things
1) your hardware (you seem to have sufficient hardware)
2) the modelling of your data. what is in your facts from a data type perspective? How wide are they? do you need all the columns? 300000 is not a lot in my opinion, sounds like the design is the issue here
are you doing this in power bi or power pivot?
What is in your facts?
I am pulling in various stock characteristic items. P/E, P/S, Total Return, etc. For various index's across time (1990-Current).
How wide are they?
They are about 10 columns wide for each fact table.
Do I need all the columns?
Not at a given time, probably 20-30 columns at any given time.
Doing the work in Power Pivot using various DAX percentile, productx, measures that have to iterate over the data set to find the top third, middle third, etc, & calculate stock performance.
How could I improve the data modeling?
The data feed is limiting how many columns I can pull out, so I've had to harvest them seperatly into various fact tables. Which stinks.
Any thoughts would be very appreciated because I've been banging my head on a wall trying to optimize the data performance.
Jkaelinwhat are your data types in your facts? i really dont see an issue in combining them if they are most numerical, why doing you just try and see what happens? essentially its mostly up to your machine. and obviously because this is in memory technology you want to make sure you standarize your data as much as possible , the less unique values you have the better in order to get the most compression. Start only with what you need rather than everything.
Data types are all whole numbers & decimals.
Is flattening okay? Sometimes I have to create a financial ratio such as Sales / Assets. Sales will be in fTable 1 and Assets in fTable 2. I must have this data already combined into ratio for me to calculate statistics on it, such as Top third, Middle third, etc.
I really appreciate your help & thank you!
