Forum Discussion
Helper I100 GB data on AWS Workspace
Pre-aggregate means to summarize the data using GROUP BY instead of loading all rows. For example instead of loading by transaction id or transaction lines, group the rows by a certain column.
Ohh yess, that is something I could do. Instead of applying the measures after importing shall apply GROUP BY before importing. Thanks a lot!!
Super UserWhat is the number of rows you are working with?
Helper IMore than 213M rows.
- MFelix
Hi SaiK,
With this level of granularity, the import mode will bring issues like the size and also the refresh.
You need to do use of direct query or summarre your data.
Depending on the final result you may want to take advantage of aggregation tables, dynamic filtering or reducing your data.
- SaiK
The thing is as of now I am working with 2months of data, which is almost 150M records. So 6 months data will be really huge and I did try Direct Query earlier, but the performance was really bad. Do you think making use of stored procedures and indexes during the time of import would help?
- danextian
Do you really need those 150M records? Do you need all the columns? Can't they be pre-aggregated?