Forum Discussion
100 GB data on AWS Workspace
- 1 year ago
Pre-aggregate means to summarize the data using GROUP BY instead of loading all rows. For example instead of loading by transaction id or transaction lines, group the rows by a certain column.
- 1 year ago
Ohh yess, that is something I could do. Instead of applying the measures after importing shall apply GROUP BY before importing. Thanks a lot!!
Well I have 5 columns and I need to display those 5 columns with six months data to the user. Currently I am working on 2 months data and its 150M rows. So once I start with six months data it will be easily above 300M. Also what do you mean by pre aggregation?
Also another issue is am working on Amazon workspace so I am not able to save such huge data after loading it because of memory constraints in AWS.
Pre-aggregate means to summarize the data using GROUP BY instead of loading all rows. For example instead of loading by transaction id or transaction lines, group the rows by a certain column.
- SaiK1 year ago
Helper I
Ohh yess, that is something I could do. Instead of applying the measures after importing shall apply GROUP BY before importing. Thanks a lot!!
- SaiK1 year ago
Helper I
Hello,
An update on the same:
I tried various aggregations and simplifications to my query, however have been getting this error: MySQL: Fatal error encountered during command execution.
I have tried indexing the columns, using window functions instead of group by ,etc. But only the basic queries like SELECT * FROM table works. The rest of the queries with aggregation or partitioning ,etc fails. - danextian1 year ago
Super User
Did you create a view in the database?
- SaiK1 year ago
Helper I
No. Should I create one? How should I go about this issue? Any bit of complexity I add to my code and it gives me the error
- danextian1 year ago
Super User
This is a sample script:
CREATE VIEW ProductCategoryView AS SELECT p.ProductID, p.ProductName, p.CategoryID, c.CategoryName FROM Product p INNER JOIN Category c ON p.CategoryID = c.CategoryID;Note: Different databases will have different syntaxes so this
This wil create ProductCategoryView view and instead of connecting to the tables, you connect to this instead in power query.
- SaiK1 year ago
Helper I
I did try creating the view as you said, even that failed and gave me the same error. I have also changed the query timeout settings to limitless(0).
- SaiK1 year ago
Helper I
Hi Felix,
After having a discussion with my team, turns out we will have to build a ETL pipeline to extract the data, peform some transformations on it prolly on DBT and then load the data to Power BI.