Forum Discussion
Project doubles in size when creating dimension tables
I have one large data set (300MB about 700k rows). I would like to make a series of dimension tables to reduce the size. My process has been to duplicate the original query, remove the columns I ...
First, are you confusing the query size with the data model size? You might see the table queried 3 times, but your resulting data model will be much smaller if you've replaced a non-integer dimension with integers.
Second, instead of duplicating the table, duplicate just the data source, remove all of the other columns (after duplicating the filter logic if necessary), then make distinct and add the index. Then perform your join. If your indexes are both sorted the same way, then you can use Table.Join instead of NestedJoin, and use the JoinAlgorithm.Sortmerge for the JoinAlgorithm parameter. This way the data will be streamed and not have the table brought into memory.
--Nate