Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
7 years ago
Solved

Apply query changes hangs at 1000 rows

I have worked around this issue for some time and finally broke it down to its smallest component. I have a table with 1.8m rows that I refresh once a week from a Netezza\DB2 source but do it with...
Anonymous's avatar
Anonymous
Not applicable
7 years ago

Hi Anonymous,

 

Did you apply an advanced operations in your query tables?(e.g append, combine, reference other query, invoke custom functions...)

 

If this is a case, it may caused the duplicate memory spend on calculation and will hang 'apply' operation and keep loading data until finish calculations.

 

I'd like to suggest you try to use Table.buffer/List.Buffer to loading these resource which used in advanced operation to memory to avoid duplicate memory cost

 

Regards,

Xiaoxin Sheng

Anonymous's avatar
Anonymous
Not applicable
7 years ago

In looking for proper M langauage on Table.Buffer I never found an example I could mimick.

 

However I took a more serious look at the query folding and updated my select statement rather than handling in ETL steps.

By selecting only the columns I was intending to use and note later removing them in step 2...the query went smother.

 

If anyone has a good example of using Table.Buffer specfically for merges I would be interested.

 

As well, I unpivot a fact table into another table by using the orginal fact table as a Reference Source..  I would think that be optimal as it is already and only need a few more steps for some aggregate measurements.