Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Help with processing larger dataset in desktop

Hi all,

 

I am currently trying to add a new table to our 'shared dataset' - its a larger table than normal (approx 50 million rows).

 

I have no issues processing this in the service (incremental refresh) but i am having issues trying to process this in desktop, it takes about an hour to run and will often fail citing memory issues. Yet the query only takes about 1 min to run in snowflake.

 

Is there something I am doing wrong - is there any way i can amend/add to the model while it is in the service? I looked at connecting via the XMLA endpoint but could not see anything about that being possible.

 

I feel like 50 million rows is not very big in the grand scheme of things and something that i should be able to handle easily.

Need Help

2 Replies

  • Samarth_18's avatar
    Samarth_18
    Icon for Community Champion rankCommunity Champion
    3 years ago

    Hi Anonymous 

     

    Did you try with removing uneccesary/unused column and also restrict the data with the required filters?

     

    Thanks,

    Samarth

    • Anonymous's avatar
      Anonymous
      Not applicable
      3 years ago

      Hi Samarth,

       

      Yes the table is modelled very specifically for one this use case so is using the minimum amount of columns possible. There is 14 columns being selected currently.

       

      The data is filtered in the sql queries beforehand.

       

      With incremental refresh it takes about 13 minutes to refresh in the service which is fine.