Forum Discussion

dinoscool3's avatar
dinoscool3
Helper II
4 years ago
Solved

Out of Memory Error - Managing multiple datasets

So a bit of a weird problem herre. I have a large number of queries that I'm importing into power BI. Currently I have about 600 queries so far that I've merge into 12 tables. I disable load on all the other queries and just use the 12 tables. My final number of queries will be about 3000-4000. These queries are imports from a forum software called Discourse, I use a Blank Query that links with SQL data stored there.

 

I've been running into problems with memory, Power BI was running slow but now I can barely open the file and any time I try to refresh I get an Out of Memory error and it crashes. It seems to be overusing both memory and CPU. I'm currently using a Windows 365 with 4 core CPU and 16gb memory but I'm planning on upgrading to the 8 core and 32gb memory asap. But is there something else I can do to prevent these crashes or use less memory and CPU?

 

Here's an example of the query I use if that's helpful:

 

= (queryID) => let
        resultCount = 1000,
        otherNameForPage = 0,

        GetPage = (otherNameForPage) =>
            let
                content1 = "params={""page"":""" & Number.ToText(otherNameForPage) & """}",
                RawData = Json.Document(Web.Contents(
                    "https://test.com/admin/plugins/explorer/queries",
                    [RelativePath=Number.ToText(queryID) & "/run",
                        Query=
                        [
                            params="{""page"":""" & Number.ToText(otherNameForPage) & """}"
                        ],

                        Headers = [ 
                        #"api-username"="XXX",
                        #"api-key"="XXX",
                        #"Content-Type" = "application/x-www-form-urlencoded"],
                        Content = Text.ToBinary(content1)
                    ]
                    
                ) ),
                resultCount = RawData[result_count]
            in
                if RawData[result_count] = 0 then null else RawData,
            Pages = List.Generate(
                () => [i = 0, RawData = GetPage(i)],
                each [RawData] <> null,
                each [i=[i]+1, RawData = GetPage(i)],
                each Table.Combine(let raw = [RawData] in List.Transform(raw[rows], each Table.FromRows({_}, raw[columns])))),
            Output = Table.Combine(Pages)
in
    Output

 

  Basically it allows me to import SQL tables that are written in Discourse.

5 Replies