Forum Discussion

manoj_0911's avatar
manoj_0911
Kudo Commander
1 month ago
Solved

Power BI Desktop - rsQueryMemoryLimitExceeded while importing from Amazon Athena (50M rows, 18M rows

Power BI Desktop - rsQueryMemoryLimitExceeded while importing from Amazon Athena (50M rows, 18M rows/day) Question Hi everyone, I'm importing data from an Amazon Athena table into Power BI Deskt...
  • Azadsingh's avatar
    1 month ago

    I’ve seen similar behavior with large Athena imports, and in your case the error looks more like a Power BI limitation than an Athena one.

    The rsQueryMemoryLimitExceeded message is typically coming from the Power BI Desktop side (the local Analysis Services / VertiPaq engine used during refresh). Athena may execute the SQL fine, but Power BI still needs to pull, materialize, and compress the dataset into memory, and that’s where it’s hitting the ~1 GB per-query limit.

    Regarding increasing the memory limit in Desktop — as far as I know, there’s no straightforward supported setting to raise that specific per-query cap. More RAM on the machine can help overall performance, but it usually doesn’t bypass this limit.

    For incremental refresh, since your source only has a Unix timestamp, I’d recommend filtering directly on the raw BIGINT column instead of doing from_unixtime(eventtime/1000) in the WHERE clause. Wrapping the column in a function often hurts pushdown/query performance. Converting RangeStart / RangeEnd into Unix epoch values and comparing directly against eventtime is usually much more efficient.

    About Value.NativeQuery(): it’s useful when you need custom SQL (especially with Athena), but the downside is that it can interfere with query folding, which matters for incremental refresh. If incremental refresh is the goal, I’d test carefully whether folding is still happening. If not, that may become a bottleneck.

    Also, even after reducing to 4 columns, the row count is still huge. If you’re pulling ~2 days of data and ingesting ~18M rows/day, that’s potentially ~36M rows in one refresh. Even with fewer columns, fields like conversationid or attributevalue can have high cardinality and compress poorly.

    If I were designing this pipeline, I’d look at:

    • Partitioning the Athena table by date (if not already)

    • Using incremental refresh

    • Pre-aggregating or reducing data in Athena (views/CTAS)

    • Avoiding large full imports into Desktop whenever possible

    At 50M+ rows with 18M new rows daily, I’d strongly lean toward a partitioned + incremental architecture rather than relying on full import refreshes in Desktop.

     

    Helpful? Give a Kudos 👍
    Solved? Mark as Solution ✔️
    — Azad Singh Thakur | Power BI Developer