Compute engine memory (%) (Preview) - Dataflow workload setting

[PaulKn]

I'm struggling to find any documentation on  what the Compute engine memory (%) (Preview) setting in the Dataflow workload controls and what the positives and negatives are to increasing it.

I have started experiencing memory errors on one of my dataflows that is merging and grouping large linked entities from the same workspace e.g.:

Error: Data Source Error : Microsoft SQL: There is insufficient memory available in the buffer pool

Error: Data Source Error : Microsoft SQL: Error: 802 Severity: 17 State: 2. (Params:). The error is printed in terse mode because there was error during formatting. Tracing ETW notifications etc are skipped.

As the dataflow is working only with linked entities that have already been successfully ingested these error messages must relate to the SQL engine behind the enhanced compute engine rather than any SQL source of my own. The Premium Capacity metrics show that the dataflow memory limit is nowhere near being hit.

I assume the setting will allow more memory to be used by the enhanced compute engine but I don't know if there are potential drawbacks to this (the default is 30%) that could cause problems elsewhere. Has anybody any more knowledge about this?

[v-deddai1-msft]

Hi @PaulKn ,

Because this is preview's feature, it hasn't been specifically introduced by Microsoft. I suggest you refer to  https://docs.microsoft.com/en-us/power-bi/admin/service-admin-premium-workloads#default-memory-setti...

If you have Pro account I suggest you  try to open a support ticket. If you have a Pro account it is free. Go to https://support.powerbi.com. Scroll down and click "CREATE SUPPORT TICKET".

Best Regards,

Dedmon Dai

[PaulKn]

@v-deddai1-msft Yes, I'm aware it's still in preview but the setting still needs to be explained (otherwise what is the point in exposing it). There is plenty of other information about the enhanced compute engine published despite it being in preview. The page you linked to explained all the other dataflow workload settings except this one.

I have submitted a support ticket, but unfortunately that has been tough going so far as they have completely misunderstood the issue and asked me to do irrelevant things, but I will persist.