Refreshing datasets using the API results in random failures

We're runnning an Azure Synapse pipeline (daily) from which we loop over our Power BI datasets (currently 9). For each dataset we execute the refresh API using a POST call. After the POST call we have a check refresh step, waiting for the refresh to complete, either succesful or in failure.

What we observe is that the datasets often fail but it appears (to me) a bit random. The number or combination of which datasets fails look a bit random (one day just two fail, another day four, the next day none fail, the day after five fail...). After monitoring for a number of days there is no pattern emerging (yet).

The error messages also look a bit off since after a manual rerun in Power BI it seems to work fine. Error messages often hint towards corrupt datalines but we did not find actual issues in the data itself. To me it looks like the data is not fetched correctly, most likely incomplete.
Example of such message (removed the value reference for privacy reasons) below.

My questions regarding this issue:

- While looping over the datasets to post the refresh command, might there be a timing issue i.e. should we implement a delay between executing the refresh for each dataset?

- Is there a data limit e.g. cache or in memory what could cause this issue?

- Other possible cause...?

Error sample:
{"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"The key didn't match any rows in the table."}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"-2147467259"}},{"code":"Microsoft.Data.Mashup.ValueError.Key","detail":{"type":1,"value":"