Forum Discussion
Compute Engine issues for DataFlows since 24/25 Sep ?
Hi - has anyone else had dataflow refresh issues that use linked entities that started sometime between Sep 24 9 AM and Sep 25 9 AM - UTC+10 timezone ?
I've logged a support request in parallel to this.
2 Replies
Anonymous - yes there were refresh errors generated on all my dataflows.
What i have since learned from the excellent blog site from Matthew Roche is that i had my understanding of compute engine and dataflows wrong
https://ssbipolar.com/2020/09/16/dataflows-enhanced-compute-engine-will-it-fold/
My set of ingestion dataflows are all set to OPTIMISE and i expect they are using the compute engine as their entites are all linked to other dataflows.
It is on these transformation dataflows that refresh was failing. The workaround i found was that setting the compute engine settings to DISABLED they worked. After reading Matts article this is how things should work. As these entities in these transformation dataflows are not then linked further downstream so no need to use compute engine.
HOWEVER....they were set to OPTIMISE which means that the Service should realise that they are NOT further linked, so they shouldn't use the compute engine and thus be essentially the same as setting DISABLED.
So i still feel there is an issue here where the service is not appropriately handling and OPTIMIZED setting and setting that dataflow to DISABLED behind the scenes when it determines that its not linked to anything else downstream.
