Forum Discussion
My DataFlow Gen 2 Fails with Unknown Evaluation Error 999999 (Request ID: fcd89914-0dd9-44ea-9b09-94
Microsoft EmployeeHi sholy29,
from a quick look to the telemetry for the Request ID you shared, I see that some of your queries with staging enabled got a problem which seems linked to a Microsoft Entra ID token: if the error persists, please raise a support ticket.
Worth mentioning that, given your dataflow is writing to destinations, I assume that you do not need to stage such queries: if that's the case, you can try to uncheck the "Enable staging" option (see picture below) for all of them and re-run the dataflow by clicking "Save & run"
Best regards,
Alessandro
Thanks Alessandro.
I do have many queries that are not writing to a destination. I need those queries because I need to merge (join) them (denormalise) with other queries to have the final ouput table that then writes to a destination. I noticed that it is those queries without a destination that has staging enabled by default.
Could this be the cause of the problem? Should I disable staging for the queries without destination?
Thank you
- Alven
Hi sholy29,
staging such queries could be useful in some cases for performance reasons (see Data in Dataflow Gen2 staging - Microsoft Fabric | Microsoft Learn and Best practices for creating a dimensional model using dataflows - Power Query | Microsoft Learn), but it really depends on the kind of transformations and how big the datasets of such queries are.
The issue you reported should not happen, and if it persists consider raising a support ticket. But, as a workaround, you can uncheck the "enable staging" option for such queries and see if
- dataflow refresh succeeds
- the running time of the dataflow refresh does not increase significantly because the dataflow does not use staging anymore
Best regards,
Alessandro
Honestly, I disenabled the staging and still did not work. I removed all of the queries and brought them again and this time it worked. Howerver, I got anothe issue, the dataflow that took two minutes to finish now take 18 minutes for the same query as before and no addtional transformation step. I am really confursed. This is not the 3rd time dataflow gen 2 ci/cd has been given me issues.
Hi Alessandro,
Thank you for your quick response, that solved the problem. Thank you very much!
Does that mean I can't use the function at all, even though it should offer advantages? Or do I have to make special settings for it? Can you help me with that?
- Alven
Hi CRadziwill ,
usage (or not) of staging should not cause any failure, and if you instead get one, it usually means there is a problem somewhere thay requires an investigation / support ticket.
In general, staging can help with performance in some cases, but it really depends on the scenario.
For example, if your query does not perform complex transformations and is simply copying data from a SQL database to a Lakehouse using an output destination, staging is not needed, and it's actually disabled automatically when setting the output destination.
Some useful information can be found in Best practices for getting the best performance with Dataflow Gen2 in Fabric Data Factory - Microsoft Fabric | Microsoft Learn.
Regards,
Alessandro
