Forum Discussion
Memorable MemberOauth2 gateway bugs affecting datasets not dataflows (WHY IN 2024)
Memorable MemberHi Anonymous
I found the text you referred to in April 2024:
... continued ..
"
Hour long refreshes
The on-premises data gateway now supports refreshes longer than one hour running on Dataflows. Refreshes longer than an hour can now refresh tokens mid-stream and continue to run on Dataflows.
"
Question - Do you know if there is a conceptual reason (on principle) why this team keeps introducing these mismatching features in datasets vs dataflows? It is very frustrating from a user perspective to have to remember what variety of PQ works in the context of a dataset or a dataflow. In the past, I was under the assumption that dataflows were the ones that were inferior, and were not on the leading edge when it comes to new features and innovations. (An example is "custom connectors" which work fine in datasets but are disallowed in dataflows).
Will it take a lot longer before this bug fix is introduced into datasets, in addition to dataflows? Is it going to be months or years? Almost half a year has passed, and I'm guessing they don't have it very high on their priority list. It almost seems like they are abandoning the existing PQ mechanism in datasets, and are planning to replace it with whatever mechanism they use for dataflows or dataflow gen 2.
This is a very bumpy ride, from a customer perspective. You are basically telling me I have to start re-engineering all my ETL's to run PQ from dataflows, if I anticipate that the refresh times will occasionally last longer than an hour. Can you see why customers don't want to worry about whether their PQ may run for 65 min instead of 45 min?
Memorable MemberAn update from me. The story changed again in the October release. Supposedly they fixed datasets/models to refresh their oauth tokens as well.
The docs were re-written again to talk about the post-October behavior of oauth tokens (2024):
... in my experience the GEN1 dataflows still don't refresh their oauth tokens after an hour, despite what the docs say above. Notice in the highlighted section that they identify other types of PBI items that aren't able to refresh tokens. Its not clear why the GEN1 dataflows are getting second-hand treatment in terms of the oauth refresh behavior, or why the docs seem to ignore the existence of GEN1 dataflows altogether.
Does anyone know how long it will take for Microsoft to abandon GEN1 dataflows?
- lbendlin
Super User
My money is on "when the last P SKU contracts have expired and everybody is on F SKUs"
- dbeavon3
After two months of waiting for Microsoft to provide support to Mindtree, they finally came back with the following response. It covers the mid-stream oauth-refresh bugs, and the deprecation of GEN 1 dataflows as well.
Quote:
Yes, the Gen 1 Dataflows will be deprecated but unfortunately, we don’t have an ETA for it. However, this is one of the reasons why the PG has discontinued “mid-stream token refresh” in the Dataflows Gen1.
...
midstream token refresh is completely discontinued from Dataflow Gen 1 and the product team will no longer support it. This is a product limitation when using Dataflows Gen 1.
If there was any doubt, Power BI customers should begin abandoning GEN1 dataflows for the more expensive GEN2 variety. I have found the costs to be substantially more... this is primarily because GEN2 dataflows will also charge for (1) the passage of time, and (2) the use of the self-hosted on-premise gateway. When accounting for those things it leads to far more CU consumption than before. With GEN1 dataflows we seemed to be paying only for the actual compute (vcores) that is used in our premium capacity. But with GEN2 they have found other meters to to use, in order to rapidly decrement our CU's.
