Dataflow - Validating Queries FOREVER + Gateway issues

[Anonymous]

I been a big evangalist for usage of Dataflow within my 60K something organization. Now all I hear is complains on how slow and unreliable it has become. And I personally experience it too. 

(PS: We own a P3 and two P2 SKUs)
To begin with preview data takes ages to show up, I am literally scared to add another step to an Entity! 
After all this I click "Save" and the dread "Validating Queries" runs forever. There is noway to tell if it wil suceed or not! I usually am greated with "Mash could not be retrieved" and there goes all my hours of efforts. Why do you need to do these multi validation? When the preview is generated, arent the validations already completed?

 Secondly, non-Gateway Admins (normal user with permission to a Gateway connection) are still not able to create an entity. PBI Service keeps telling them that they need to be a Gateway Administrator! WHY? I thought this issue was resloved months ago. Why are my users still getting this message?

[Daryl-Lynch-Bzy]

@Anonymous  I agree the performance of the Validating Output Schema when saving changes to a Dataflow is too slow.   There is no correlation between Dataflow refresh and Validation upon save and close.

I can wait up to an hour for the Validation upon saving to finish.  When done, the refresh takes less than 15 secounds.

[trutz]

Hi all,

I also see these issues. Loading a small dataset and added an AutoML calculation, the validation already took like 10 minutes. Now I added a table with 16 rows to be forecasted and applied the AutoML model and the validation takes over an hour and is still running 😞

[Robertostaltari]

I went through the same path: advocated dataflows as the best solution but the reality is sad.

Validating queries takes ages even for querying a single file on a SharePoint. Why is that?

[Anonymous]

There is a best Solution? Where? Working on Dataflow day by day is turning out to be a bad exprience.

I found that DirectQuery with Dataflow is 10 times slower than doing a DirectQuery from my 6 years old SQL Server, via a Enterprise Gateway. 

Can you imagine? Power BI Service to Power BI Service read takes more time than reading from an Power BI Service to OnPremise DB! 😄 

[v-kelly-msft]

Hi @Anonymous ,

Question 1:

Which kind of data sources does the dataflow get data from? Is there any specific features made to the entity, eg: Merge/Append queries? 

Question 2:

There are a few known limitations to using Enterprise Gateways and dataflows:

• Each dataflow may use only one gateway. As such, all queries should be configured using the same gateway.
• Changing the gateway impact the entire dataflow.
• If several gateways are needed, the best practice is to build several dataflows (one for each gateway) and use the compute or entity reference capabilities to unify the data.
• Dataflows are only supported using enterprise gateways. Personal gateways will not be available for selection in the drop down lists and settings screens.

So it requires to assign an administator role of the gateway to others once you want them to add entities to the dataflow.

Here is the reference.

https://docs.microsoft.com/en-us/power-bi/service-dataflows-on-premises-gateways

Best Regards,
Kelly

Did I answer your question? Mark my post as a solution!

[Anonymous]

Hi @v-kelly-msft 

Thank you for the quick reponse! 

1: Its an ODBC connection to Hive Database. We use the HortonWorks ODBC Drivers. There are no additional steps after reading the source. Since about 1 week, I am seeing this error message a lot 

2. We are using the Enterprise Gateway. There is only one gateway which has not been changed. The error message you see has to do with permissions. I as an Gateway Admin can create Entities, however my team mates who are just Gateway Connection Users are unable to create Entities. However they are able to use the same connection to schedule their Datasets.   
This issue was fixed by Microsoft in Septemebr 2019. But we are still experiencing this error message.
https://powerbi.microsoft.com/en-us/blog/power-bi-dataflows-september-2019-feature-summary/

[cmeu]

same here - multiple types of source systems (ODBC to HIVE/Hadoop, MS SQL Server, etc.) - enterprise gateway, P1 capacity, have been using dataflows for a long time with great success. Now recently, these problems are preventing me from making several needed updates...