Forum Discussion
Dataflow and Athena - Initial Refresh/Data Load is slow
Hi, we have configured a gateway with a data source to AWS Athena. We are using Dataflow to connect to the AWS Athena dsn and then connecting to Dataflow in Power BI Desktop to build visuals. Initially, we found in Navigator that the "Table is Empty" . Per Microfoft's documentation, we found that this issue will arise because data has not been loaded to the table. When a dataflow is initially created, a refresh must occur to load data. When we selected "refresh now", it took about 30 minutes to load 302k rows from Athena. This seems fairly long considering the fact that we can query this same table in Athena in mere seconds.
Has anyone else encountered this same issue? Is this an expected behavior? Thank you very much.
9 Replies
Hi, catcollector
It’s my pleasure to answer for you.
According to your description,You first connect the data source in the service with the data stream, and then connect the data source in the service in the desktop.
Is it slow to refresh data using refresh in the service or slow to load data in desktop?
You can try to directly connect and load data directly in the desktop.Maybe it will improve.
Links about:
Connecting to Amazon Athena with ODBC - Amazon Athena
If it doesn’t solve your problem, please feel free to ask me.
Best Regards
Janey Guo
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
v-janeyg-msft Hi, thank you for your response. We are connecting to the data using Dataflow (in Power BI server). However, you have to load data in the initial creation of the dataflow (refresh now option). We're finding that this "refresh" is taking 30+ minutes to load 302k rows. Unsure if this is common?
Hi, catcollector
Does the first refresh take half an hour or is it needed every time, and can the refresh succeed in the end?It shouldn't be so long,can you share information about refresh history of the dataflow?
If it doesn’t solve your problem, please feel free to ask me.
Best Regards
Janey Guo
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
v-janeyg-msft Hi, I apologize, I'm a bit confused. The refresh / initial load was successful. However, the concern is that it took 30 minutes to load 302k rows. This seems fairly long considering the fact that we can query this same table in Athena in mere seconds. Is this common? Is it expected behavior?
Hi, catcollector
You should know that query and loading are not the same.Loading involves the process of connecting to your data source and then loading the entire model data.It is related to the complexity of the data model, so the time may be longer. It is recommended that you directly try to connect to the data source in the desktop, which should shorten a part of the time.
Best Regards
Janey Guo
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
v-janeyg-msft Can you explain what you mean by directly connecting to the data source in Power BI desktop? Do you mean that we should use the ODBC connector in Desktop as opposed to building a dataflow in Service to connect to Athena using an ODBC connector? TIA!
