Forum Discussion
Impactful IndividualFetching AWS Athena data into Fabric
Hello everyone,
I have all my sql tables in AWS Athena.
Is there some way we can get all those tables into Fabric?
I am looking to mirror/shortcut the Athena data into Fabric.
Thank you,
Vishesh Jain
Hi visheshjain,
You’re right Athena doesn’t natively support continuous export on data arrival. Athena is a query service on top of data stored in S3. It doesn’t generate new files by itself unless you explicitly run a query that outputs data. You need another AWS service to call Athena on a schedule or in response to S3 events. You can also reach out to AWS support team to get a better understanding of the services in AWS and how they works with incremental data.
Coming to the latency part, you are also right that dumping a full CSV daily won’t scale well. If every refresh requires exporting the entire dataset into a single CSV/Parquet, the process will keep getting slower. The larger the file, the longer it takes to write it in Athena the transfer it to S3 and load it into Fabric and then refresh the Power BI dataset. So instead of a full export, only export incremental changes (new or updated rows) from Athena to S3 and store them in partitioned folders (e.g., by date). Now in Fabric, load just the new partitions into the Lakehouse table. This way, refresh time remains roughly constant even as total data grows.Also as lbendlin has already mentioned that there is no direct connector available in fabric to pull table from AWS athena and if you want it then you can raise the issue on Fabric Ideas Forum.
Best Regards,
Hammad.
19 Replies
- suparnababu8
Super User
Hi visheshjain
There is no direct connector availble in fabric to pull table from AWS athena as of now. Currently support connector from AWS is S3 bucket.
If you want Athena Connector, Please submit your Idea here: Fabric Ideas - Microsoft Fabric Community
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User! - lbendlin
There is a standard AWS Athena connector in Power BI. Are you looking for something else (shortcuts or mirroring)?
- visheshjain
Hi lbendlin,
Yes I am looking to mirror the tables. I have the data in S3 on which tables have been created in Athena.
Do you know if having the data in Fabric will reduce refresh times or not?
Currently the gateway takes about 45-50 mins for a 6 million row dataset.
Thank you!
Vishesh Jain
Hi visheshjain,
Thanks for reaching out to the Microsoft fabric community forum.
At the moment, Microsoft Fabric doesn’t provide a native connector to directly pull tables from AWS Athena into a Lakehouse or Warehouse. While Power BI Desktop does have an Athena connector that lets you import data for reporting, this connector isn’t currently available in Fabric Dataflows or Pipelines.
However, there’s a supported workaround using Amazon S3. You can export your Athena query results to S3 (in formats like CSV or Parquet), and then use the Amazon S3 connector in Fabric to bring that data into your Lakehouse.
If a direct Athena connector is important for your use case, as already recommend by lbendlin, try submitting an idea in the Ideas Forum so the product team can prioritize it.
I would also take a moment to thank suparnababu8 and lbendlin, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
If I misunderstand your needs or you still have problems on it, please feel free to let us know.
Best Regards,
Hammad.
Community Support TeamHi visheshjain,
As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help. If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.
And if you're still looking for guidance, feel free to give us an update, we’re here for you.
Best Regards,
Hammad.
Hi visheshjain,
Hope everything’s going smoothly on your end. We haven’t heard back from you, so I wanted to check if the issue got sorted.
Still stuck? No worries just drop us a message and we can jump back in on the issue.Best Regards,
Hammad.
- visheshjain
Hello,
I was not able to get the data from Athena directly into Fabric, however I was able to get the data from S3 into a Lakehouse and make a table using a SQL endpoint, which was stored into a Lakehouse table.
I created a report from this lakehouse table and published it to the service, but here is the issue.
I added new data to S3 and when I refresh this semantic model in the service, the new data does not show up.
If anyone could please help me with it.
Thank you,
Hi visheshjain,
Thanks for the detailed follow-up, the issue you are facing is likely due to how the data is being loaded from S3 into the Lakehouse. Fabric doesn't automatically sync new files added to S3 unless you've built a process like a Dataflow Gen2, Notebook, or Pipeline to handle that refresh regularly.
You can set up a recurring Dataflow Gen2 or Notebook in Fabric that reads from your S3 path and updates the Lakehouse table which makes sure the new S3 data is picked up before the semantic model refresh. If you're using COPY INTO or reading Parquet/CSV files directly from S3, you'll need to rerun that logic whenever new files are added.
Once the Lakehouse table is refreshed with the latest data from S3, the semantic model refresh in the Power BI Service will pick up the changes correctly.
Best Regards,
Hammad.
- visheshjain
Hello Anonymous,
Could you please help me out with how to design the solution?
I want to design the solution in such a way that after loading new data in S3, when the user refreshes the dataset in PBI service, all the data new and old is fetched from S3 and the report is updated.
Thank you,
v-sdhruvCommunity Support
Hi visheshjain ,
Since we didnt hear back, we would be closing this thread.
If you need any assistance, feel free to reach out by creating a new post.
Thank you for using Microsoft Community Forum
