Forum Discussion
Impactful IndividualFetching AWS Athena data into Fabric
Hi visheshjain,
You’re right Athena doesn’t natively support continuous export on data arrival. Athena is a query service on top of data stored in S3. It doesn’t generate new files by itself unless you explicitly run a query that outputs data. You need another AWS service to call Athena on a schedule or in response to S3 events. You can also reach out to AWS support team to get a better understanding of the services in AWS and how they works with incremental data.
Coming to the latency part, you are also right that dumping a full CSV daily won’t scale well. If every refresh requires exporting the entire dataset into a single CSV/Parquet, the process will keep getting slower. The larger the file, the longer it takes to write it in Athena the transfer it to S3 and load it into Fabric and then refresh the Power BI dataset. So instead of a full export, only export incremental changes (new or updated rows) from Athena to S3 and store them in partitioned folders (e.g., by date). Now in Fabric, load just the new partitions into the Lakehouse table. This way, refresh time remains roughly constant even as total data grows.Also as lbendlin has already mentioned that there is no direct connector available in fabric to pull table from AWS athena and if you want it then you can raise the issue on Fabric Ideas Forum.
Best Regards,
Hammad.
Hi visheshjain,
Thanks for reaching out to the Microsoft fabric community forum.
At the moment, Microsoft Fabric doesn’t provide a native connector to directly pull tables from AWS Athena into a Lakehouse or Warehouse. While Power BI Desktop does have an Athena connector that lets you import data for reporting, this connector isn’t currently available in Fabric Dataflows or Pipelines.
However, there’s a supported workaround using Amazon S3. You can export your Athena query results to S3 (in formats like CSV or Parquet), and then use the Amazon S3 connector in Fabric to bring that data into your Lakehouse.
If a direct Athena connector is important for your use case, as already recommend by lbendlin, try submitting an idea in the Ideas Forum so the product team can prioritize it.
I would also take a moment to thank suparnababu8 and lbendlin, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
If I misunderstand your needs or you still have problems on it, please feel free to let us know.
Best Regards,
Hammad.
Community Support Team
Hi visheshjain,
As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help. If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.
And if you're still looking for guidance, feel free to give us an update, we’re here for you.
Best Regards,
Hammad.
Hi visheshjain,
Hope everything’s going smoothly on your end. We haven’t heard back from you, so I wanted to check if the issue got sorted.
Still stuck? No worries just drop us a message and we can jump back in on the issue.Best Regards,
Hammad.
- visheshjain
Hi Anonymous
Please read my previous response!
Thank you,
