Forum Discussion
Impactful IndividualFetching AWS Athena data into Fabric
Hi visheshjain,
You’re right Athena doesn’t natively support continuous export on data arrival. Athena is a query service on top of data stored in S3. It doesn’t generate new files by itself unless you explicitly run a query that outputs data. You need another AWS service to call Athena on a schedule or in response to S3 events. You can also reach out to AWS support team to get a better understanding of the services in AWS and how they works with incremental data.
Coming to the latency part, you are also right that dumping a full CSV daily won’t scale well. If every refresh requires exporting the entire dataset into a single CSV/Parquet, the process will keep getting slower. The larger the file, the longer it takes to write it in Athena the transfer it to S3 and load it into Fabric and then refresh the Power BI dataset. So instead of a full export, only export incremental changes (new or updated rows) from Athena to S3 and store them in partitioned folders (e.g., by date). Now in Fabric, load just the new partitions into the Lakehouse table. This way, refresh time remains roughly constant even as total data grows.Also as lbendlin has already mentioned that there is no direct connector available in fabric to pull table from AWS athena and if you want it then you can raise the issue on Fabric Ideas Forum.
Best Regards,
Hammad.
Hi visheshjain,
You can try to precompute in Athena then export joined eesult to S3 and then load to Fabric Lakehouse Table. This way you you offload heavy joins to Athena where they run efficiently and lands a single clean dataset into Fabric, minimizing Power Query transformation needs. You also get a proper Lakehouse table that supports incremental refresh and partitions while avoiding the need to run joins at runtime in Power BI and keep performance manageable.
If I misunderstand your needs or you still have problems on it, please feel free to let us know.
Best Regards,
Hammad.
Impactful IndividualHi Anonymous,
It seems like a feasible solution but I am not sure if we can automate exporting a CSV from Athena to S3 every time new data comes in.
Also, even if we figure that out, latency will be a question, as exporting data, which is growing every day, to CSV will keep on increasing time and the user will have to wait for some time before s/he can hit the refresh button on the service.
Thank you,
Hi visheshjain,
You’re right Athena doesn’t natively support continuous export on data arrival. Athena is a query service on top of data stored in S3. It doesn’t generate new files by itself unless you explicitly run a query that outputs data. You need another AWS service to call Athena on a schedule or in response to S3 events. You can also reach out to AWS support team to get a better understanding of the services in AWS and how they works with incremental data.
Coming to the latency part, you are also right that dumping a full CSV daily won’t scale well. If every refresh requires exporting the entire dataset into a single CSV/Parquet, the process will keep getting slower. The larger the file, the longer it takes to write it in Athena the transfer it to S3 and load it into Fabric and then refresh the Power BI dataset. So instead of a full export, only export incremental changes (new or updated rows) from Athena to S3 and store them in partitioned folders (e.g., by date). Now in Fabric, load just the new partitions into the Lakehouse table. This way, refresh time remains roughly constant even as total data grows.Also as lbendlin has already mentioned that there is no direct connector available in fabric to pull table from AWS athena and if you want it then you can raise the issue on Fabric Ideas Forum.
Best Regards,
Hammad.
Hi visheshjain,
We noticed there hasn’t been any recent activity on this thread. If you still need support, just drop a reply here and we’ll pick it up from where we left off.Best Regards,
Hammad.