Forum Discussion
Scheduled Refresh for Data imported from Amazon S3
Hey asfdwd ,
at the current moment, there is no other way than to use Python to get S3 data when using Power BI, this might change in the future when Power BI Dataflows Gen 2 become available (a feature of Microsoft Fabric).
Using Python has its own downsides as it requires a gateway in personal mode, this gateway machine also needs a Python installation. There are downsides when using a gateway in Personal mode: https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-personal-mode?WT.mc_id=DP-MVP-5003068
My recommendation, use Azure data Factory to get the S3 data (https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-s3-compatible-storage?tabs=data-factory), when this is done you can trigger a dataset refresh from a Azure Data Factory pipeline (https://community.fabric.microsoft.com/t5/Service/Trigger-the-dataset-refresh-after-Azure-ETL-process-completes/m-p/1829999). I have to admit that this is not trivial in comparison to the Power BI Desktop solution you have, but this is most stable solution I can currently think of.Everything S3 related will become more simple with Microsoft Fabric, but Fabric is in Preview at the current moment and comes with extra costs (as well as the Azure Data Factory approach).
Hopefully, this provides some new ideas, helping to tackle your challenge.
Regards,
Tom
Super UserHey asfdwd ,
at the current moment, there is no other way than to use Python to get S3 data when using Power BI, this might change in the future when Power BI Dataflows Gen 2 become available (a feature of Microsoft Fabric).
Using Python has its own downsides as it requires a gateway in personal mode, this gateway machine also needs a Python installation. There are downsides when using a gateway in Personal mode: https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-personal-mode?WT.mc_id=DP-MVP-5003068
My recommendation, use Azure data Factory to get the S3 data (https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-s3-compatible-storage?tabs=data-factory), when this is done you can trigger a dataset refresh from a Azure Data Factory pipeline (https://community.fabric.microsoft.com/t5/Service/Trigger-the-dataset-refresh-after-Azure-ETL-process-completes/m-p/1829999). I have to admit that this is not trivial in comparison to the Power BI Desktop solution you have, but this is most stable solution I can currently think of.
Everything S3 related will become more simple with Microsoft Fabric, but Fabric is in Preview at the current moment and comes with extra costs (as well as the Azure Data Factory approach).
Hopefully, this provides some new ideas, helping to tackle your challenge.
Regards,
Tom