Forum Discussion
Can I connect to a Amazon S3 bucket using Power Query?
- 9 years ago
Hi fenixen,
As the Amazon S3 is a web service and supports the REST API. You can try to use web data source to get data. See:
Amazon S3 REST API Introduction
How to call REST APIs and parse JSON with Power BI
Another I can think of is importing data from Amazon S3 into Amazon Redshift. then in Power BI desktop, use Amazon Redshift connector get data. See: Loading Data from Amazon S3.
Best Regards,
Qiuyun Yu - 8 years ago
Hi,
ZappySys has released API drivers (XML and JSON) with AWS API support. See below blog post it explains scenario of how to access AWS S3 data in Power BI. It also explains Billing / Cost API usecase via API calls. ZappySys will rease CSV driver very soon which will support your scenario of reading CSV from S3 in Power BI but until that you can call Billing API (JSON format)
https://zappysys.com/blog/read-amazon-s3-data-power-bi-aws-json-xml-api/
Query Amazon Billing APIConfigure AWS Connection using ZappySys ODBC Driver (XML or JSON)Import Amazon Billing data in Power BI using ZappySys Driver
Hi fenixen,
As the Amazon S3 is a web service and supports the REST API. You can try to use web data source to get data. See:
Amazon S3 REST API Introduction
How to call REST APIs and parse JSON with Power BI
Another I can think of is importing data from Amazon S3 into Amazon Redshift. then in Power BI desktop, use Amazon Redshift connector get data. See: Loading Data from Amazon S3.
Best Regards,
Qiuyun Yu
I've been through this information and the process seems non-trivial, as compared to connecting to other REST API services. Has anyone actually connected PBI to an S3 bucket as a data source? I'm looking for actual experience and not theory per se. Thanks!
- spinwards8 years agoRegular Visitor
Is anyone still looking for this? I have this up and running .... I can't share my query, but I might be able to point you in the right direction.
- Srujana8 years agoFrequent Visitor
please share the steps to connect to S3 bucket from PBI desktop.
- spinwards8 years agoRegular Visitor
Sure. This is how I do it:
1. Create a new Lambda Function ... it can be empty for now
2. Set up a new API in API Gateway
3. Create a new GET method
3.1 Select Lambda Function for the integration type
3.2 Select the Use Lambda Proxy integration option
3.3 Select the region and type in the name of the lambda function you created in step 1
4. Edit your lambda function.
4.1 Using the AWS SDK, generate a url w/ pre-signed key for your file
4.2 Return a 303 redirect to the url from step 4.1Here is a sample of a lambda function in python 2.7:
bucket = 'bucket-name' key = 'path-to-file' client = boto3.client('s3') link = client.generate_presigned_url( 'get_object', {'Bucket': bucket, 'Key': key}, 7200, 'GET') return { "statusCode": 303, "headers": {'Location': link} }You can use this PowerBI query as a starting point:
let // Replace the URI with the "invoke url" of your API Gateway endpoint
// See: https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-call-api.html#how-to-call-api-console uri = "https://your-web-api.execute-api.us-east-1.amazonaws.com/stage/endpoint",
// PowerBI should understand the 303 redirect and follow it to your presigned s3 url
// Make sure to set IsRetry to true so that PowerBI gets a fresh url with a
// fresh presigned key every time
raw = Web.Contents(uri, [IsRetry=true]), // My file is a gzipped utf-8 tsv, so I need to decompress it // and tell the csv parser the delimiter and encoding binary = Binary.Decompress(raw, Compression.GZip), csv = Csv.Document(binary, [ Delimiter="#(tab)", extraValues=ExtraValues.Ignore, Encoding=65001]) // 65001 is utf8 in csvOnce you get everything working with a single file, you can parameterize your API and Lambda function to accept an s3 path so that you can pull in anything from your S3 account.