Forum Discussion

Jeanxyz's avatar
Jeanxyz
Power Participant
2 years ago
Solved

pull AWS s3 data into power bi

I need to import some csv files from aws s3 into power bi. Below is the python script I try to use. I have received API url and token from another colleague, how can I complete the bucket and key var...
  • Jeanxyz's avatar
    Jeanxyz
    2 years ago

    Thanks, amitchandak . will go through the tutorials when I get some time.

     

    I talked to our AWS admin and made some change with the python script, it works now. (see script below).

    limitations of python script connector:

    1. this import mode is slow, hence I can only import a small csv file. If there is a small error with the csv file, the import query will fail. Is there a way to ignore csv reading errors?

    2. to import multiple files from the s3 bucket. I need to write a loop function in python script

    *****************************************

    import boto3

    import pandas as pd

    import io

    import os

    my_bucket_name="xx"

    my_file_path="xx.csv"

    my_key="xx"

    my_secret="xx"

     

    session=boto3.Session(aws_access_key_id=my_key,aws_secret_access_key=my_secret)

    s3Client=session.client("s3")

    f = s3Client.get_object(Bucket=my_bucket_name, Key=my_file_path)

    aws_data = pd.read_csv(io.BytesIO(f['Body'].read()), header=0, index_col=0)

    print(aws_data)