Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago
Solved

Create multiple Tables from Python

Hi,

 

Using Python I'm connecting to Google Cloud Storage, I want to be able to loop through the various files in a bucket, create pandas dataframes for each one and then use the contents in Power Query.

 

I can get it all working in Pycharm with the below...

 

 

 

for file in blobs:
    file_path = "gs://{}/{}".format(file.bucket.name, file.name)
    with fs.open(file_path) as f:
        df = pd.DataFrame
        df.name = file.name
        df = pd.read_csv(f)
        print(df.name)
        print(df)

 

 

 

it loops through and prints the contents and name of each of the 3 files in the bucket, but in Power Query I only pull through the last iteration with the name df. 

 

 

I've tried various angles using dictionaries and lists but can't figure out how to end up with three separate tables.  I've seen examples concatenating various files but I want to keep these all separate.  I want to be able to loop through the files in the bucket because there will eventually be hundreds, so don't want to hard code each one.

 

Any ideas?  Is it possible?

 

Cheers

 

Mark

 

  • Anonymous's avatar
    Anonymous
    4 years ago

    Hi,

     

    I'm not sure is this thread still relevant. I had the same issue and I solved this by associating each data frame to a unique global variable:

     

    for file in blobs:
        file_path = "...."
        with fs.open(file_path) as f:
            globals()[f"{file.name}"] = pd.read_csv(f)
            print(globals()[f"{file.name}"])

     

12 Replies

Replies have been turned off for this discussion
  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi,

     

    I'm not sure is this thread still relevant. I had the same issue and I solved this by associating each data frame to a unique global variable:

     

    for file in blobs:
        file_path = "...."
        with fs.open(file_path) as f:
            globals()[f"{file.name}"] = pd.read_csv(f)
            print(globals()[f"{file.name}"])

     

    • Anonymous's avatar
      Anonymous
      Not applicable

      Thank you mate, I've moved company so not working on that solution anymore but this might prove handy in the future!

  • v-kelly-msft's avatar
    v-kelly-msft
    Community Support

    Hi Anonymous ,

     

    How about filter table name to seperate them one by one after you have loaded all to Power bi?

     

    Best Regards,
    Kelly

    Did I answer your question? Mark my reply as a solution!

    • Anonymous's avatar
      Anonymous
      Not applicable

      Not sure what you mean...it's the loading them to Power BI part that I have a problem with...

       

      I can only see one table in Power Query, I want to have a list of all the tables representing the files that are in the GCS Bucket...

      • v-kelly-msft's avatar
        v-kelly-msft
        Community Support

        Hi Anonymous ,

         

        Sorry for the misunderstanding.

        Are the python statements full?Have you defined "blobs" first?

         

        Best Regards,
        Kelly

        Did I answer your question? Mark my reply as a solution!