Forum Discussion
Create multiple Tables from Python
Hi,
Using Python I'm connecting to Google Cloud Storage, I want to be able to loop through the various files in a bucket, create pandas dataframes for each one and then use the contents in Power Query.
I can get it all working in Pycharm with the below...
for file in blobs:
file_path = "gs://{}/{}".format(file.bucket.name, file.name)
with fs.open(file_path) as f:
df = pd.DataFrame
df.name = file.name
df = pd.read_csv(f)
print(df.name)
print(df)
it loops through and prints the contents and name of each of the 3 files in the bucket, but in Power Query I only pull through the last iteration with the name df.
I've tried various angles using dictionaries and lists but can't figure out how to end up with three separate tables. I've seen examples concatenating various files but I want to keep these all separate. I want to be able to loop through the files in the bucket because there will eventually be hundreds, so don't want to hard code each one.
Any ideas? Is it possible?
Cheers
Mark
- Anonymous4 years ago
Hi,
I'm not sure is this thread still relevant. I had the same issue and I solved this by associating each data frame to a unique global variable:
for file in blobs: file_path = "...." with fs.open(file_path) as f: globals()[f"{file.name}"] = pd.read_csv(f) print(globals()[f"{file.name}"])
12 Replies
- AnonymousNot applicable
Hi,
I'm not sure is this thread still relevant. I had the same issue and I solved this by associating each data frame to a unique global variable:
for file in blobs: file_path = "...." with fs.open(file_path) as f: globals()[f"{file.name}"] = pd.read_csv(f) print(globals()[f"{file.name}"])- AnonymousNot applicable
Thank you mate, I've moved company so not working on that solution anymore but this might prove handy in the future!
- v-kelly-msftCommunity Support
Hi Anonymous ,
How about filter table name to seperate them one by one after you have loaded all to Power bi?
Best Regards,
KellyDid I answer your question? Mark my reply as a solution!
- AnonymousNot applicable
Not sure what you mean...it's the loading them to Power BI part that I have a problem with...
I can only see one table in Power Query, I want to have a list of all the tables representing the files that are in the GCS Bucket...
- v-kelly-msftCommunity Support
Hi Anonymous ,
Sorry for the misunderstanding.
Are the python statements full?Have you defined "blobs" first?
Best Regards,
KellyDid I answer your question? Mark my reply as a solution!