Forum Discussion
dynamically import files from a folder WITHOUT combining them
Must admit that I'm a bit lost with that new approach. I still doubt that you will be able to create multiple queries at once with it. But in order to nail your current problem down to RAM-issues, I'd suggest that you try your new method on a folder with just 2 files in it to test the general setup.
Disclaimer: I'm new to both Power BI and R (only a couple months into learning both).
I wrote an R script that dynamically generates dataframes for each file of a specified file type from a specified folder and then saves those dataframes to an R environment '.RData' file. Then, in Power BI Desktop, I use R Script as a datasource and use 'load("filename.RData") to load the dataframes from that environment without running the entire underlying script that generated those dataframes -- which gives me control over 'refresh' (manually accomplished in R) and saves processing time in Power BI. When an R script is used as a data source, it pulls compatible R objects (dataframes, matrices, lists) into Power BI as separate queries/data sources. This is what I wanted to achieve. However, there seems to be a limit to the number of R objects Power BI can load into a Desktop session. I can't load all ~120 dataframes, so now I'm using trial and error to find my limit for successful dataframe loading. Regardless, there does seem to be a limit, but I'm not sure if the limit is due to an internal Power BI constraint (number of objects and/or the size of the objects), or due to my own system constraints (RAM, etc).
Cool - that's a very interesting discovery!
I'm new to R as well and haven't spent much time with it yet. Also haven't seen a lot of R-scripting in PowerBI yet, so don't know which expert to call in here. But as you seem to discover already: This doesn't seem to be an R-issue, but more a PowerBI limitation.
You must have a very specific use case here that seems worth to make a big efford to skip the first step of the query design - for queries that then will be totally different. Normally I would have said that it is a good thing that we cannot import multiple files the way you want it. As if you apply the same steps in each query, you should instead create one query with the source-names as attributes in an additional column.
So sorry, no further ideas or help from my side here.
Hi,
Can you please have a look at myissue
I think I have the similar issue. Can you guide me through it?
Thanks.
Hi ajuanto,
this is the error-message I'm getting when trying to open the link you've posted:
So please post link that can be accessed publicly, thanks.
Hi @Imke,
Here is the message > \
Hi I am new to Power BI and I am facing an issue.
I have a Folder of csv files which I have to import. The csv files has readings of a electronic device,
I imported the csv folder and I got such a screen
Then I tried to remove the unwanted columns
now I have a column that shows the sourse file that is in my case the csv file for eg-MFT_RB1_2017-08-08_11-26-56.csv
I am getting all the data i.e the combined data of over 100 csv file in one sigle table as shown above. I want to create diffrent tables for each csv files. I can do it manually by importing csv files seperately like this -
But when I am importing a folder I am getting a sigle combined table, I want to split the main table into tables=number of csv files and visualiye the output as above
(I got the above visualization by importing 6 csv one by one, which works perfect)
But when importing a complete folder I am getting one big table with all 100 csv combined and I want to create diffrent tables for diffrent csv like I did with the above visualization.
Also, I have over 100 csv files in the folder and if I add 50 more in the folder it should automatically create tables of those newly added csv files.
Can anyone help me with this?
