Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
efilipe
Helper IV
Helper IV

Stop reading all text files again

Hello!

 

I have a directory with about 3 thousand text files of 1kb each. I ask PowerBI to open each of them, get the header of the file, and then split by position - voilá.

 

In this directory new files are added everyday. But it will always have about 3 thousand files, because I have a routine that erases files older than an year.

 

The files are in the network, so access is slower than working local.

 

So, what I'd like, is in a way I dont have to update it manually, and that PowerBI doesnt have to read all the files again just because of those new files that came in today. 🙂

 

Any suggestion?

1 ACCEPTED SOLUTION
v-xicai
Community Support
Community Support

Hi @efilipe ,

 

Assuming that you have uploaded your text files into SharePoint Online, then you may use SharePoint Folder connector to combine or consolidate these files into a single table, transform the data in Query Editor, like "get the header of the file, and then split by position - voilá", see how to get SharePoint Online data from the link: Combining Excel Files hosted on a SharePoint folder with Power BI .

 

Then for the updated data today you may set Incremental refresh , and the Incremental refreshes only data that has changed.  Since incremental refresh is Premium only feature, so you may need to purchase Power BI Premium capacity for your organization, see How to purchase Power BI Premium. Incremental refresh policies are defined in Power BI Desktop and applied when published to the Power BI service.  See more detail steps: Incremental refresh in Power BI Premium , Incremental refresh for files in a Folder or SharePoint – Power BI

 

Best Regards,

Amy 

 

Community Support Team _ Amy

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

3 REPLIES 3
v-xicai
Community Support
Community Support

Hi @efilipe ,

 

Assuming that you have uploaded your text files into SharePoint Online, then you may use SharePoint Folder connector to combine or consolidate these files into a single table, transform the data in Query Editor, like "get the header of the file, and then split by position - voilá", see how to get SharePoint Online data from the link: Combining Excel Files hosted on a SharePoint folder with Power BI .

 

Then for the updated data today you may set Incremental refresh , and the Incremental refreshes only data that has changed.  Since incremental refresh is Premium only feature, so you may need to purchase Power BI Premium capacity for your organization, see How to purchase Power BI Premium. Incremental refresh policies are defined in Power BI Desktop and applied when published to the Power BI service.  See more detail steps: Incremental refresh in Power BI Premium , Incremental refresh for files in a Folder or SharePoint – Power BI

 

Best Regards,

Amy 

 

Community Support Team _ Amy

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Well, you know it's not that... 🙂

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Power BI Monthly Update - February 2024

Check out the February 2024 Power BI update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.