Sharepoint folder not synchronising new files

[pasiK]

Hi Guys,

I'm new here and would like some advice reguarding an issue.

PowerBi is not synchronising the dataset using the SharePoint folder source when new files are added to the location. this was working fine couple of weeks ago but when checked on 9th June, the new files are not getting synced. no errors are popping up

Hope i can get some infomration about this 

**

When a single file is updated, there are no issues and that cahnges does sync. 

the issue is only with new files added to the location

[v-pnaroju-msft · Accepted solution]

Thankyou, @IoannisPhilip, @burakkaragoz and @Nasif_Azam for your responses.

Hi pasiK,

We sincerely appreciate your inquiry posted on the Microsoft Fabric Community Forum.

From my understanding, since the existing files are syncing correctly but the newly added files are not appearing, the issue most likely arises from how Power BI processes the folder structure during refresh. This is especially relevant if you are using SharePoint.Files(), which may cache results and fail to detect new entries.

To resolve this matter, kindly follow the steps outlined below:

1. Use SharePoint.Contents() instead of SharePoint.Files(). This method offers more reliable folder-level navigation and helps avoid metadata caching issues.
   a. In Power BI Desktop, please remove the existing SharePoint.Files connection.
   b. Use the function SharePoint.Contents("https://yourtenant.sharepoint.com/sites/yoursite").

2. Avoid hardcoding file names or folder paths. Instead, employ dynamic filtering. For example, filter by file extension as demonstrated below:
   Table.SelectRows(Source, each Text.EndsWith([Extension], ".csv"))

3. Slightly modify the Power Query steps, such as renaming a step temporarily, then save and revert the change. Re-publish the report to Power BI Service. This action may force schema re-evaluation and enable detection of new files.

Additionally, please refer to the following links for more information:
Power Query SharePoint folder connector - Power Query | Microsoft Learn
SharePoint.Contents - PowerQuery M | Microsoft Learn

If you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Should you have any further questions, please feel free to contact the Microsoft Fabric community.

Thank you.

[pasiK]

Hi @v-pnaroju-msft 

Thank you for your support.

replacing the SharePoint.Files() with SharePoint.Contents() did help the issue and the data set seems to be updating with new files now.

* Only downside was custom colums that were made was removed due to the source change. 

[Nasif_Azam]

Hey @pasiK ,

Here’s a detailed guide to troubleshoot and resolve your issue where new files in a SharePoint folder are not syncing with Power BI, while updated files are syncing fine.

You mentioned:

• Updated files sync fine, so Power BI can still access the SharePoint folder.

• New files added to the folder do not appear in Power BI refresh results.

• No errors are shown during refresh.

This strongly suggests a metadata caching or indexing issue in how Power BI accesses the SharePoint folder contents, especially common with the SharePoint folder connector.

Common Fixes & Checks

1. Check File Type Consistency

Power BI often filters based on Extension or Name in the query steps. If new files are of a different extension (e.g., .xlsb instead of .xlsx), they may be skipped silently.

• Go to Power Query Editor > Applied Steps.

• Review steps like Filtered Rows are they filtering specific names, extensions, or patterns?

• Action: Temporarily remove filtering steps and check if new files now show up.

2. Verify Folder Path and File Attributes

Sometimes SharePoint may show the file in the UI but not expose it to the API if:

• The file is not checked in.

• It was uploaded as a draft and is not published.

• The file is in a subfolder not covered by the query.

Action: Ensure that new files:

• Are checked in (if versioning is enabled).

• Have similar attributes as older files.

• Are not in folders excluded by your Power Query logic.

3. Clear Power BI Data Source Cache

Power BI might be using a cached version of the folder content list.

Action: Clear permissions and re-authenticate:

1. In Power BI Desktop, go to:

   • File > Options and Settings > Data Source Settings

2. Select the SharePoint folder connection.

3. Clear permissions and sign in again.

4. Refresh your query.

4. Use SharePoint Folder API URL Instead of Web Connector

Ensure you are using the SharePoint folder connector, not the “Web” connector with a single file path.

The SharePoint folder connector URL should look like:

https://yourcompany.sharepoint.com/sites/YourSite/

Not:

https://yourcompany.sharepoint.com/sites/YourSite/Shared%20Documents/filename.xlsx

If you're not using the folder connector, it will not pick up new files.

5. Rebuild File List in Power Query

Sometimes the folder structure inside Power BI becomes stale.

Action:

1. Open Power Query Editor.

2. Go to the first step (SharePoint.Files or Folder.Files).

3. Refresh preview.

4. Rebuild or reset transformations if you find Filtered Rows or Removed Other Columns steps blocking new files.

Last-Resort Solutions

• Manually refresh the dataset in Power BI Service and check if the new files are picked up.

• Duplicate the query, remove all steps after Navigation, and inspect what files are visible.

• Schedule Refresh Troubleshooting:

  • Go to Power BI Service > Dataset Settings.

  • Ensure no errors in refresh history.

  • Try an on-demand refresh and compare the row count of new vs old files.

Things to remember:

• If your workflow depends on regularly uploading files, consider standardizing file names or maintaining an index list in SharePoint to track active files.

• Long term, consider creating a SharePoint list with metadata about uploaded files and use that as the source.

If you found this solution helpful, please consider accepting it and giving it a kudos (Like) it’s greatly appreciated and helps others find the solution more easily.

Best Regards,
Nasif Azam

[burakkaragoz]

Hi @pasiK ,

This is a common issue with Power BI and SharePoint integration. Here are some steps and things to check that may help resolve your problem:

1. Check the Power BI Refresh Schedule

   • Make sure your dataset is scheduled to refresh regularly in the Power BI Service. If refresh is paused or failing, new files won’t appear.
   • Go to the workspace, find your dataset, and look at the refresh history for any errors or warnings.

2. Verify SharePoint Folder Connection

   • Sometimes the connection to the SharePoint folder needs to be re-authenticated, especially if credentials have changed or expired.
   • In Power BI Service, check the data source credentials for your dataset and re-enter them if needed.

3. File Permissions and Access

   • Ensure the account used by Power BI has access to the new files and the folder in SharePoint.
   • If files are uploaded by different users or permissions are changed, Power BI might not see them.

4. File Metadata and Supported Formats

   • Power BI may not recognize files if their format or metadata is different from the original files.
   • Make sure new files are of the same type and structure as those previously synced.

5. Check for Hidden Errors

   • Even if no error message appears, check the Power BI refresh history for any subtle issues.
   • Sometimes, Power BI will skip files it cannot parse without showing a visible error.

6. Recent Service Updates

   • Occasionally, changes or outages in Power BI or SharePoint can affect synchronization. Check the Microsoft 365 Service Health Dashboard for any known incidents around June 9th.

What you can try:

• Manually refresh the dataset in Power BI Service and check if the new files appear.
• Reconnect the SharePoint folder source in Power BI Desktop, republish the report, and see if that resolves the issue.
• Double-check the SharePoint folder path hasn’t changed, and that the folder still contains all expected files.

If you provide more details (such as the refresh history or errors in the Power BI Service), the community can give more targeted advice.

Hope this helps you get your files syncing again!
translation and formatting supported by AI

[IoannisPhilip]

Hi @pasiK ,

is the refresh successful? Can you elaborate a bit on how you connected the data and if you made any tests to check the updates?

Best,
Ioannis