Forum Discussion

pasiK's avatar
pasiK
Regular Visitor
1 year ago
Solved

Sharepoint folder not synchronising new files

Hi Guys,   I'm new here and would like some advice reguarding an issue.   PowerBi is not synchronising the dataset using the SharePoint folder source when new files are added to the location. thi...
Need Help
  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    1 year ago

    Thankyou, IoannisPhilip, burakkaragoz and Nasif_Azam for your responses.

    Hi pasiK,

    We sincerely appreciate your inquiry posted on the Microsoft Fabric Community Forum.

    From my understanding, since the existing files are syncing correctly but the newly added files are not appearing, the issue most likely arises from how Power BI processes the folder structure during refresh. This is especially relevant if you are using SharePoint.Files(), which may cache results and fail to detect new entries.

    To resolve this matter, kindly follow the steps outlined below:

    1. Use SharePoint.Contents() instead of SharePoint.Files(). This method offers more reliable folder-level navigation and helps avoid metadata caching issues.
      a. In Power BI Desktop, please remove the existing SharePoint.Files connection.
      b. Use the function SharePoint.Contents("https://yourtenant.sharepoint.com/sites/yoursite").

    2. Avoid hardcoding file names or folder paths. Instead, employ dynamic filtering. For example, filter by file extension as demonstrated below:
      Table.SelectRows(Source, each Text.EndsWith([Extension], ".csv"))

    3. Slightly modify the Power Query steps, such as renaming a step temporarily, then save and revert the change. Re-publish the report to Power BI Service. This action may force schema re-evaluation and enable detection of new files.

    Additionally, please refer to the following links for more information:
    Power Query SharePoint folder connector - Power Query | Microsoft Learn
    SharePoint.Contents - PowerQuery M | Microsoft Learn

    If you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

    Should you have any further questions, please feel free to contact the Microsoft Fabric community.

    Thank you.

Nasif_Azam's avatar
Nasif_Azam
Icon for Super User rankSuper User
1 year ago

Hey pasiK ,

Here’s a detailed guide to troubleshoot and resolve your issue where new files in a SharePoint folder are not syncing with Power BI, while updated files are syncing fine.

 

You mentioned:

  • Updated files sync fine, so Power BI can still access the SharePoint folder.

  • New files added to the folder do not appear in Power BI refresh results.

  • No errors are shown during refresh.

This strongly suggests a metadata caching or indexing issue in how Power BI accesses the SharePoint folder contents, especially common with the SharePoint folder connector.

 

Common Fixes & Checks

1. Check File Type Consistency

Power BI often filters based on Extension or Name in the query steps. If new files are of a different extension (e.g., .xlsb instead of .xlsx), they may be skipped silently.

  • Go to Power Query Editor > Applied Steps.

  • Review steps like Filtered Rows are they filtering specific names, extensions, or patterns?

  • Action: Temporarily remove filtering steps and check if new files now show up.

2. Verify Folder Path and File Attributes

Sometimes SharePoint may show the file in the UI but not expose it to the API if:

  • The file is not checked in.

  • It was uploaded as a draft and is not published.

  • The file is in a subfolder not covered by the query.

Action: Ensure that new files:

  • Are checked in (if versioning is enabled).

  • Have similar attributes as older files.

  • Are not in folders excluded by your Power Query logic.

3. Clear Power BI Data Source Cache

Power BI might be using a cached version of the folder content list.

Action: Clear permissions and re-authenticate:

  1. In Power BI Desktop, go to:

    • File > Options and Settings > Data Source Settings

  2. Select the SharePoint folder connection.

  3. Clear permissions and sign in again.

  4. Refresh your query.

4. Use SharePoint Folder API URL Instead of Web Connector

Ensure you are using the SharePoint folder connector, not the “Web” connector with a single file path.

The SharePoint folder connector URL should look like:

https://yourcompany.sharepoint.com/sites/YourSite/

Not:

https://yourcompany.sharepoint.com/sites/YourSite/Shared%20Documents/filename.xlsx

If you're not using the folder connector, it will not pick up new files.

5. Rebuild File List in Power Query

Sometimes the folder structure inside Power BI becomes stale.

Action:

  1. Open Power Query Editor.

  2. Go to the first step (SharePoint.Files or Folder.Files).

  3. Refresh preview.

  4. Rebuild or reset transformations if you find Filtered Rows or Removed Other Columns steps blocking new files.

Last-Resort Solutions

  • Manually refresh the dataset in Power BI Service and check if the new files are picked up.

  • Duplicate the query, remove all steps after Navigation, and inspect what files are visible.

  • Schedule Refresh Troubleshooting:

    • Go to Power BI Service > Dataset Settings.

    • Ensure no errors in refresh history.

    • Try an on-demand refresh and compare the row count of new vs old files.

Things to remember:

  • If your workflow depends on regularly uploading files, consider standardizing file names or maintaining an index list in SharePoint to track active files.

  • Long term, consider creating a SharePoint list with metadata about uploaded files and use that as the source.

 

If you found this solution helpful, please consider accepting it and giving it a kudos (Like) it’s greatly appreciated and helps others find the solution more easily.


Best Regards,
Nasif Azam