Forum Discussion

thaisstefani18_'s avatar
thaisstefani18_
Frequent Visitor
5 months ago
Solved

Mirrored SharePoint List (Preview) - “Failed to get document libraries” Error 404

Hello everyone, I’m exploring the Mirrored SharePoint Online List (preview). I ran a test with a small list and it has been replicating changes correctly. However, I keep seeing the error 'led to ge...
Mirroring
  • thaisstefani18_'s avatar
    thaisstefani18_
    4 months ago

    Hi Tamanchu 

    I managed to finalize yesterday an approach with the pipeline:

    • LookupOldWatermark: reads the last processed date from the DW.
    • Copy Activity: filters the SharePoint List using OData between the previous watermark and the TriggerTime, writing Parquet to the Lake House.
    • Notebook: performs a MERGE of the Parquet into the Delta table using Id as the key, UPDATE if it exists, INSERT if it’s new. Returns the MAX(Modificado) of the processed data.
    • SP UpdateWatermark: saves the MAX(Modificado) as the new watermark for the next run.

     A separate Copy Job was used to perform the initial full load, outside of the pipeline.

    Thanks for your attention 🙂

     

thaisstefani18_'s avatar
thaisstefani18_
Frequent Visitor
4 months ago

Hello Tamanchu 

 

  1. OneLake shortcut to SharePoint
    As I understand, this is currently only available for SharePoint Folder, correct?
  2. Pipeline with copy activity
    I’m trying to implement an incremental approach since the list currently has nearly 2 million records. However, I’m facing some challenges due to source limitations.
  3. Mirroring
    At the moment, mirroring is not replicating all records from the source and also does not keep the selection limited to List, which could help avoid the 404 error.

Other attempts:


In the image below, even when selecting only the “List” option, the mirroring does not preserve this setting, it always ends up with everything selected.

 

Thank you 🙂

Tamanchu's avatar
Tamanchu
Super User
4 months ago

Hi thaisstefani18_,

Hi @thaisstefani18_,
Thank you for the detailed feedback these are real-world challenges that many of us face with large SharePoint lists. Let me address each point with references from the official documentation.
 
1. Mirroring selecting everything instead of just "List"
This is a known behavior in the Preview. According to the Mirrored SharePoint List documentation, mirroring currently replicates both Document Library metadata (via OneLake shortcuts) and managed table data (converted to Parquet/Delta). The selection UI doesn't yet allow granular filtering this is expected to improve as the feature moves toward GA.
My recommendation: Open a support ticket referencing this behavior. Microsoft tracks Preview feedback to prioritize GA improvements. In the meantime, consider the workarounds below for your 2M+ records scenario.
 
2. Pipeline Copy Activity Incremental loading for 2M+ records
The SharePoint Online List connector in Copy Activity does support OData queries for filtering. You can use the $filter parameter with the Modified column to implement incremental loading. From the Copy Activity documentation:
Example OData query for incremental load:
$filter=Modified ge datetime'2026-04-01T00:00:00Z'&$top=5000
Recommended pattern Pipeline with watermark:
1. Use a Lookup Activity to get the MAX(Modified) from your Lakehouse destination table
2. Pass that value as a parameter to the Copy Activity's OData query property:
$filter=Modified ge datetime'@{activity('GetWatermark').output.firstRow.MaxModified}'
3. Set the destination to Append mode in your Lakehouse
Important limitation: SharePoint has a URL length limit of ~2,100 characters for OData queries. Keep your $filter and $select expressions concise. (Troubleshooting reference)
Also, increase the httpRequestTimeout beyond the default 5 minutes — for 2M+ records, consider setting it to 00:30:00 (30 minutes).
 
3. Dataflow Gen2 Memory limit failures
For very large SharePoint lists, Dataflow Gen2 can hit memory limits because OData filters are applied within Fabric after the data is received from the source (as noted in the incremental refresh documentation).
Workaround using incremental amassing pattern:
1. Create a query in your Dataflow that reads the MAX(ID) or MAX(Modified) from your Lakehouse destination
2. Use that value to filter the source, loading only new/changed records
3. Set the destination update method to Append
This way, each refresh only processes a small batch instead of the full 2M records, staying within memory limits.
 
4. Recommended production approach for 2M+ SharePoint records
  • Pipeline + Copy Activity with OData$filer : best for Incremental loads using Modifed column (GA)
  • Dataflow Gen2 + Incremental : best for Small incremental batches with transformations (GA)
  • Mirroring is : Wait for GA with better list selection & large list support (Preview)

The Pipeline approach is the most reliable for your scale.

 

Sources:

Hope this helps!

  • thaisstefani18_'s avatar
    thaisstefani18_
    Frequent Visitor
    4 months ago

    Hi Tamanchu 

    I managed to finalize yesterday an approach with the pipeline:

    • LookupOldWatermark: reads the last processed date from the DW.
    • Copy Activity: filters the SharePoint List using OData between the previous watermark and the TriggerTime, writing Parquet to the Lake House.
    • Notebook: performs a MERGE of the Parquet into the Delta table using Id as the key, UPDATE if it exists, INSERT if it’s new. Returns the MAX(Modificado) of the processed data.
    • SP UpdateWatermark: saves the MAX(Modificado) as the new watermark for the next run.

     A separate Copy Job was used to perform the initial full load, outside of the pipeline.

    Thanks for your attention 🙂