Forum Discussion

thaisstefani18_'s avatar
thaisstefani18_
Frequent Visitor
5 months ago
Solved

Copy Job 2100 / InternalServerError on SharePoint List >2M Items with Incremental Load

Hello everyone, I’ve been exploring Copy Job, and I’m encountering the following error: Operation on target CopyJobActivityLoop failed: Activity failed because an inner activity failed; Inner ac...
  • MJParikh's avatar
    5 months ago

    Hi thaisstefani18_ ,

    This is a known limitation. The error is indicating a constraint on the SharePoint side.

    Root Cause

    When the Copy Job executes its incremental checkpoint, it sends an OData query with the syntax $orderby=Modificado desc&$top=1 on your SharePoint list. Your SharePoint list has over 2 million items. The SharePoint OData endpoint returns an InternalServerError because the query has exceeded the SharePoint List View Threshold (LVT). SharePoint does not execute OData queries that are not indexed when the query scans over 5,000 items. This fails the checkpoint read operation before it transfers any data.

    In response to your question about modifying the Copy Job JSON, this is not supported. This is a confirmed product constraint.

    Recommended Approaches

    1. Index incremental column

    To resolve this issue, you should go to your SharePoint list settings. You should create a column index on your incremental column Modificado. This will enable SharePoint to execute the OData queries with $orderby and $filter on the Modificado column without hitting the threshold. This is the least disruptive option. This should be your first option.

    2. Fabric Data Pipeline with custom watermark

    If the Copy Job is not working, replace it with a Copy Activity in a regular Fabric Data Pipeline. This gives complete control over OData pagination, query folding, and watermark logic managed via a config table in your Lakehouse. This approach is more work but much more reliable with this volume of data.
    3. Microsoft Graph API with delta queries

    For complete data change history with support for inserts, updates, and deletes, the Microsoft Graph API is the most reliable way to go. The sites/{id}/lists/{id}/items endpoint with delta token support is better for large lists than using OData and gives us CDC support.
    4. Fabric Open Mirroring

    For near-real-time data synchronization with minimal custom engineering, Open Mirroring mirrors changes to your SharePoint lists in Fabric without relying on OData polling.
    Important Note on Deletes

    Even with the OData threshold issue fixed, using Merge mode with your Copy Job does not support deletes. Any rows that were removed from your SharePoint list will still be present in your Lakehouse data store. For complete support with inserts, updates, and deletes, the Graph API delta query approach is the correct answer.


    Thank you!
    Proud to be a Super User!
    📩 Need more help?
    ✔️ Don’t forget to Accept as Solution if this guidance worked for you.
    💛 Your Like motivates me to keep helping