Full Load from API Skipping some files

Hi max_mrc ,

Thanks for reaching out to the Microsoft fabric community forum.
Caues 

API Pagination/Throttling Behaviour

• Some API calls may fail silently or be throttled, returning partial data.
• If you're calling the API in parallel or in large batches, errors may not be retried correctly, and Fabric might skip the failed chunks.
• Check if your pipeline has error-handling or retries on failed API calls.

2. Fabric Capacity Bursting/Smoothing

• Fabric tries to burst workloads over the set capacity using smoothing.
• If your job exceeds the burst buffer too quickly or for too long, operations may get dropped or throttled — especially less critical ones, like background loads.
• This could explain why some files/data are skipped without error.

3. Concurrency / Parallelism

• If your pipeline processes API calls in parallel, it can spike your capacity usage.
• That spike may lead to task drops or silent failure of lower-priority tasks.

Best Practices & Fixes

1. Limit Concurrency

• In your copy activity or loop, set degree of parallelism to a lower number (e.g., 2–4).
• Fabric defaults to high parallelism in some cases, which might spike capacity use.
• 2. Throttle API Calls

• Introduce a wait/delay (e.g., sleep 1s) between each API call or page fetch.
• Use a custom Until loop or ForEach with delay to slow down execution and reduce load.

3. 3. Implement Retries and Logging

• Use robust retry logic in each API call (3+ retries with exponential backoff).
• Log each API call result (success/failure), even in a separate table if needed, to detect any silent skips.

4. 4. Partition Your Load

• Break the full load into smaller, deterministic partitions (e.g., by date, region, or ID range).
• This helps in tracking what's been loaded and allows for easy retry of failed partitions.

5. 5. Monitor Capacity Consumption

• Use Fabric Monitoring tools or Azure Monitor Metrics to watch CPU and memory usage in real time.
• Set alerts for near-capacity thresholds to know when you’re close to limits.

6. 6. Use Dataflows Gen2 or Data Pipeline Alternatives

• For larger full loads, consider:

• Dataflows Gen2 (if available) for chunked API ingestion.
• Staging into Blob/ADLS first, then processing into Lakehouse.

 Optional: "Take Only 80% of Capacity"

There’s no direct way to say “only use 80% of capacity,” but you can

simulate that behaviour by:

• Reducing parallelism.
• Adding delay/sleep in loops.
• Spreading load over more pipeline runs (e.g., time-partitioned loads).
• Reducing dataset size per copy activity.

Evaluate and optimize your Microsoft Fabric capacity - Microsoft Fabric | Microsoft Learn

Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn
Plan your capacity size - Microsoft Fabric | Microsoft Learn

Smoothing and Throttling - Microsoft Fabric | Microsoft Learn

Scale your Fabric capacity - Microsoft Fabric | Microsoft Learn

If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana

ana.