Forum Discussion

sivanarayanan's avatar
sivanarayanan
New Member
9 months ago
Solved

AdlsGen2ForbiddenError on Fabric Data Warehouse staged copy for one large table (after ~1 hour)

The problem The copy activity runs for about 1–1:15 hours. After that, it fails with AdlsGen2ForbiddenError on a CreateFile operation to the staging path. Other tables using the same pipelin...
data engineering
data pipeline
lakehouse
  • sivanarayanan's avatar
    sivanarayanan
    9 months ago

    Hi wardy912 , nielsvdc 

    Thanks for your reply. 

    With my use cases,

    wardy912, I cannot have multiple copy data activities at the same time within the ForEach 
    nielsvdc As I use built-in enable staging, I am unable to use a service principal for authentication, and splitting the copy activity into smaller batches will not work for me. 

    I solved the issue by using external storage instead of workspace storage. 

nielsvdc's avatar
nielsvdc
Icon for Super User rankSuper User
9 months ago

Hi sivanarayanan, Fabric pipelines use Azure Storage SAS tokens or OAuth tokens for staging in ADLS Gen2. These tokens typically have a limited lifetime, commonly 1 hour by default for SAS tokens. If your copy activity runs longer than that, the token may expire before the staging write completes, causing the Forbidden error. This aligns with your problem: the failure occurs after ~1 hour.

 

You can solve this in a couple of ways:

  1. Create a shortcut to the storage in a Lakehouse and use this as your source.
  2. Use a service principal for authentication to the storage account instead of a SAS token or OAuth.
  3. Use a Copy Job instead of a pipeline with a copy activity. A Copy Job uses parallelism and bypasses intermediate staging of the data before ingesting it into you warehouse.
  4. Consider splitting the copy actvity into smaller batches to keep each copy activity process under 1 hour.

Hope this helps. If so, please give kudos 👍 and mark as Accepted Solution ✔️ to help others.