Copy Activity Reports Success but Writes Truncated Parquet Files (After migration to Sweden Region)

Hi everyone,

We are currently facing serious stability issues with Parquet files written to OneLake in Sweden region, which started immediately after migrating from Northern Europe.

This is not a new solution — it has been running stable for a long time in Northern Europe. No architectural or pipeline changes were made during the migration.

Architecture

• Fabric Pipelines
• Copy activity
• Source → Parquet files in OneLake (Files/bronze)
• ~120 tables loaded in parallel
• Each table writes to its own unique path
• No concurrent writes to the same file
• No concurrent reads during write

No schema merge or overwrite conflicts

The Issue

Almost daily, one of our environments produces a Parquet file that:

• Pipeline run = Success
• File exists in OneLake

But Spark fails to read it

Error:

org.apache.spark.SparkException: [CANNOT_READ_FILE_FOOTER]Could not read footer for file: abfss://...parquet

Root cause:

java.lang.RuntimeException: ...parquet is not a Parquet file (length is too low: 4)

This strongly indicates that the file is truncated (essentially only a few bytes written) and the Parquet footer was never committed.

Why This Is Concerning

• The same pipelines were stable for a long time in Northern Europe.
• Problems started immediately after region migration.
• Failures occur randomly across tables.
• No solution changes were introduced.
• The pipeline reports success despite the file being corrupt.

  This suggests that:

  • The write operation may not be fully committed before the activity completes.
  • There could be region-specific storage/commit behavior differences.
• Or a potential issue in OneLake / Copy activity handling in Sweden region.

Questions

Has anyone experienced:

• Copy activity reporting success while file is incomplete?
• Region-specific stability differences?
• Known issues in Sweden/Other region?

  At this point, this looks less like a workload design issue and more like a potential platform or regional infrastructure problem.

   

Any insights, confirmations, or similar experiences would be highly appreciated.
If we can not see a stable fix will we most likely redesign to use delta at ingest/bronze stage and avoid raw parquet totally.

Thanks.