Polars write_delta 408 timeout causes notebook failure

Hello sean_cochran 

Your code reads and writes data from a Delta table one node at a time using Polars’ delta-rs engine. While delta-rs works well, Fabric notebooks still perform big, time-consuming storage tasks like listing and reading files, committing changes, and saving new files. If there’s heavy demand or if a table creates a few very large files, Azure Storage or OneLake requests might hit limits and return a 408 OperationTimedOut error. This usually happens off and on, depending on things like how busy the system is, the size of the files, and how many tasks are running at once.

If you use the following change in your code, you may see less of this happening:

import time, random
import polars as pl

MAX_RETRIES = 5
BASE_DELAY = 10
JITTER = (0.5, 1.5)

def is_transient_error(exc: Exception) -> bool:
    m = str(exc)
    return (
        "408" in m
        or "OperationTimedOut" in m
        or "timed out" in m.lower()
        or "Timeout" in m
        or "temporarily unavailable" in m.lower()
        or "connection" in m.lower()
    )

def copy_delta_with_retries(src: str, dst: str):
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            df = pl.scan_delta(src).collect()
            df.write_delta(
                dst,
                mode="overwrite",
                delta_write_options={"schema_mode": "overwrite", "engine": "rust"},
            )
            print(f"SUCCESS: {src} → {dst}")
            return

        except Exception as e:
            if attempt == MAX_RETRIES or not is_transient_error(e):
                print(f"FAILED: {src} → {dst} after {attempt} attempts. Error: {e}")
                raise
            delay = BASE_DELAY * (2 ** (attempt - 1)) * random.uniform(*JITTER)
            print(f"Attempt {attempt} failed for {src}: {e}. Retrying in {delay:.1f}s…")
            time.sleep(delay)

# Driver loop
for item in notebookutils.fs.ls(source_lakehouse):
    src=item.path
    dst = f"{target_lakehouse}/{item.name}"

    # Optional: ensure it's a Delta table
    try:
        notebookutils.fs.ls(f"{src}/_delta_log")
    except Exception:
        print(f"Skipping non-Delta path: {src}")
        continue

    copy_delta_with_retries(src, dst)