Forum Discussion
Polars write_delta 408 timeout causes notebook failure
I have a simple Python notebook that copies tables form one lakehouse to another using polars. There are only 5 tables, and they're small: less than 500k each. Other than a few imports, setting paths, and some print statements, this is the whole notebook:
for item in notebookutils.fs.ls(source_lakehouse):
df = pl.read_delta(item.path)
df.write_delta(
f"{target_lakehouse}/{item.name}",
mode="overwrite",
delta_write_options={"schema_mode": "overwrite", 'engine': 'rust'}
)
The notebook works perfectly and runs quickly. However, it failed in a pipeline last night with a timeout error: "OSError: Generic MicrosoftAzure error: Client error with status 408 Request Timeout: <?xml version="1.0" encoding="utf-8"?><Error><Code>OperationTimedOut</Code><Message>Operation could not be completed within the specified time."
According to the logs in the monitor, the notebook ran for less than 60 seconds. Later this morning, I ran the notebook again and it worked fine. I have setup retries on the notebook activity in the pipeline that I used to schedule it, but I'd prefer to know why it failed in the first place.
Can anyone shed any light on why I might be seeing this error?
Thanks!
Thanks for your response. I believe the most likely root cause is user error - after switching the engine to pyarrow (removing 'engine': 'rust' from the delta_write_options), I received a clear error notification about concurrent write transactions when running the pipeline these notebooks are in. I checked, and instead of running separate dev and prod notebooks concurrently, I was running the same prod notebook twice concurrently. I updated my pipeline to run the correct notebooks and I am no longer receiving errors.
2 Replies
Hello sean_cochran
Your code reads and writes data from a Delta table one node at a time using Polars’ delta-rs engine. While delta-rs works well, Fabric notebooks still perform big, time-consuming storage tasks like listing and reading files, committing changes, and saving new files. If there’s heavy demand or if a table creates a few very large files, Azure Storage or OneLake requests might hit limits and return a 408 OperationTimedOut error. This usually happens off and on, depending on things like how busy the system is, the size of the files, and how many tasks are running at once.
If you use the following change in your code, you may see less of this happening:
import time, random import polars as pl MAX_RETRIES = 5 BASE_DELAY = 10 JITTER = (0.5, 1.5) def is_transient_error(exc: Exception) -> bool: m = str(exc) return ( "408" in m or "OperationTimedOut" in m or "timed out" in m.lower() or "Timeout" in m or "temporarily unavailable" in m.lower() or "connection" in m.lower() ) def copy_delta_with_retries(src: str, dst: str): for attempt in range(1, MAX_RETRIES + 1): try: df = pl.scan_delta(src).collect() df.write_delta( dst, mode="overwrite", delta_write_options={"schema_mode": "overwrite", "engine": "rust"}, ) print(f"SUCCESS: {src} → {dst}") return except Exception as e: if attempt == MAX_RETRIES or not is_transient_error(e): print(f"FAILED: {src} → {dst} after {attempt} attempts. Error: {e}") raise delay = BASE_DELAY * (2 ** (attempt - 1)) * random.uniform(*JITTER) print(f"Attempt {attempt} failed for {src}: {e}. Retrying in {delay:.1f}s…") time.sleep(delay) # Driver loop for item in notebookutils.fs.ls(source_lakehouse): src=item.path dst = f"{target_lakehouse}/{item.name}" # Optional: ensure it's a Delta table try: notebookutils.fs.ls(f"{src}/_delta_log") except Exception: print(f"Skipping non-Delta path: {src}") continue copy_delta_with_retries(src, dst)Thanks for your response. I believe the most likely root cause is user error - after switching the engine to pyarrow (removing 'engine': 'rust' from the delta_write_options), I received a clear error notification about concurrent write transactions when running the pipeline these notebooks are in. I checked, and instead of running separate dev and prod notebooks concurrently, I was running the same prod notebook twice concurrently. I updated my pipeline to run the correct notebooks and I am no longer receiving errors.
