Access denied to LK files randomly using same connection always.

[ysantana]

Hi,

I've a notebook to ingest files from LH File path to Delta table. Basically, do a copy command to delta table, and then delete the file from the path if succeed. Randomly, i'm receiving the exception: "Operation on target Load File to Bronze failed: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - PermissionError, Error value - [Errno 13] Permission denied'" What's curious is the file get processed usually in following trigger without any exception. Seen few other posts with access denied but none like mine. Can you help me figure it out?

Thanks.

[v-sgandrathi · Accepted solution]

Hi @ysantana,

The PermissionError: [Errno 13] Permission denied you’re encountering appears to be intermittent and transient, likely caused by file-level locking or timing conflicts during access or deletion. This behavior is common in distributed storage systems (like ADLS Gen2 or DBFS) and is not typically related to static permission settings.

Recommended Mitigations:

Introduce a retry mechanism with backoff around file deletion to handle transient lock scenarios gracefully.

Confirm that no other jobs or triggers are accessing the same files or directories simultaneously. Use partitioned file paths or introduce file-naming conventions to avoid collisions.

Consider copying files to a staging area first for processing, then perform the delete operation after confirming success.

Add logging for failed delete operations and consider tracking file access patterns to identify repeat offenders or timing-related issues.

Hope my suggestion gives you good idea, if you have any more questions, please feel free to ask we are here to help you.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thank you,

Sahasra_CST Member.