topic Re: Python Notebook duckdb/delta intermittent permission errors in Data Engineering

Python Notebook duckdb/delta intermittent permission errors

abm_chris — Fri, 29 Aug 2025 07:01:30 GMT

Hi,

I have been having intermittent issues over the past couple of weeks with duckdb executing sql queries and getting the below error:

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: '{table}'): Hit error: 8 (ObjectStoreError) with message (Error interacting with object store: Generic LocalFileSystem error: Unable to open file {table}/_delta_log/00000000000000000228.json: Permission denied (os error 13))

This isn't isolated to one table or file but seems to randomly happen at any point in time. I have had a period of around 10 days it hasn't reoccured until yesterday.

Has anyone had experience with this issue before?

Re: Python Notebook duckdb/delta intermittent permission errors

BhaveshPatel — Fri, 29 Aug 2025 10:04:04 GMT

You can observe such issues with Microsoft Fabric Delta Lake. It is an open source & relatively new platform. But my question is why are you using Duckdb.

Instead use pyspark and delta lake to get in memory engine features.

python --> pyspark --> delta lake. This is how databricks has designed the data architecture.

Re: Python Notebook duckdb/delta intermittent permission errors

cvonredapt — Fri, 29 Aug 2025 14:03:39 GMT

I will admit to not knowing anything about DuckDB, but since this sounds like an intermittent issue, it may be a concurrency issue. I have encountered concurrency issues in Databricks, and this sounds like it may be a similar issue. Thank you for poking my curiousity on DuckDB. I am a fan of pyspark, but it is always good to have my horizons expanded.

Re: Python Notebook duckdb/delta intermittent permission errors

v-lgarikapat — Mon, 01 Sep 2025 07:02:35 GMT

Hi @abm_chris ,

Thanks for reaching out to the Microsoft fabric community forum.,

@cvonredapt , @BhaveshPatel ,

Thanks for your prompt response

@abm_chris ,

I wanted to follow up and confirm whether you’ve had the opportunity to review the information provided by @cvonredapt , @BhaveshPatel . If you have any questions or need further clarification, please don’t hesitate to reach out.

We appreciate your collaboration and support!

Best regards,
Lakshmi.

Re: Python Notebook duckdb/delta intermittent permission errors

Shahid12523 — Mon, 01 Sep 2025 07:49:02 GMT

The error is from DuckDB’s Delta Kernel when reading _delta_log files — it’s not a real permission issue, but an intermittent I/O/concurrency bug in Fabric.

✅ Fixes

Add retry logic around DuckDB queries.

Avoid concurrent Spark + DuckDB reads on the same table.

Keep DuckDB/Fabric runtime updated.

If critical, use Spark SQL instead of DuckDB for stability.

👉 It’s a platform-side intermittent bug, not your config.

Re: Python Notebook duckdb/delta intermittent permission errors

abm_chris — Mon, 01 Sep 2025 07:54:40 GMT

This is what I was expecting it to be. Have added retry logic in for now and hopefully gets fixed soon.

Data is too small to consider spark which is why we went with the DuckDB pattern, just strange it has been fine for months and suddenly an issue.

Re: Python Notebook duckdb/delta intermittent permission errors

v-lgarikapat — Mon, 01 Sep 2025 08:01:10 GMT

Hi @abm_chris Thank you for sharing this workaround it's truly appreciated and will be incredibly helpful for others in the community.

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

Best Regards,

Lakshmi.

Re: Python Notebook duckdb/delta intermittent permission errors

abm_chris — Mon, 01 Sep 2025 08:04:32 GMT

I'll still be waiting for a fix from PG 👍