Python Notebook duckdb/delta intermittent permission errors

[abm_chris]

Hi, 

I have been having intermittent issues over the past couple of weeks with duckdb executing sql queries and getting the below error:

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: '{table}'): Hit error: 8 (ObjectStoreError) with message (Error interacting with object store: Generic LocalFileSystem error: Unable to open file {table}/_delta_log/00000000000000000228.json: Permission denied (os error 13))

This isn't isolated to one table or file but seems to randomly happen at any point in time. I have had a period of around 10 days it hasn't reoccured until yesterday.

Has anyone had experience with this issue before?

[Shahid12523 · Accepted solution]

The error is from DuckDB’s Delta Kernel when reading _delta_log files — it’s not a real permission issue, but an intermittent I/O/concurrency bug in Fabric.

✅ Fixes 

Add retry logic around DuckDB queries.

Avoid concurrent Spark + DuckDB reads on the same table.

Keep DuckDB/Fabric runtime updated.

If critical, use Spark SQL instead of DuckDB for stability.

👉 It’s a platform-side intermittent bug, not your config.

[abm_chris]

This is what I was expecting it to be. Have added retry logic in for now and hopefully gets fixed soon.

Data is too small to consider spark which is why we went with the DuckDB pattern, just strange it has been fine for months and suddenly an issue.

[v-lgarikapat]

Hi @abm_chris  Thank you for sharing this workaround it's truly appreciated and will be incredibly helpful for others in the community.

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

Best Regards,

Lakshmi.

[abm_chris]

I'll still be waiting for a fix from PG 👍

[v-lgarikapat]

Hi @abm_chris ,

Thanks for reaching out to the Microsoft fabric community forum.,

@cvonredapt , @BhaveshPatel , 

Thanks for your prompt response

@abm_chris ,  

I wanted to follow up and confirm whether you’ve had the opportunity to review the information provided by @cvonredapt , @BhaveshPatel . If you have any questions or need further clarification, please don’t hesitate to reach out.

We appreciate your collaboration and support!

Best regards,
Lakshmi.

[cvonredapt]

I will admit to not knowing anything about DuckDB, but since this sounds like an intermittent issue, it may be a concurrency issue. I have encountered concurrency issues in Databricks, and this sounds like it may be a similar issue. Thank you for poking my curiousity on DuckDB. I am a fan of pyspark, but it is always good to have my horizons expanded. 

[BhaveshPatel]

You can observe such issues with Microsoft Fabric Delta Lake. It is an open source & relatively new platform. But my question is why are you using Duckdb. 

Instead use pyspark and delta lake to get in memory engine features.

python --> pyspark --> delta lake. This is how databricks has designed the data architecture.