Forum Discussion

abm_chris's avatar
abm_chris
Icon for Advocate I rankAdvocate I
1 year ago
Solved

Python Notebook duckdb/delta intermittent permission errors

Hi,  I have been having intermittent issues over the past couple of weeks with duckdb executing sql queries and getting the below error: IOException: IO Error: Hit DeltaKernel FFI error (from: ...
data engineering
notebook
  • Shahid12523's avatar
    Shahid12523
    1 year ago

    The error is from DuckDB’s Delta Kernel when reading _delta_log files — it’s not a real permission issue, but an intermittent I/O/concurrency bug in Fabric.

    ✅ Fixes 

    Add retry logic around DuckDB queries.

    Avoid concurrent Spark + DuckDB reads on the same table.

    Keep DuckDB/Fabric runtime updated.

    If critical, use Spark SQL instead of DuckDB for stability.

    👉 It’s a platform-side intermittent bug, not your config.

cvonredapt's avatar
cvonredapt
Frequent Visitor
1 year ago

I will admit to not knowing anything about DuckDB, but since this sounds like an intermittent issue, it may be a concurrency issue. I have encountered concurrency issues in Databricks, and this sounds like it may be a similar issue. Thank you for poking my curiousity on DuckDB. I am a fan of pyspark, but it is always good to have my horizons expanded.