Forum Discussion
Mismatched minReaderVersion and readerFeatures
Hi Anonymous ,
You can take a backup of the table and try to alter the property by running the SQL command below in Spark notebook:
ALTER TABLE <table-identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')
This might help to fix the issue.But as you said, this looks like a bug since it had been working in previous executions.
Hi Anonymous
Thanks for using Fabric Community.
The error SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3) indicates a compatibility issue between the Spark version used for writing data and the Spark version used for reading it. This often occurs due to library or configuration changes.
Troubleshooting Steps:
1) Check Spark Versions: Verify the Spark versions used in your notebook and pipeline nodes. They should be consistent. If necessary, upgrade or downgrade Spark versions to align them.
2) Review Library Versions: Ensure compatibility between Spark and any external libraries or dependencies you're using. Check for updates or known issues with specific library versions.
3) Inspect Data Format: Examine the structure and format of the data being written to the bot table. Ensure it's compatible with the Spark version used for reading. If necessary, adjust the data format or writing process to maintain compatibility.
You can also refer to this post which is related to the same error: https://community.fabric.microsoft.com/t5/Dataflows/Error-using-data-imported-through-DataFlow-when-using-pyspark/m-p/3530620
Hope this helps. Please let us know if you have any further queries.
The scheduler executed 10 or more times and successfully append API data into table, I don't think the error is due to the reason you listed, since I didn't modify anything after I setup up the scheduler.
I found there's another thread which created at 2023-12-01, and the solution said it's a bug, will be fixed in the end of that week. However, another reply at 2023-12-28 said he/she still met the same problem.
https://community.fabric.microsoft.com/t5/General-Discussion/Notebook-error-reading-delta-table/m-p/3605205