Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Mismatched minReaderVersion and readerFeatures

1. I used csv file to create table `bot` in lakehouse, it had 3 rows in the begining. 2. I created a notebook to get the latest id in table `bot`, the code is as below:         # Welcome to yo...
power bi
  • govindarajan_d's avatar
    govindarajan_d
    2 years ago

    Hi Anonymous ,

    You can take a backup of the table and try to alter the property by running the SQL command below in Spark notebook:

    ALTER TABLE <table-identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')
    This might help to fix the issue. 
     
    But as you said, this looks like a bug since it had been working in previous executions. 
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous 
Thanks for using Fabric Community.
The error SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3) indicates a compatibility issue between the Spark version used for writing data and the Spark version used for reading it. This often occurs due to library or configuration changes.
Troubleshooting Steps:

1) Check Spark Versions: Verify the Spark versions used in your notebook and pipeline nodes. They should be consistent. If necessary, upgrade or downgrade Spark versions to align them.

2) Review Library Versions: Ensure compatibility between Spark and any external libraries or dependencies you're using. Check for updates or known issues with specific library versions.

3) Inspect Data Format: Examine the structure and format of the data being written to the bot table. Ensure it's compatible with the Spark version used for reading. If necessary, adjust the data format or writing process to maintain compatibility.

You can also refer to this post which is related to the same error: https://community.fabric.microsoft.com/t5/Dataflows/Error-using-data-imported-through-DataFlow-when-using-pyspark/m-p/3530620

Hope this helps. Please let us know if you have any further queries.

christian_c's avatar
christian_c
New Member
2 years ago

thanks nikhilan. This solved the issue for me