Forum Discussion

12angrymentiger's avatar
12angrymentiger
Advocate III
2 years ago
Solved

Spark SQL Query errors - Case Sensitivity issues

All of a sudden as of almost 2 weeks ago our spark sql notebooks have been failing due to case sensitivity in the table and field names.  This has always worked fine in Synapse before, so I'm not sur...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi 12angrymentiger 
    Apologies for the issue you have been facing. At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.

    A simple workaround to fix this issue is to run the below piece of code before executing the main code.

    spark.conf.set('spark.sql.caseSensitive', False)

    Hope this helps. Please let me know if you have any further questions. Glad to help.

12angrymentiger's avatar
12angrymentiger
Advocate III
2 years ago

Anonymous  I wanted to check with you regarding this issue.

 Please let me know if you have heard of any fixes for this.

 

Thanks

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi 12angrymentiger 
Thanks for using Fabric Community. Apologies for the issue you have been facing.
Can you please confirm which Spark Version are you using? 
I have confirmed from the internal team that there were no changes done to the case sensitivity. Did you set any spark configurations? If yes can you please give the details about it.
Thanks.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi 12angrymentiger ,
    I tried to repro the scenario, but did not get any error. I have attached the screenshot.

     

     

    Please try to check the Spark version which you ar using.

     

     

     

    Make sure you use the Spark 1.2 version. 
    Hope this helps. Please let me know if you have any further queries.

    • 12angrymentiger's avatar
      12angrymentiger
      Advocate III
      2 years ago

      By the way this is still in My Workspace and under the Fabric Trial, but I did change the workspace to use Runtime 1.2 and I'm still getting the error table or view not found.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi 12angrymentiger 
        Apologies for the issue you have been facing. At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.

        A simple workaround to fix this issue is to run the below piece of code before executing the main code.

        spark.conf.set('spark.sql.caseSensitive', False)

        Hope this helps. Please let me know if you have any further questions. Glad to help.