Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Pyspark Parametrization not working correctly

Hello,

According to the Fabric Runtime 1.2 release notes for Microsoft Fabric Runtime 1.2 in Fabric - Microsoft Fabric | Microsoft Learn, the parameterization of Spark SQL should be supported. However, when I attempt to run a simple SQL query as shown below, I encounter a syntax error.

I get the following syntax error 

Could you please help me identify the issue and suggest a solution?

Thank you.

 

data engineering
Lakehouse
Notebook
Spark Job Definition
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Anonymous 
    Thanks for bringing this to our notice. The internal team has confirmed this as a bug. I have created an ICM for the same. I will let you know once this gets resolved.

    Thanks

5 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hey Anonymous  ,
    Thanks for replying.
    Your solution does indeed work , however what I am trying to do is to convert from what you suggested to the format suggested here [SPARK-41271] Parameterized SQL - ASF JIRA (apache.org).
    and I don't understand why it is not supported even though it says it is in the fabric runtime relase notes 

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Anonymous 
      Thanks for bringing this to our notice. The internal team has confirmed this as a bug. I have created an ICM for the same. I will let you know once this gets resolved.

      Thanks

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hey Anonymous 
        Thank you for raising this , can you link me the ICM ticket please.

        Thanks

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Anonymous 
    Thanks for using Fabric Community.
    You can try the below code:

    param1 = "abcde"
    spark.sql(f"CREATE TABLE {param1}").show()


    Hope this helps. Please let me know if you have any further questions.