Forum Discussion

kmvaishakha's avatar
kmvaishakha
New Member
1 year ago

The Spark SQL phase analysis failed with an internal error

Environment: Runtime 1.3 (Spark 3.5 , delta 3.2)
I am getting this error when i do Select * from schema.table
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. org.apache.spark.SparkException$.internalError(SparkException.scala:107) org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:800) org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:812) org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:440) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:439) org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:174) org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:163) org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:114) org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:98) org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:752) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:743) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:774) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:805) org.apache.livy.repl.SQLInterpreter.execute(SQLInterpreter.scala:163) org.apache.livy.repl.Session.$anonfun$executeCode$1(Session.scala:909) scala.Option.map(Option.scala:230) org.apache.livy.repl.Session.executeCode(Session.scala:906) org.apache.livy.repl.Session.$anonfun$execute$17(Session.scala:597) org.apache.livy.repl.Session.withRealtimeOutputSupport(Session.scala:1169) org.apache.livy.repl.Session.$anonfun$execute$3(Session.scala:597) scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) scala.util.Success.$anonfun$map$1(Try.scala:255) scala.util.Success.map(Try.scala:213) scala.concurrent.Future.$anonfun$map$1(Future.scala:292) scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:829)

data engineering
Lakehouse
Notebook

5 Replies

  • spaceman127's avatar
    spaceman127
    Super User
    11 months ago

    Hi kmvaishakha ,

     

    Creating a ticket is one option.

    But what did you do in Spark Notebook?

    Give us more information about your request as an example or what a lakehouse was created?
    A lakehouse based on Schema Lakehouse?

    We may be able to help.

     

    Best regards

  • anilgavhane's avatar
    anilgavhane
    Super User
    11 months ago

    What You Can Try

    • Run DESCRIBE TABLE schema.table
    • See if the table metadata is accessible. If this fails, the issue is likely with the table itself.
      • Use SHOW TABLES IN schema
      • Confirm the table is listed and accessible.

         

        • Try a column-specific query

           

           

           

          SELECT column1, column2 FROM schema.table

           

           

          • This can help isolate if the issue is with a specific column or schema structure.

             

            • Refresh the table metadata

               

               

               

              REFRESH TABLE schema.table

               

              1. Check Delta Table Versioning
                • If the table was created with a newer Delta version than supported in Runtime 1.3, compatibility issues may arise.
                • Review Spark Logs
                  • Look for clues in the Spark driver and executor logs. Sometimes the root cause is buried deeper than the stack trace suggests.
                  • Recreate the Table (if feasible)
                    • If the table is small or early-stage, consider recreating it with clean metadata.

  • v-prasare's avatar
    v-prasare
    Community Support
    11 months ago

    We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?
    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.
    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

    Thank you for your understanding and participation.