The Spark SQL phase analysis failed with an internal error

[kmvaishakha]

Environment: Runtime 1.3 (Spark 3.5 , delta 3.2)
I am getting this error when i do Select * from schema.table
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. org.apache.spark.SparkException$.internalError(SparkException.scala:107) org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:800) org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:812) org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:440) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:439) org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:174) org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:163) org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:114) org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:98) org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:752) org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:961) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:743) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:774) org.apache.spark.sql.SparkSession.sql(SparkSession.scala:805) org.apache.livy.repl.SQLInterpreter.execute(SQLInterpreter.scala:163) org.apache.livy.repl.Session.$anonfun$executeCode$1(Session.scala:909) scala.Option.map(Option.scala:230) org.apache.livy.repl.Session.executeCode(Session.scala:906) org.apache.livy.repl.Session.$anonfun$execute$17(Session.scala:597) org.apache.livy.repl.Session.withRealtimeOutputSupport(Session.scala:1169) org.apache.livy.repl.Session.$anonfun$execute$3(Session.scala:597) scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) scala.util.Success.$anonfun$map$1(Try.scala:255) scala.util.Success.map(Try.scala:213) scala.concurrent.Future.$anonfun$map$1(Future.scala:292) scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:829)

[v-prasare]

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?
If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.
Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

[v-prasare]

Hi @kmvaishakha ,
If your issue still persists, please consider raising a support ticket for further assistance.
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

thanks,

Prashanth Are

[anilgavhane]

What You Can Try

• Run DESCRIBE TABLE schema.table
• See if the table metadata is accessible. If this fails, the issue is likely with the table itself.
  • Use SHOW TABLES IN schema
  • Confirm the table is listed and accessible.
    • Try a column-specific query

      SELECT column1, column2 FROM schema.table

      • This can help isolate if the issue is with a specific column or schema structure.
        • Refresh the table metadata

          REFRESH TABLE schema.table

          1. Check Delta Table Versioning
             • If the table was created with a newer Delta version than supported in Runtime 1.3, compatibility issues may arise.
             • Review Spark Logs
               • Look for clues in the Spark driver and executor logs. Sometimes the root cause is buried deeper than the stack trace suggests.
               • Recreate the Table (if feasible)
                 • If the table is small or early-stage, consider recreating it with clean metadata.

[spaceman127]

Hi @kmvaishakha ,

Creating a ticket is one option.

But what did you do in Spark Notebook?

Give us more information about your request as an example or what a lakehouse was created?
A lakehouse based on Schema Lakehouse?

We may be able to help.

Best regards

[tayloramy]

Hi @kmvaishakha, 
This looks like a platform bug. I'd suggest opening a ticket with Microsoft so one of their engineers can investigate this. 

You can submit a ticket here: https://support.fabric.microsoft.com/support/

Did I answer your question? Mark my post as a solution and give some Kudos!