Another spark pool error - Application id is null

[dbeavon3]

For most of of the spark errors in Fabric, I will at least find driver logs.  Eg. the executors are allocated on the cluster.

Here is one without any logs whatsoever:

"Application id is null"

I think the bug is being thoroughly discussed on reddit as well.  As-of now there are no google hits that refer to authoritative communication from Microsoft.  I suspect they will soon step up and communicate more transparently about these spark bugs which are impacting so many of their customers.  In the meantime, I will revert to HDInsight for the rest of the day - it is is a one of the better implementations of Spark from Microsoft.

[dbeavon3]

A couple things I want to point out are:

- livy ID is available, indicating that the orchestration was initiated, but the pool did not want to begin processing. 

- The "Spark History Server" link is grayed out.  This also means the problem is with the launching of the notebook on the pool, rather than with Apache Spark's core.

- I have many, many other spark bugs at the moment, and I'm at the limit of what I'm allowed to open at Mindtree.  If someone else encounters this, please open a support ticket in the Power BI admin portal.

- the failed notebook was mid-way thru a parallel pipeline that launched about 40 other notebooks.  This was the only one that behaved abnormally.

- My theory is that the (hidden) spark pool/cluster was probably unhappy at this particular moment of time.  It was probably busy doing something that (theoretically) shouldn't imact user workloads, like performing an autoscale operation or something along those lines.

-  There is no way for customers to have direct visibility to monitor their spark pools.  We have visibility on a per-notebook basis but, for the cluster as a whole, we have no way to inspect it for problems.  This limitation will inevitably prevent us from digging deeply into Microsoft bugs, and wont allow us to find our own workarounds either.  I'm convinced that the only path forward is to open a support request (SR) with Mindtree.  I will probably do that once the severity grows to be more than two or three of these failures per week.  (Even if they agree there is a bug, I'm certain Microsoft will demand that customers should implement workarounds, like "retries".  If a failure happens, try again.  If another failure happens, try again.  Eventually the notebook will work, and Microsoft will also get to add more CU's to the meter.  It is a win/win)

[v-prasare · Accepted solution]

@dbeavon3, Hope you your doing well.

Can you confirm whether this issue is transient or persistent? also please share the Reddit link to the discussion for better visibility and reference?

Thank you for sharing your insights regarding the visibility and limitations with Spark pools and the challenges in identifying issues. Your perspective highlights an important gap that could benefit from further attention. Could you kindly formalize this as feedback, outlining the specific challenges and potential impacts that mentioned? Microsoft Fabric Ideas feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

Thanks,

Prashanth Are

MS Fabric Community Support

[dbeavon3]

Hi @v-prasare 

I had problems two days in a row.  Both days had meaningless messages.

... one day the error was:

Application id is null

... the next day the spark session started, but the stderr subsequently encountered errors and died with a different message:

Session is unable to register ReplId: default

Here is a link to others who are having the meaningless error message:

Slow/Failing Spark Startup Times : r/MicrosoftFabric

[v-prasare]

@dbeavon3,

The details you have shared along with the link to other user's discussions, will undoubtedly contribute to improving our understanding of these issues. by sharing your experience, you are helping to enhance not only our knowledge but also the overall support and experience for our community.

Thank you for taking the time to share your experience and observations regarding the error messages you've encountered. we truly appreciate you bringing this to our attention.

Thanks,

Prashanth Are

MS Fabric Community Support