topic Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions' in Data Engineering

Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

dbeavon3 — Fri, 31 Jan 2025 16:00:11 GMT

I'm helping a coworker with his pyspark notebooks in Fabric.

This morning all his notebooks failed with an error:

No module named 'azure.core.exceptions'

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - ModuleNotFoundError, Error value - No module named 'azure.core.exceptions'' :

He was loading these libraries, and internally it was causing a failure when it tried to load a Microsoft-specific module:

from darts.models import RandomForest, ExponentialSmoothing, StatsForecastAutoARIMA, TBATS

Here is the full stdout error from the driver ...

--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) Cell In[4], line 1 ----> 1 import notebookutils 3 # Personalize Session 4 from notebookutils.common.initializer import initializeLHContext File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/__init__.py:5 1 __version__ = "1.1.8" 3 __all__ = ["cognitiveService", "data", "fs", "lakehouse", "notebook", "session", "runtime", "help", "warehouse", "workspace", "fabricClient", "credentials", "PBIClient", "udf"] ----> 5 from . import cognitiveService, data, fs, lakehouse, notebook, session, warehouse, workspace, fabricClient, credentials, PBIClient, udf 7 # TODO: this line should be removed after update runner ipynb code 8 from notebookutils.visualization import displayHTML File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/cognitiveService.py:1 ----> 1 from .mssparkutils.handlers import CognitiveServcieHandler 3 cs = CognitiveServcieHandler() 5 def getEndpointAndKey(lsName): File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/mssparkutils/__init__.py:5 2 from os.path import dirname, basename, isfile, join 3 import glob ----> 5 from .handlers import RuntimeHandler 6 from ..common.logger import deprecated, print_deprecated_message 8 modules = glob.glob(join(dirname(__file__), "*.py")) File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/mssparkutils/handlers/__init__.py:1 ----> 1 from .fsHandler import SynapseFSHandler 2 from .notebookHandler import SynapseNotebookHandler 3 from .runtimeHandler import RuntimeHandler File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/mssparkutils/handlers/fsHandler.py:2 1 from six import string_types ----> 2 from notebookutils.visualization import displayHTML, display_mount_points 3 from notebookutils.common.logger import log4jLogger 5 from .baseHandler import SynapseBaseHandler File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/visualization/__init__.py:1 ----> 1 from .display import display, display_mount_points 2 from .displayHTML import displayHTML 3 from .msInlinePlotlib import enable_msinline_backend as enableMatplotlib File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/visualization/display.py:17 14 from notebookutils.ipythoninterpreter import is_ipython_enabled 15 from notebookutils.ipython import runtime ---> 17 from notebookutils.visualization.display_jupyter import display_without_spark 18 from notebookutils.visualization.constants import SYNAPSE_DISPLAY_WIDGET_TYPE_KEY, MAX_ROW_COUNT 19 from notebookutils.visualization.utils import sparkContextHelper File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/visualization/display_jupyter.py:11 8 from notebookutils.common.logger import log4jLogger 9 from notebookutils.visualization.constants import SYNAPSE_DISPLAY_WIDGET_TYPE_KEY, \ 10 MAX_ROW_COUNT, MAX_CONTENT_LENGTH ---> 11 from notebookutils.visualization.dataWrangler import get_wrangler_display_entry_context, WRANGLER_ENTRY_CONTEXT_KEY 13 # Map to the unified type set that will be consumed by client side 14 # We use Spark data type .simpleString() as the unified type set 15 # Some of the types are not common Pandas dtype, like boolean/uint64. But it may appear in some DataFrame generated 16 # by 3rd party library like sempy. 17 _pandas_type_mapping = { 18 'datetime64[ns]': 'timestamp', 19 'int8': 'tinyint', (...) 31 'boolean': 'boolean' 32 } File ~/cluster-env/clonedenv/lib/python3.11/site-packages/notebookutils/visualization/dataWrangler.py:3 1 from notebookutils.common.logger import log4jLogger 2 import sys ----> 3 import pandas as pd 4 import pyspark 6 WRANGLER_ENTRY_CONTEXT_KEY = "wranglerEntryContext" File ~/cluster-env/clonedenv/lib/python3.11/site-packages/pandas/__init__.py:236 195 __doc__ = """ 196 pandas - a powerful data analysis and manipulation library for Python 197 ===================================================================== (...) 232 conversion, moving window statistics, date shifting and lagging. 233 """ 235 from fsspec.registry import register_implementation --> 236 from fsspec_wrapper.trident.core import OnelakeFileSystem 237 register_implementation('abfs', OnelakeFileSystem, clobber=True) 238 register_implementation('abfss', OnelakeFileSystem, clobber=True) File ~/cluster-env/clonedenv/lib/python3.11/site-packages/fsspec_wrapper/__init__.py:1 ----> 1 from .core import AzureBlobFileSystem 2 from .trident.core import OnelakeFileSystem 3 from .version import VERSION as __version__ # noqa File ~/cluster-env/clonedenv/lib/python3.11/site-packages/fsspec_wrapper/core.py:5 2 import time 3 from urllib.parse import urlsplit ----> 5 import adlfs 6 from fsspec.utils import infer_storage_options 8 from .utils import logger as synapseml_pandas_logger File ~/cluster-env/clonedenv/lib/python3.11/site-packages/adlfs/__init__.py:2 1 from .gen1 import AzureDatalakeFileSystem ----> 2 from .spec import AzureBlobFile, AzureBlobFileSystem 4 __all__ = ["AzureBlobFileSystem", "AzureBlobFile", "AzureDatalakeFileSystem"] 6 try: File ~/cluster-env/clonedenv/lib/python3.11/site-packages/adlfs/spec.py:20 17 from glob import has_magic 18 from typing import Optional, Tuple ---> 20 from azure.core.exceptions import ( 21 HttpResponseError, 22 ResourceExistsError, 23 ResourceNotFoundError, 24 ) 25 from azure.storage.blob import ( 26 BlobBlock, 27 BlobProperties, (...) 30 generate_blob_sas, 31 ) 32 from azure.storage.blob.aio import BlobPrefix ModuleNotFoundError: No module named 'azure.core.exceptions'

I have too many cases open with the Fabric Spark team already, and hoping to avoid another. Has anyone seen this before? Is it related to using custom pools? Is it specific to our azure region (north central)? Why would this happen today without any warning? What are other customers doing when they encounter these issues? Does everyone just repeat their workloads 2 or 3 times a day to accommodate these reliability problems in Fabric? Is there any way for a customer to investigate the logs on the cluster nodes themselves? I'm guessing there is something wrong on the VM itself.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

dbeavon3 — Fri, 31 Jan 2025 17:30:53 GMT

Sounds like the fix is to update the version of azure-core library in Fabric. Odd that customers would be doing this independently of Microsoft's direction.

I hope it is still a supported configuration. We don't want to risk even more problems, and get even less support for our customized environment.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

jflo08 — Mon, 03 Feb 2025 10:11:39 GMT

Did you find a solution to this? I have just had the exact same error, regarding the same package "notebookutils". Something must be happening with updates since we are both getting the same error within the same week.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

v-saisrao-msft — Mon, 03 Feb 2025 12:40:13 GMT

Hi @dbeavon3,

Thank you for reaching out to the Microsoft Forum Community.

We greatly appreciate your efforts and are glad to hear that your issue has been resolved while updating the version of the azure-core library in Fabric.

Please accept your response as a solution, as this will help other community members can resolve similar issues more efficiently.

Thank you.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

dbeavon3 — Mon, 03 Feb 2025 14:18:38 GMT

Hi @v-saisrao-msft

Are you a python guy? I think you would agree that customers shouldn't be tinkering with the default version of azure-core in the Fabric runtime.

Do you know where we can find out about sudden breaking changes like this one, that would impact the reliability of our scripts?

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

jflo08 — Mon, 03 Feb 2025 15:58:06 GMT

Hi,

I have been troubleshooting this issue today, as I experienced it whilst running my custom packages in my custom Fabric environment. I ended up creating a new environment and testing in a notebook to see if the issues persisted, miraculously this actually resolved the issue. All the built-in libraries were the same, and my custom packages also work in this new environment.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

v-saisrao-msft — Fri, 07 Feb 2025 10:31:12 GMT

Hi @dbeavon3,

As a member of the Microsoft Fabric Community Support team, I can refer you to the following links that will help address your query regarding sudden breaking changes that could affect the reliability of the scripts.

Troubleshoot Python function apps in Azure Functions | Microsoft Learn
Issues · Azure/azure-functions-python-worker

Additionally, please review the Idea Forum where you can contribute your own ideas and suggestions on any topic. These will be reviewed by Microsoft and may be implemented in future updates.

Home

Thank you.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

v-saisrao-msft — Thu, 13 Feb 2025 10:48:07 GMT

Hi @dbeavon3,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

v-saisrao-msft — Tue, 18 Feb 2025 06:28:42 GMT

Hi @dbeavon3,

I would like to confirm if the issue has been resolved on your end. If so, kindly mark the helpful reply and accept it as the solution. This will assist other community members in resolving similar issues more quickly.

Thank you

Re: Another random spark failure in fabric on custom pool - No module named 'azure.core.exceptions'

v-saisrao-msft — Sat, 22 Feb 2025 16:07:17 GMT

Hi @dbeavon3,

Thank you