Forum Discussion

RobertoNavarro's avatar
RobertoNavarro
Frequent Visitor
2 years ago
Solved

Problems with the AutoML process

Hi team,   I'm having an issue with the AutoML functionlity in Fabric (trial).   Several months ago (around 4 or 5), I followed the example provided on Microsoft Learn (https://learn.microsoft.co...
environment
ml model
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi RobertoNavarro,

    It seems like I forget the recently update on the spark runtime environments, they may affect all the packages usage that requires the specific environment versions.

    For this scenario, you can wait for the new version release or try to contact to the AutoML package dev team to update the compatibility version.

    Regards,

    Xiaoxin Sheng

RobertoNavarro's avatar
RobertoNavarro
Frequent Visitor
1 year ago

Hi,

 

First of all, thank you for your response.

 

Regarding your question, yes, I have tried using pip with various options—upgrading, installing, and uninstalling/reinstalling the library. I’ve also created a new environment and manually included the library, but the same result.

 

I exported and executed the notebook in a Conda environment on my laptop, and it worked perfectly. Interestingly, when I run the following commands:

 

import flaml
import inspect
 
print(inspect.getmembers(flaml, inspect.isclass)) # Lists all classes
print(inspect.getmembers(flaml, inspect.isfunction)) # Lists all functions

 

In Fabric:

[] []

 

In Conda:

[('AutoML', <class 'flaml.automl.automl.AutoML'>), ('AutoVW', <class 'flaml.onlineml.autovw.AutoVW'>), ('BlendSearch', <class 'flaml.tune.searcher.blendsearch.BlendSearch'>), ('BlendSearchTuner', <class 'flaml.tune.searcher.blendsearch.BlendSearchTuner'>), ('CFO', <class 'flaml.tune.searcher.blendsearch.CFO'>), ('FLOW2', <class 'flaml.tune.searcher.flow2.FLOW2'>), ('RandomSearch', <class 'flaml.tune.searcher.blendsearch.RandomSearch'>)]
[]

 

I'm not sure if this is helpful...

 

Another point that I mentioned in my previous message, I ran this notebook in Fabric about 4 or 5 months ago, and it worked perfectly back then.

 

Regards

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi RobertoNavarro,

    According to the result, it seems like these package havn't initialized correctly in fabric environment and Conda environment works well.

    If this works at previous, I suppose they may relate to the package update and it seems not compatible with the version that fabric supported. (normally the notebook environment library not update immediately followed the packages and it will keep use the most popular stable versions)

    Regards,

    Xiaoxin Sheng

    • RobertoNavarro's avatar
      RobertoNavarro
      Frequent Visitor
      1 year ago

      Hi,

       

      I think that I have found a possible answer, if I execute help(modules) among all the output, I have got this:

       

      ****** AutoML Prerequisites ******

      AutoML requires an Apache Spark pool (version 2.4) in your Azure Synapse Analytics workspace.

      You are using a Spark 3.x pool, please use Spark 2.x pool. For details, see: https://docs.microsoft.com/en-us/azure/synapse-analytics/machine-learning/tutorial-automl#prerequisites

       

      And in this webpage:

       

       Warning

      • Effective September 29, 2023, Azure Synapse will discontinue official support for Spark 2.4 Runtimes. Post September 29, 2023, we will not be addressing any support tickets related to Spark 2.4. There will be no release pipeline in place for bug or security fixes for Spark 2.4. Utilizing Spark 2.4 post the support cutoff date is undertaken at one's own risk. We strongly discourage its continued use due to potential security and functionality concerns.
      • As part of the deprecation process for Apache Spark 2.4, we would like to notify you that AutoML in Azure Synapse Analytics will also be deprecated. This includes both the low code interface and the APIs used to create AutoML trials through code.
      • Please note that AutoML functionality was exclusively available through the Spark 2.4 runtime.
      • For customers who wish to continue leveraging AutoML capabilities, we recommend saving your data into your Azure Data Lake Storage Gen2 (ADLSg2) account. From there, you can seamlessly access the AutoML experience through Azure Machine Learning (AzureML). Further information regarding this workaround is available here.

      So I think that this feature of AutoML cannot be use either in fabric regarding that the underlying engine is Synapse Spark Pools, is this correct?

      • RobertoNavarro's avatar
        RobertoNavarro
        Frequent Visitor
        1 year ago

        Hi Anonymous,

         

        Any comment about this? I have received a couple of e-mails about if this issue is closed, but I don't have a solution, is it necessary more time to analize the problem from your side?

         

        Regards and thanks for your support

        Roberto