Forum Discussion

JosueMolina's avatar
JosueMolina
Icon for Helper III rankHelper III
2 years ago

Environment with Runtime 1.3 fails when uploading custom library

I began experiencing issues when loading a custom library into our Environment yesterday. 

 

I have other environments in different workspaces with the same library working fine. 

I found this error which I believe is related: Known issue - Fabric Runtime 1.3 causes invalid libraries - Microsoft Fabric | Microsoft Learn but I do not get an error when starting a Spark session in a notebook, just when trying to upload the library in new environment. 

 

I am able to see this sort of error : 

Pip subprocess error:
WARNING: Error parsing dependencies of notebookutils: Invalid version: '3.5.0_20240224.2'

Which I believe has something to do with an update in the backend, either with Runtime 1.3 or Spark 3.5 and versioning.

 

Anyone have any idea what to do? I am testing downgrading this environment to Fabric Runtime 1.2 in the meantime.

data engineering
environment

7 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi JosueMolina 

     

    It is possible that not all environments where custom libraries are installed will get an error when running a notebook. However, it cannot be ruled out that the failure to install custom libraries in a new environment has nothing to do with this known issue.

     

    From the error message you got, it seems it failed to parse dependencies of notebookutils. While notebookutils is a built-in package in the Runtime. So it seems a Runtime issue. 

     

    Have you finished testing downgrading this environment to Fabric Runtime 1.2? Did it work well with Fabric Runtime 1.2?

     

    Best Regards,
    Jing

    • JosueMolina's avatar
      JosueMolina
      Icon for Helper III rankHelper III
      2 years ago

      Downgrading to Runtime 1.2 does work with the same custom library.

      But seems like a bad solution as the regular Workspace environment is Runtime 1.3 and we would be losing many improvements in our Data Flow by downgrading.

      I have other Environment running Runtime 1.3 and the same library, but those were set up last month and those work fine.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi JosueMolina 

         

        If there is another environment running Runtime 1.3 and the same library working fine in the same workspace, you may set that environment as the default environment of the workspace in Worksapce Settings. 

        If there is no other environment working fine in the same workspace, you may still use Runtime 1.3 for the workspace environment, but create a custom environment running on Runtime 1.2 and upload the custom library. For some specific notebooks which uses the custom library, run them on the custom Runtime 1.2 environment. Run the other notebooks on the workspace default Runtime 1.3 environment. This may be a temporary alternative. 

         

        Feel free to let me know if you have any question. 

         

        Best Regards,
        Jing