Fabric environment + public library not working

[gregahren]

I am trying to create fabric environmet which would contain pydantic==2.5.2 pypi package, so I can make use of the library in the notebook or in my spark jobs:

from pydantic import RootModel

Note that RootModel is only available in pydantic version 2.5.2

The result is:

If I do %pip list I see that pydantiv v1 is still installed:

Here is also screen of how my environment is set up:

[HimanshuS-msft]

Hello @gregahren 
Thanks for using the Fabric community.

Are you hitting this error : [Bug]: cannot import name 'RootModel' from 'pydantic' · Issue #1237 · binary-husky/gpt_academic · Gi...
As mentioned in the issue , can you please try 
gradio==3.48.0

and see if that fixes the problem . 
Thanks 
Himanshu 

Thanks
HImanshu

[gregahren]

Hi,

thank you for the response.

Actually I dont want to install gradio into my environment since that is not neccessary, also that introduces a lot of uneccessary packages to be install into environment as a direct dependencies of gradio package.

I can do %pip install pydantic==2.5.2 directly in notebook and that installs library as expected. But what if I want to use it in my spark job where magic commands are not available?

Thanks

[v-gchenna-msft]

Hi @gregahren ,

In current design, for PySpark, beside the main execution file, user can also upload other python module as .py file into the SJD as additional reference. But .whl package is not supported yet. 

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

[gregahren]

Hi @v-gchenna-msft ,

I appreciate your prompt response.

While I understand the suggestion to share this in the feedback channel and submit it as an idea, I believe the issue at hand is more aligned with a bug rather than a feature.

Allow me to reiterate: I am attempting to install version 2.5.3 of the Pydantic library from PyPI, as illustrated in the provided image:

However, when I utilize this created environment in a notebook or Spark job, the desired package is unavailable. Instead, I observe the presence of Pydantic version 1, contrary to my explicit specification of version 2.5.3 in the environment.

I hope this clarifies the situation. Your assistance in resolving this matter would be greatly appreciated.

Thank you.

[v-gchenna-msft]

Hi @gregahren ,

When I tried to repro your scenario i didn't face any issue like you mentioned.

Sample Environment -

Sample Notebook -

If you are still the facing, can you please try to recreate new environment and try again.

Hope this is helpful. Please let me know incase of further queries.

[Anonymous]

I have the same issue. Has there been found a solution?

[gregahren]

Hi @Anonymous 

there is still an active ticket opened and PG team is still in the process to find the root cause of the issue.

Will let you know after I recieve more info.

[Anonymous]

thank you, i would appreciate that very much.

[gregahren]

If you need a quick and dirty solution to still be able to install pydantic and use notebook in pipelines or similar we found a workaround which is not pretty but it works for us for now:

import os
from notebookutils import mssparkutils

os.system("pip install pydantic==2.6.0")
mssparkutils.session.restartPython()

Still we expect this to not be neccessary if this is specified already in environment.

[Anonymous]

my issue is with numpy = 1.25 and tensorflow = 2.11 instead of 1.23.5 and 2.12 respectively. But I will check if I can use your solution until they fix it. Thank you.

[gregahren]

Im sorry but still I am facing the same issue:

I also have a ticket opened with support so I will update you if they figure it out.

[v-gchenna-msft]

Hi @gregahren ,

Thanks for the reply.
Can you please provide the support ticket number as it would help us to track for more information?

[gregahren]

Sure here it is: #2401220050004122

[gregahren]

I will retry it again and let you know. Thanks.