AI functions with PySpark error

[Mauro89]

Hi there,

I have tried the AI functions capabilities in notebooks. The example with pandas dataframe worked for me. But the example as of the MS documentation with PySpark does not work.

Example from docs:

# This code uses AI. Always review output for mistakes.

df = spark.createDataFrame([
("Scarves",),
("Snow pants",),
("Ski goggles",)
], ["product"])

responses = df.ai.generate_response(prompt="Write a short, punchy email subject line for a winter sale.", output_col="response")
display(responses)

Error message is:

AttributeError: 'DataFrame' object has no attribute 'ai'

Anyone a idea about it?

Thanks!

PS: yes, openai lib is installed and admin setting is enabled, thats why it works with pandas df's 😉

[v-karpurapud]

Hi @Mauro89

Could you let us know if the issue was resolved after creating the support case?If you have any further questions, please let us know. we can assist you further.

Regards,

Microsoft Fabric Community Support Team

[Mauro89]

Hi @v-karpurapud,

we are still on it, trying to figure out whats the issue.

I will keep the thread posted 🙂

Best regards!

[v-karpurapud]

Hi @Mauro89 

Thank You for the response.

[Mauro89 · Accepted solution]

@v-karpurapud sooooo. I turned out to be the issue was sitting infront of the screen not in the screen 😅

I was only importing the libary for pandas and was not award that the pyspark library is a different one. So for all ready this post, the solution is to import the correct relevant library.

import synapse.ml.aifunc as aifunc --> for Pandas

import synapse.ml.spark.aifunc as aifunc --> for PySpark

Nevertheless, thanks for the support here in this thread!

@v-karpurapud may I can take my own answer as solution?

[v-karpurapud]

Hi @Mauro89 

Glad to hear, you found the solution. Yeah, You can mark your own post as Solution.

[ABD128]

Hi @Mauro89 

The error happens because your notebook isn’t running with the Spark AI features that Microsoft Fabric requires for the .ai accessor to work on PySpark DataFrames. Even though pandas AI functions run fine, the Spark version only works when your notebook uses Fabric Runtime 1.2 or later and when your workspace has AI in Notebooks and AI Spark Extensions enabled. If either of these is missing, Spark won’t load the AI extension, which is why you see the AttributeError. To fix it, switch your notebook to Fabric Runtime 1.2+, enable the AI features in the Admin Portal, and restart the kernel. After that, the PySpark example from the Fabric docs will work as expected. If Spark AI still isn’t available in your environment, you can temporarily call it using the AISpark helper class until the extension is enabled.

[Mauro89]

Hi @ABD128,

runtime is 1.3.

Do you mean this setting in the admin portal? If yes, it is enabled:

Any other advise?

[v-karpurapud]

Hi @Mauro89 
Thank you for contacting the Microsoft Fabric community forum.
 

The setting shown in your screenshot enables Copilot/OpenAI features for the whole tenant, but it doesn't ensure that the Spark AI extension is active in your notebook session. The .ai accessor comes from the Spark AI integration, which relies on the AI Spark Extensions being enabled on the specific capacity where your workspace is running.
 

Even with Runtime 1.3, the extension might not load if the environment kernel starts without the AI Spark package or if the capacity-level AI features for “AI in notebooks / Spark AI Extensions” are not enabled.To check this, you can run the following in your notebook:

hasattr(spark.createDataFrame([("x",)], ["col"]), "ai")

If this returns False, the Spark AI extension did not load. If your capacity settings are correct but it still doesn’t load, it could be a deployment issue in that case, you should contact Fabric support, since .ai should be available on Runtime 1.3.

As a workaround, you can use the helper class:

from fabricext.ai import AISpark

I hope this information is helpful. If you have any further questions, please let us know. we can assist you further.

Regards,

Microsoft Fabric Community Support Team.
 

[Mauro89]

Hi @v-karpurapud,

Iam still not sure to which extension you relate (AI Spark Extensions being enabled on the specific capacity)?
Is it in the admin portal or where can I find it? As Iam Admin in the Fabric environment as well as capacity I assume I can enable it once found.

And the workaround throughs the follwoing error:
No module named 'fabricext'

Thanks and best regards!

[v-karpurapud]

Hi @Mauro89 

To clarify, when I mentioned “AI Spark Extensions being enabled on the capacity,” I did not mean there is a manual setting in the Admin Portal, as such an option does not exist. The Spark AI layer is included, but Microsoft activates it automatically during the Fabric Runtime rollout, and its availability may differ based on capacity or region.

In this context, “enabled” signifies that the Spark AI components are present in the capacity’s runtime and accessible to notebooks. In your case, the absence of both df.ai and the fabricext module suggests that the Spark AI package is not yet available on your capacity, despite being on Runtime 1.3 with tenant-level Copilot/OpenAI enabled.

Since this extension is generally expected to be available, we recommend submitting a support ticket so the Fabric team can review your capacity and assist further.

You can submit a ticket via the Microsoft Power BI Support Portal:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Regards,

Microsoft Fabric Community Support Team.

[Mauro89]

Thanks @v-karpurapud for clarification.

I have created a ticket and will keep this thread posted about the solution.

[v-karpurapud]

Hi @Mauro89 

Thanks for the update. Hope your issue gets resolved soon. when it does, please share the insights with the community as it can be helpful to others.