User Data Function: 'NotFound' Error (WorkloadException) after Publishing, but Success in Develop

[Izyaan_sc]

I am setting up a monitoring workflow in Microsoft Fabric using Fabric Activator. I have created an alert based on a KQL query that monitors row ingestion. kql query - 

table1

| where ingestion_time() > ago(10m)

| summarize RowsIngested = count()
I am calling a User Data Function (UDF) as the action to push this data to an external Mimir endpoint via Remote Write.

@udf.function()
def testFunction(metricName: str, metricValue: float, labels: dict) -> str:
    logging.info(f"Starting Prometheus Remote Write push for: {metricName}")

    try:
        writer = RemoteWriter(
            url=MIMIR_URL,
            auth={
                "username": MIMIR_USERNAME,
                "password": MIMIR_PASSWORD
            }
        )

        if not isinstance(labels, dict):
            logging.error("Input 'labels' was not provided as a dictionary/object.")
            return "Push failed: labels must be a JSON object"

        timestamp_ms = int(datetime.datetime.now().timestamp() * 1000)

        metric_definition = labels.copy()
        metric_definition["__name__"] = metricName

        series = [
            {
                "metric": metric_definition,
                "values": [float(metricValue)],
                "timestamps": [timestamp_ms]
            }
        ]

        writer.send(series)

        logging.info(f"Successfully pushed {metricName} to Mimir.")
        return f"Metric pushed successfully at {datetime.datetime.now()}"

    except Exception as e:
        logging.error(f"Remote Write failed: {str(e)}")
        return f"Push failed: {str(e)}"

1. Develop Mode (Success)

When testing the function in Develop Mode using the Test Panel with the following parameters:

• metricName: "fabric_rows_ingested_test"

• metricValue: 111

• metricLabels: {"workspace": "dev", "table": "table1"}

Output: "Metric pushed successfully at 2026-03-23 06:53:51" Logs: Successfully pushed fabric_rows_ingested_test to Mimir.

above exposed metric can be accessed in grafana.

2. Run/Published Mode (Failure)

After clicking Publish and ensuring all libraries (prometheus-remote-writer, requests) are listed as "Published" in Library Management, I switch to Run only mode & invoke the function, 

{
  "functionName": "testFunction",
  "invocationId": "00000000-0000-0000-0000-000000000000",
  "status": "Failed",
  "errors": [
    {
      "errorCode": "WorkloadException",
      "subErrorCode": "NotFound",
      "message": "User data function: 'testFunction' invocation failed."
    }
  ]
}

Why does the Fabric backend fail to resolve the endpoint in Published mode (returning NotFound) when the same function is fully functional in the Develop sandbox?

[v-saisrao-msft]

Hi @Izyaan_sc,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

[v-saisrao-msft]

Hi @Izyaan_sc,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

[Izyaan_sc]

Hi @v-saisrao-msft, 
As mentioned in my post, I have already published the function multiple times. The function works as expected in Develop mode, but the invocation issue only persists in Run mode after publishing.

[v-saisrao-msft]

Hi @Izyaan_sc,

The function is working correctly in Develop mode, which confirms that the KQL query and Python logic are valid.

The failure in Run (Published) mode is due to Microsoft Fabric runtime limitations. In the published environment, User Data Functions run in a managed backend with restricted capabilities. Specifically, outbound network calls and certain custom dependencies are not fully supported or guaranteed to work.

Because the function attempts to push data to an external Mimir endpoint, the backend is unable to execute the request, resulting in a generic “WorkloadException: NotFound” error. This error indicates a runtime execution failure rather than a missing function.

As per Microsoft Fabric design guidance, User Data Functions should be used for lightweight, in-platform processing. For external integrations such as pushing metrics to external systems, it is recommended to use Fabric pipelines, notebooks, or external services (e.g., Azure Functions) instead of UDFs.

We recommend redesigning the solution so that Activator triggers a Fabric-native output (such as a pipeline or table), and the external call to Mimir is handled outside the UDF.

Service details and limitations - Microsoft Fabric | Microsoft Learn

Thank you.

[Izyaan_sc · Accepted solution]

Actually, connecting to the Mimir endpoint isn't the bottleneck here(tested using another udf and it worked in run mode). The issue appears to be specifically related to the UDF runtime failing to recognize the Prometheus library. I’ll follow your suggestion and move this to a Fabric Notebook, thanks.

[v-saisrao-msft]

Hi @Izyaan_sc,

Thank you for considering the suggestion to use Fabric Notebook. If you encounter any further issues, you can start a new thread in the community.

[v-saisrao-msft]

Hi @Izyaan_sc ,

Fabric User Data Functions need to be published before invocation, and only published functions are accessible in Run mode or from other services. In Develop mode, you can test functions without publishing, but Run mode uses only the deployed version. Each function is called through its public REST endpoint, requiring a published function and correct input parameters. If the function isn’t published or the request parameters are incorrect, the invocation will fail. Make sure the function is published and the parameters are correctly structured.

Quickstart - Create a Fabric User data functions item - Microsoft Fabric | Microsoft Learn

Tutorial - Invoke user data functions from a Python application - Microsoft Fabric | Microsoft Learn

Thank you.