user data functions
57 TopicsFabric UDF works in Test mode but fails when invoked on Run mode
Hi guys! I'm experiencing an issue with a Fabric User Data Function that works perfectly in the UDF Test experience, but consistently fails when invoked from a Power BI Report using a Translytical Task Flow or on Run mode. Scenario The function receives several parameters from the report and writes records into an Azure PostgreSQL database using psycopg2. The code executes successfully when I run it directly from the Fabric User Data Functions test interface. However, when the same function is triggered from a Power BI report or on Run mode, the invocation fails before I can identify the root cause. import datetime import fabric.functions as fn import logging import psycopg2 udf = fn.UserDataFunctions() @udf.function() def StatusAENovosRegistros( aeIdListaStr: str, produto: str, recorte: str, status: str, justificativa: str, comentario: str, nomeUsuario: str, economia: int, emailUsuario: str ) -> str: conn = psycopg2.connect( database="xxx-xxx-xxx", user="user", host="xxx-xxx-xxx-xxx", password="""xxxyyyvvv""", port=5432 ) cur = conn.cursor() aeIdLista = aeIdListaStr.split(",") for areaExpansao in aeIdLista: dados = ( areaExpansao, produto, recorte, status, justificativa, comentario, nomeUsuario, economia, emailUsuario ) cur.execute( """ INSERT INTO "SCHEMA".table ( ae_id, produto, recorte, status, justificativa, comentario, nome_usuario, economia, email_usuario ) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s) """, dados ) conn.commit() cur.close() conn.close() return "Dados inseridos com sucesso" I always get the message below: { "functionName": "StatusAENovosRegistros", "invocationId": "00000000-0000-0000-0000-000000000000", "status": "Failed", "errors": [ { "errorCode": "WorkloadException", "subErrorCode": "NotFound", "message": "User data function: 'StatusAENovosRegistros' invocation failed." } ] }137Views0likes3CommentsScheduling a User Data Functions item directly fails every time with "FunctionName is null or empty"
Hi all, I'm running into an issue with the native Schedule option on a User Data Functions item (not the Data Pipeline Functions activity, the Schedule button on the UDF item itself). Every single scheduled run fails with: [{"errorCode":"WorkloadException","subErrorCode":"WorkloadException","message":"Invalid Argument: FunctionName. FunctionName is null or empty. Please provide a valid value"}] At first I thought this was because I had more than one function in the item and the scheduler didn't know which one to trigger, but I trimmed the item down to a single function and it still fails the exact same way. So it doesn't seem to be a "multiple functions" thing. What I did: Created a UDF item with one function, published it, tested it manually - works fine, returns the expected output. Set up a Schedule on the item and let it run for a while. Checked Monitor - every run failed with the error above. In the Monitor run details, both "Function name" and "Invocation ID" show up blank (--), so it really looks like the scheduled trigger just isn't passing a function name at all when it kicks off the run. Things I already tried: Deleted and recreated the schedule Reduced the udf to a single function Manual run of the function: works every time Calling the same function from a Data Pipeline Functions activity (where you explicitly pick the function in the activity settings) and scheduling the pipeline instead: works fine, no problems at all So the function itself is clearly not the problem, it's specifically the item-level Schedule that seems broken - it's just not sending FunctionName in the invocation request the way the Pipeline activity does. Anyone else seen this, or know of a fix/workaround? Happy to share invocation IDs or more logs if it helps.119Views2likes3CommentsFabric Data Factory (REST API): 3 connection blockers migrating a parameterized ADF
Migrating an ADF ingestion factory (~219 pipelines, 36 linked services) to Fabric Data Factory programmatically via the REST API. Three connection issues I can't resolve through the API β looking for the supported path or confirmation they're UI/manual-only: whats is the best apprach to migrate adf pipleins with having parametarized linked services call 1. Connector types not creatable via the Connections REST API. POST /v1/connections returns "Kind: AzureDatabricks is not supported" and "Kind: Ftp is not supported." Which connector kinds does the public Connections API support, and for the rest (Databricks, FTP, β¦) is the UI the only option, or is there an automation/API path? 2. ADLS Gen2 + Workspace Identity β DMTS_UntrustedEndpointForWorkspaceIdentity. Creating an AzureDataLakeStorage connection with Workspace Identity fails as an untrusted endpoint. What's required to create it via API end-to-end β does the storage account need trusted-workspace-access / the workspace identity pre-granted? Is there an API to establish that trust? 3. Parameterized linked services with per-call dynamic endpoints. Several ADF REST linked services use url = @{linkedService().absolute_url} β the URL is passed by each dataset at runtime, so one linked service calls many URLs. Fabric connections need a fixed endpoint. What's the recommended pattern β one connection per host + dynamic relative URL? Parameterize the connection field by GUID via a Variable Library? Any way to represent a fully dynamic-URL connection? Environment: Fabric Trial capacity; connections created via a Service Principal. Official docs/links appreciated.Solved569Views0likes6CommentsA Typo when collapsed the left panel in UDF (Korean)
Hi Team, When I collapsed left explorer window in the UDF, I noticed the title was upside down. That should be rotated as follows: I will submit a support ticket to support engineer team. https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket Korean English Thanks, Hong244Views1like3Commentscreate connections with api calls same as adf to fabric ui migration utility do
I'm building an automated ADF β Fabric pipeline migration tool using the REST API. I successfully: Parse ADF ARM templates and extract linked services Create Fabric connections via POST /v1/connections using a Service Principal (application token / client_credentials flow) Deploy 7 pipelines via POST /workspaces/{id}/items with inline datasetSettings and embedded connection GUIDs All pipelines deployed successfully (200 OK) The problem: The connections created by the SPN are NOT visible in the personal "Manage Connections and Gateways" page for any human user. The deployed pipelines show "Failed to load the connection" in Fabric Studio because the connections are owned by the SPN, not a user account. What the UI Migration Utility does differently: When you use the Fabric UI migration tool, the created connections appear correctly in the Connections page and can be edited/tested/deleted. My assumption is that the UI tool uses a delegated OAuth token (user context) instead of an application token, so the connection is owned by the user. My questions: Is it possible to call POST /v1/connections with a delegated token (device_code flow) to create user-owned connections programmatically β same result as the UI utility? Is there a way to transfer ownership of an SPN-created connection to a user via API? Is there a different API endpoint the UI migration utility uses to create connections that doesn't have this ownership restriction? For enterprise migrations (100+ linked services), what is the recommended approach to create connections programmatically while keeping them user-visible and manageable in the UI?Solved632Views1like10CommentsProduction-scale ADF β Fabric migration: REST API approach vs Microsoft Migration Utility
I'm automating Azure Data Factory β Microsoft Fabric Data Factory migration using ARM export + Fabric REST APIs instead of the Microsoft migration UI. Current approach: Parse ARM templates (pipelines, datasets, linked services, etc.) using a custom parser. Create Fabric resources via REST APIs. i am building a reusable ADFβFabric migration tool (REST-API based, headless) and testing it on a deliberately complex 219-pipeline factory. Our assessment classifies each pipeline Ready / Needs review / Coming soon / Not compatible. Most land in "Needs review" for two reasons: (1) heavy use of Execute Pipeline (parent/child) which Fabric replaces with the Invoke Pipeline activity, and (2) connections bound to a self-hosted integration runtime, which Fabric needs an On-Premises Data Gateway for. We resolve the IR type from the ARM template (connectVia β runtime.type = SelfHosted) to flag on-prem precisely. Question: what's the most efficient pattern to (a) auto-convert ExecuteβInvoke with correct child-first deploy ordering, and (b) provision gateway-bound connections at scale once a gateway exists?Solved499Views0likes5CommentsFabric connections appear to require a static endpoint
We're migrating Azure Data Factory (ADF) pipelines to Microsoft Fabric Data Factory using the REST API. Many of our ADF linked services are parameterized. For example, an Azure SQL linked service uses a server name like: @{concat('company-sqlsrv-', linkedService().environment, '-database.database.windows.net')} where environment is provided at runtime (for example, dev, test, or prod) through a global parameter. We also have REST and ADLS linked services where the endpoint or storage path is supplied dynamically (for example, linkedService().baseUrl or a storage path constructed from runtime parameters such as {layer}/{source}). Fabric connections appear to require a static endpoint, so these don't seem to map directly. I have a few questions: Is there a supported way to create a Fabric connection that remains parameterized like an ADF linked service, or is creating one connection per concrete endpoint the only supported approach? For "generic" linked services that are reused with many runtime URLs or storage locations, what is the recommended design pattern in Microsoft Fabric? When calling POST /v1/connections, what is the correct credentialDetails payload for creating an Azure SQL connection through an On-Premises Data Gateway? We're receiving a 400 InvalidInput response with the message: "The Values field is required." Additionally, WorkspaceIdentity appears to be rejected for gateway-bound connections. If anyone has documentation, sample payloads, or recommended migration patterns for this scenario, I'd really appreciate the guidance.390Views0likes3CommentsA Typo When collapsing the explorer menu in the User Data Function (Korean)
Hi Team, When I collapsed the left explorer menu, I noticed that the text was displayed upside down. Korean English The text should be displayed straight, like the following. ν¨ μ ν μ κΈ° (Collapsed the menu in the Lakehouse) Thanks, HongSolved663Views0likes4CommentsData Engineering Template
Does anyone use Excel/PowerQuery for Data Cleansing/Engineering? An Excel template I developed could be of interest. There are only three buttons. The IQR Process button runs a process that flags your a column of data for unusually low or high statistical outliers using the IQR Statistical Rule. The K-means Clustering button lets you run a simple K-means Clustering process on the same original column of data. The Start Over button deletes Columns B and C created by the two buttons above. The pop-up messages and 3 pivot tables let data analysts compare the results from the two processes and provide a means to filter extremes in their data when creating Power BI and other dashboards so KPIs are not skewed in reporting.380Views1like4Comments