Forum Discussion
dbt job pipeline activity: bug
Hi everone,
The operation (like run, build and compile, under "Advanced Settings") of a dbt job activity in a pipeline is not being pushed to the repo when doing a commit. As a result, my fabric-cicd deployments to other workspaces, don't include this setting. Running the pipeline results then in an error, linked to the missing "Operation" value.
I can add the setting directly in the pipeline code ("typeProperties": {"operation": "build"}), but Fabric automatically proposes a commit that removes this line again.
Can this be solved please?
Kind regards,
Michael
I created a support ticket and have the following update to share: Microsoft support has excalated this issue and a fix will be implemented:
"We have done further tests on this internally and were able to reproduce the behavior. Then escalated it internally to our Product team. As a result, this will be addressed and the current tentative for the fix is in September.
If things go as planned, it should be during the first week of September but deployment can vary depending on different regions."Please check the reply of ryan-schofield for a temporary workaround.
12 Replies
- ryan-schofieldAdvocate II
Since I wasn't able to wait for this bug to be resolved. I worked around the problem by scheduling the dbt job item via the API and then polling until complete in a notebook.
The full notebook is too large to share here, but here is a basic example of how it works. Note that some of this leverages the API in a way that is not documented yet, but it appears to work fine.import time from urllib.parse import unquote, urlparse import requests FABRIC_API_BASE_URL = "https://api.fabric.microsoft.com/v1" WORKSPACE_ID = "<workspace-id>" DBT_ITEM_ID = "<dbt-project-item-id>" ACCESS_TOKEN = "<bearer-token>" # acquire via your identity provider (e.g. MSAL / service principal) # The command to execute: dbt operation + arguments (mirrors `dbt run --select ...`) execution_data = { "operation": "run", "arguments": { "select": "tag:daily", "fullRefresh": False, "threads": 4, }, } headers = { "Authorization": f"Bearer {ACCESS_TOKEN}", "Content-Type": "application/json", } # 1. Submit the job submit_url = f"{FABRIC_API_BASE_URL}/workspaces/{WORKSPACE_ID}/items/{DBT_ITEM_ID}/jobs/Execute/instances" response = requests.post(submit_url, headers=headers, json={"executionData": execution_data}) response.raise_for_status() # Fabric returns the job instance location (may be relative or absolute) location = response.headers["Location"] job_instance_id = unquote(urlparse(location).path.rstrip("/").split("/")[-1]) # 2. Poll the job instance until it reaches a terminal state instance_url = f"{FABRIC_API_BASE_URL}/workspaces/{WORKSPACE_ID}/items/{DBT_ITEM_ID}/jobs/instances/{job_instance_id}" while True: status_response = requests.get(instance_url, headers=headers) status_response.raise_for_status() status = status_response.json().get("status", "").upper() if status in {"COMPLETED", "SUCCEEDED"}: print("dbt job succeeded") break if status in {"FAILED", "CANCELLED"}: raise RuntimeError(f"dbt job failed with status: {status}") time.sleep(20) # poll interval - michaelfabricFrequent Visitor
I created a support ticket and have the following update to share: Microsoft support has excalated this issue and a fix will be implemented:
"We have done further tests on this internally and were able to reproduce the behavior. Then escalated it internally to our Product team. As a result, this will be addressed and the current tentative for the fix is in September.
If things go as planned, it should be during the first week of September but deployment can vary depending on different regions."Please check the reply of ryan-schofield for a temporary workaround.
- sannavajjalaResolver II
Hi,
Hi Michael,
This sounds like a product limitation or serialization issue with the current Fabric Git integration rather than something you're doing incorrectly. If the Operation setting (Run, Build, Compile) is configured in the UI but is not persisted in the repository artifacts, deployments through Fabric CI/CD can result in an invalid activity definition in the target workspace.
The fact that manually adding "operation": "build" to the pipeline JSON resolves the runtime error, but Fabric subsequently removes it during the next commit, suggests the Git representation of the dbt activity is not fully round-tripping all properties. In that case, the source of truth in the UI and the source-controlled artifact are becoming inconsistent.
A few things I'd recommend checking:
- Verify all environments are running the same Fabric feature set and Git integration experience.
- Confirm whether the issue occurs with newly created dbt activities or only existing ones.
- Compare the JSON generated before and after changing the Operation value to see whether any related properties are being persisted.
- Test deploying the pipeline directly between workspaces without Git to determine whether the issue is specifically tied to source control serialization.
If the behavior is reproducible, it is likely a bug and worth raising with Microsoft Support. Be sure to include:
- The pipeline JSON stored in Git
- Screenshots of the Operation setting in the UI
- Deployment logs showing the missing Operation error
- The Fabric workspace and Git integration experience being used
In the meantime, if Fabric continues to remove the property during commits, any manual JSON edits are unlikely to be a sustainable workaround because the platform will overwrite them during synchronization. Hopefully others in the community can confirm whether they're seeing the same behavior with dbt activities and Git-managed deployments.
Thanks,
Manoj Annavajjala
- ryan-schofieldAdvocate II
I'm running into the same issue. Here's a rundown:
Fabric Git integration drops the `operation` property from `InvokeDataBuildToolJob` (dbt job) activities on commit
Summary
When a data pipeline containing a dbt job activity (`InvokeDataBuildToolJob`) is committed to a Git-connected workspace, the Fabric Git integration silently removes the `operation` property from `typeProperties`. The property is not substituted with a logical ID or placeholder like `workspaceId` and `dataBuildToolJobId` are; it is simply dropped from the serialized definition entirely.Because the drop is silent (no error, no warning during commit or merge), the pipeline definition in the repo is incomplete. On deployment/sync to another environment, the activity has no `operation` value and the dbt job cannot run as authored.This looks like a serialization gap in the Git integration for the `InvokeDataBuildToolJob` activity type.Environment
- Feature: dbt job activity in Fabric Data Factory pipeline (Preview)
- Activity type: `InvokeDataBuildToolJob`
- Source control: Fabric Git integration (Azure DevOps)
Steps to Reproduce
- Create a data pipeline in a Git-connected workspace.
- Add a dbt job activity (`InvokeDataBuildToolJob`).
- Set the operation field.
- Set `select` and `threads`, and configure the connection.
- Save the pipeline and commit the workspace to Git from the Fabric UI.
- Inspect the committed pipeline definition in the repo.
Expected: The `operation` property is serialized to the repo (as the expression/value that was authored), consistent with the documented support for dynamic content on all dbt job activity settings.Actual: The `operation` property is missing from `typeProperties` in the committed definition.Example
Activity as authored in the workspace (correct)
{"name": "RefreshEpmSeeds","type": "InvokeDataBuildToolJob","dependsOn": [],"policy": {"timeout": "0.12:00:00","retry": 0,"retryIntervalInSeconds": 30,"secureOutput": false,"secureInput": false},"typeProperties": {"dataBuildToolJobId": "5cbc04b3-f9ba-451d-a7ce-ab904a68d0e1","workspaceId": "71b4b3df-cf7c-43a3-b170-0f87ca8f84f9","operation": "seed","select": "epm","threads": 4},"externalReferences": {"connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"}}Same activity in the repo after committing from the Fabric UI (bug)
{"type": "InvokeDataBuildToolJob","typeProperties": {"dataBuildToolJobId": "bae2094a-fd82-98c2-4c7d-46699b0172aa","workspaceId": "00000000-0000-0000-0000-000000000000","select": "epm","threads": 4},"externalReferences": {"connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"},"policy": {"timeout": "0.12:00:00","retry": 0,"retryIntervalInSeconds": 30,"secureInput": false,"secureOutput": false},"name": "RefreshEpmSeeds","dependsOn": []}The `workspaceId` (physical GUID to all-zeros "current workspace") and `dataBuildToolJobId` (physical to logical ID) transformations are expected Git integration behavior. The removal of the `operation` block is not; it is dropped rather than transformed.Impact
- Committed pipeline definitions are incomplete and not a reliable source of truth.
- Deployments/syncs to other environments produce a dbt job activity with no operation, causing deployed pipelines to fail.
- The failure is silent at commit time, so it is easy to ship broken definitions without noticing.
I'm happy to provide additional detail or a full pipeline definition if helpful.- michaelfabricFrequent Visitor
Thank you ryan-schofield for providing additional details!
- Prince0011Solution Sage
Hi User,
Based on your description, this appears to be a product issue with the serialization of the dbt Job Pipeline Activity configuration during Git commits.
Since the Operation setting (for example, run, build, or compile) isn't being persisted to the repository, downstream CI/CD deployments recreate the pipeline without this required property, which then causes the activity to fail at runtime.
As a workaround, you can:
Verify whether the issue occurs across multiple workspaces and repositories to rule out a workspace-specific configuration.
Check if you're using the latest Fabric experience, as this may have been addressed in a recent update.
Continue using the manual JSON edit temporarily, although as you've observed, Fabric currently removes the operation property during the next synchronization.
If the behavior is consistently reproducible, collect the pipeline JSON, deployment logs, and Git commit history, then open a Microsoft Support ticket, as this appears to be a product bug rather than a configuration issue.
For more information:
CI/CD in Microsoft Fabric: https://learn.microsoft.com/fabric/cicd/
Git integration in Microsoft Fabric: https://learn.microsoft.com/fabric/cicd/git-integration/intro-to-git-integration
Could you also confirm:
Does this occur for all dbt operations (run, build, and compile) or only specific ones?
Is the issue reproducible in a newly created pipeline?
Which Git provider are you using (GitHub or Azure DevOps)?
If others in the community can reproduce the same behavior, it would help confirm whether this is a broader product issue affecting dbt pipeline activities.
💡 Helpful? Give a Kudos 👍 — keep the community growing.
✅ Solved your issue? Mark this as the Accepted Solution ✔️
Best regards,
Prince Singh | Data Science & Microsoft Fabric Enthusiast - v-sathmakuriCommunity Support
Thank you Prince0011 , ryan-schofield and sannavajjala for the immediate response.
Hi michaelfabric ,
Thanks for reaching out to fabric community.
Could you please let us know whether the responses provided above helped you in resolving the issue and let us know if you have any additional questions, we are happy to address.
Thanks!!
- michaelfabricFrequent Visitor
Hi v-sathmakuri ,
Based on the other responses, the issue appears to be a MS Fabric bug. ryan-schofield provided additional details. He also suggested a workaround, but the issue will only be fully resolved when the bug is fixed.
Can't this discussion be forwarded to Microsoft support?Kind regards,
Michaël
- v-sathmakuriCommunity Support
Hi michaelfabric ,
Please raise a ticket with microsoft support team (https://support.fabric.microsoft.com/en-IN/support/) and you can share the thread link there so that they will report it to the concerned team.
Thanks!!