Forum Discussion

michaelfabric's avatar
michaelfabric
Frequent Visitor
2 months ago
Solved

dbt job pipeline activity: bug

Hi everone,   The operation (like run, build and compile, under "Advanced Settings") of a dbt job activity in a pipeline is not being pushed to the repo when doing a commit. As a result, my fabric-...
data engineering
data pipeline
  • michaelfabric's avatar
    michaelfabric
    1 month ago

    I created a support ticket and have the following update to share: Microsoft support has excalated this issue and a fix will be implemented:

    "We have done further tests on this internally and were able to reproduce the behavior. Then escalated it internally to our Product team. As a result, this will be addressed and the current tentative for the fix is in September. 

    If things go as planned, it should be during the first week of September but deployment can vary depending on different regions."

     

    Please check the reply of ryan-schofield for a temporary workaround.

ryan-schofield's avatar
ryan-schofield
Icon for Advocate II rankAdvocate II
1 month ago

I'm running into the same issue. Here's a rundown:

Fabric Git integration drops the `operation` property from `InvokeDataBuildToolJob` (dbt job) activities on commit

 

Summary

 
When a data pipeline containing a dbt job activity (`InvokeDataBuildToolJob`) is committed to a Git-connected workspace, the Fabric Git integration silently removes the `operation` property from `typeProperties`. The property is not substituted with a logical ID or placeholder like `workspaceId` and `dataBuildToolJobId` are; it is simply dropped from the serialized definition entirely.
 
Because the drop is silent (no error, no warning during commit or merge), the pipeline definition in the repo is incomplete. On deployment/sync to another environment, the activity has no `operation` value and the dbt job cannot run as authored.
 
This looks like a serialization gap in the Git integration for the `InvokeDataBuildToolJob` activity type.
 

Environment

 
  • Feature: dbt job activity in Fabric Data Factory pipeline (Preview)
  • Activity type: `InvokeDataBuildToolJob`
  • Source control: Fabric Git integration (Azure DevOps)
 

Steps to Reproduce

 
  1. Create a data pipeline in a Git-connected workspace.
  2. Add a dbt job activity (`InvokeDataBuildToolJob`).
  3. Set the operation field.
  4. Set `select` and `threads`, and configure the connection.
  5. Save the pipeline and commit the workspace to Git from the Fabric UI.
  6. Inspect the committed pipeline definition in the repo.
 
Expected: The `operation` property is serialized to the repo (as the expression/value that was authored), consistent with the documented support for dynamic content on all dbt job activity settings.
 
Actual: The `operation` property is missing from `typeProperties` in the committed definition.
 

Example

 

Activity as authored in the workspace (correct)

{
    "name": "RefreshEpmSeeds",
    "type": "InvokeDataBuildToolJob",
    "dependsOn": [],
    "policy": {
        "timeout": "0.12:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false,
        "secureInput": false
    },
    "typeProperties": {
        "dataBuildToolJobId": "5cbc04b3-f9ba-451d-a7ce-ab904a68d0e1",
        "workspaceId": "71b4b3df-cf7c-43a3-b170-0f87ca8f84f9",
        "operation": "seed",
        "select": "epm",
        "threads": 4
    },
    "externalReferences": {
        "connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"
    }
}
 

Same activity in the repo after committing from the Fabric UI (bug)

 
{
    "type": "InvokeDataBuildToolJob",
    "typeProperties": {
        "dataBuildToolJobId": "bae2094a-fd82-98c2-4c7d-46699b0172aa",
        "workspaceId": "00000000-0000-0000-0000-000000000000",
        "select": "epm",
        "threads": 4
    },
    "externalReferences": {
        "connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"
    },
    "policy": {
        "timeout": "0.12:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureInput": false,
        "secureOutput": false
    },
    "name": "RefreshEpmSeeds",
    "dependsOn": []
}
 
The `workspaceId` (physical GUID to all-zeros "current workspace") and `dataBuildToolJobId` (physical to logical ID) transformations are expected Git integration behavior. The removal of the `operation` block is not; it is dropped rather than transformed.
 

Impact

 
  • Committed pipeline definitions are incomplete and not a reliable source of truth.
  • Deployments/syncs to other environments produce a dbt job activity with no operation, causing deployed pipelines to fail.
  • The failure is silent at commit time, so it is easy to ship broken definitions without noticing.
 
I'm happy to provide additional detail or a full pipeline definition if helpful.