Forum Discussion

michaelfabric's avatar
michaelfabric
Frequent Visitor
3 months ago
Solved

dbt job pipeline activity: bug

Hi everone,   The operation (like run, build and compile, under "Advanced Settings") of a dbt job activity in a pipeline is not being pushed to the repo when doing a commit. As a result, my fabric-...
data engineering
data pipeline
  • michaelfabric's avatar
    michaelfabric
    2 months ago

    I created a support ticket and have the following update to share: Microsoft support has excalated this issue and a fix will be implemented:

    "We have done further tests on this internally and were able to reproduce the behavior. Then escalated it internally to our Product team. As a result, this will be addressed and the current tentative for the fix is in September. 

    If things go as planned, it should be during the first week of September but deployment can vary depending on different regions."

     

    Please check the reply of ryan-schofield for a temporary workaround.

sannavajjala's avatar
sannavajjala
Icon for Resolver II rankResolver II
3 months ago

Hi,

 

Hi Michael,

This sounds like a product limitation or serialization issue with the current Fabric Git integration rather than something you're doing incorrectly. If the Operation setting (Run, Build, Compile) is configured in the UI but is not persisted in the repository artifacts, deployments through Fabric CI/CD can result in an invalid activity definition in the target workspace.

The fact that manually adding "operation": "build" to the pipeline JSON resolves the runtime error, but Fabric subsequently removes it during the next commit, suggests the Git representation of the dbt activity is not fully round-tripping all properties. In that case, the source of truth in the UI and the source-controlled artifact are becoming inconsistent.

A few things I'd recommend checking:

  • Verify all environments are running the same Fabric feature set and Git integration experience.
  • Confirm whether the issue occurs with newly created dbt activities or only existing ones.
  • Compare the JSON generated before and after changing the Operation value to see whether any related properties are being persisted.
  • Test deploying the pipeline directly between workspaces without Git to determine whether the issue is specifically tied to source control serialization.

If the behavior is reproducible, it is likely a bug and worth raising with Microsoft Support. Be sure to include:

  • The pipeline JSON stored in Git
  • Screenshots of the Operation setting in the UI
  • Deployment logs showing the missing Operation error
  • The Fabric workspace and Git integration experience being used

In the meantime, if Fabric continues to remove the property during commits, any manual JSON edits are unlikely to be a sustainable workaround because the platform will overwrite them during synchronization. Hopefully others in the community can confirm whether they're seeing the same behavior with dbt activities and Git-managed deployments.

 

Thanks,

Manoj Annavajjala

  • ryan-schofield's avatar
    ryan-schofield
    Icon for Advocate II rankAdvocate II
    3 months ago

    I'm running into the same issue. Here's a rundown:

    Fabric Git integration drops the `operation` property from `InvokeDataBuildToolJob` (dbt job) activities on commit

     

    Summary

     
    When a data pipeline containing a dbt job activity (`InvokeDataBuildToolJob`) is committed to a Git-connected workspace, the Fabric Git integration silently removes the `operation` property from `typeProperties`. The property is not substituted with a logical ID or placeholder like `workspaceId` and `dataBuildToolJobId` are; it is simply dropped from the serialized definition entirely.
     
    Because the drop is silent (no error, no warning during commit or merge), the pipeline definition in the repo is incomplete. On deployment/sync to another environment, the activity has no `operation` value and the dbt job cannot run as authored.
     
    This looks like a serialization gap in the Git integration for the `InvokeDataBuildToolJob` activity type.
     

    Environment

     
    • Feature: dbt job activity in Fabric Data Factory pipeline (Preview)
    • Activity type: `InvokeDataBuildToolJob`
    • Source control: Fabric Git integration (Azure DevOps)
     

    Steps to Reproduce

     
    1. Create a data pipeline in a Git-connected workspace.
    2. Add a dbt job activity (`InvokeDataBuildToolJob`).
    3. Set the operation field.
    4. Set `select` and `threads`, and configure the connection.
    5. Save the pipeline and commit the workspace to Git from the Fabric UI.
    6. Inspect the committed pipeline definition in the repo.
     
    Expected: The `operation` property is serialized to the repo (as the expression/value that was authored), consistent with the documented support for dynamic content on all dbt job activity settings.
     
    Actual: The `operation` property is missing from `typeProperties` in the committed definition.
     

    Example

     

    Activity as authored in the workspace (correct)

    {
        "name": "RefreshEpmSeeds",
        "type": "InvokeDataBuildToolJob",
        "dependsOn": [],
        "policy": {
            "timeout": "0.12:00:00",
            "retry": 0,
            "retryIntervalInSeconds": 30,
            "secureOutput": false,
            "secureInput": false
        },
        "typeProperties": {
            "dataBuildToolJobId": "5cbc04b3-f9ba-451d-a7ce-ab904a68d0e1",
            "workspaceId": "71b4b3df-cf7c-43a3-b170-0f87ca8f84f9",
            "operation": "seed",
            "select": "epm",
            "threads": 4
        },
        "externalReferences": {
            "connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"
        }
    }
     

    Same activity in the repo after committing from the Fabric UI (bug)

     
    {
        "type": "InvokeDataBuildToolJob",
        "typeProperties": {
            "dataBuildToolJobId": "bae2094a-fd82-98c2-4c7d-46699b0172aa",
            "workspaceId": "00000000-0000-0000-0000-000000000000",
            "select": "epm",
            "threads": 4
        },
        "externalReferences": {
            "connection": "b75be457-2790-4774-a97a-bfc1d0b7c2b1"
        },
        "policy": {
            "timeout": "0.12:00:00",
            "retry": 0,
            "retryIntervalInSeconds": 30,
            "secureInput": false,
            "secureOutput": false
        },
        "name": "RefreshEpmSeeds",
        "dependsOn": []
    }
     
    The `workspaceId` (physical GUID to all-zeros "current workspace") and `dataBuildToolJobId` (physical to logical ID) transformations are expected Git integration behavior. The removal of the `operation` block is not; it is dropped rather than transformed.
     

    Impact

     
    • Committed pipeline definitions are incomplete and not a reliable source of truth.
    • Deployments/syncs to other environments produce a dbt job activity with no operation, causing deployed pipelines to fail.
    • The failure is silent at commit time, so it is easy to ship broken definitions without noticing.
     
    I'm happy to provide additional detail or a full pipeline definition if helpful.

     

    • michaelfabric's avatar
      michaelfabric
      Frequent Visitor
      2 months ago

      Thank you ryan-schofield for providing additional details!