Create Data Pipeline with existing definition using Terraform

[chetanhiwale]

Hi Folks,

I am trying to create data pipelines using Terraform where  I already have defined definition at a path. In path, the defination is in JSON format so no need to decode the defination from Base64 format. I am getting following error, any one has faced similar error. 

Error : 

Error: Create operation
│
│ with fabric_data_pipeline.bronze_pipelines["PL_BRONZE_"],
│ on items.tf line 5, in resource "fabric_data_pipeline" "bronze_pipelines":
│ 5: resource "fabric_data_pipeline" "bronze_pipelines" {
│
│ Could not create resource: The request could not be processed due to an error
│
│ Error Code: UnknownError

Terraform Source Code

resource "fabric_data_pipeline" "bronze_pipelines" {

  for_each = toset(var.bronze_datapipelines)

  display_name              = each.value

  description               = "example with definition bootstrapping"

  workspace_id              = var.workspace_id

  format                    = "Default"

  definition = {

    "pipeline-content.json" = {

      source = "${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json"

      tokens = {

        "MyValue" = "World"

      }

    }

  }

}

Path : 

[v-aatheeque]

Hi @chetanhiwale ,
Thanks for raising this query in Fabric Community Forum.

Thanks for sharing the error and the context.

The error you’re encountering could you try the below trouble shooting steps :

• Ensure that the JSON file located at the specified path (${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json) is valid and correctly formatted. You can use a JSON validator to check for any syntax errors. Confirm that the path to the JSON file is correct and accessible.
•  Ensure that the Terraform process has the necessary permissions to read the file.
• Double-check the values of " var.bronze_datapipelines " and " var.workspace_id "to ensure they are correctly defined and populated. Incorrect or empty values could lead to errors during resource creation.
• Ensure that the fabric_data_pipeline resource configuration is correct.
• Ensure you're using the latest version of the Microsoft Fabric provider. Older versions may not support all features or may have unresolved bugs.

for more info prefer this blog :  Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft Fa...

Hope this helps !!

If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

If you continue to face issues, feel free to reach out to us for further assistance!

[v-aatheeque]

Hi  @chetanhiwale 

I wanted to follow up regarding your inquiry about the "UnknownError" you encountered while creating data pipelines using Terraform. I provided some troubleshooting steps in my previous response, but I haven't received any feedback from you yet.

To ensure we can assist you effectively, please take a moment to review the suggestions and let us know if you have made any progress or if you need further assistance.

If the issue is resolved accept the solution.
Your input is valuable, and we want to ensure you receive the support you need.

Thank you for your understanding!