Forum Discussion

chetanhiwale's avatar
chetanhiwale
Resolver I
1 year ago
Solved

Create Data Pipeline with existing definition using Terraform

Hi Folks,

I am trying to create data pipelines using Terraform where  I already have defined definition at a path. In path, the defination is in JSON format so no need to decode the defination from Base64 format. I am getting following error, any one has faced similar error. 

Error : 

Error: Create operation
│
│ with fabric_data_pipeline.bronze_pipelines["PL_BRONZE_"],
│ on items.tf line 5, in resource "fabric_data_pipeline" "bronze_pipelines":
│ 5: resource "fabric_data_pipeline" "bronze_pipelines" {
│
│ Could not create resource: The request could not be processed due to an error
│
│ Error Code: UnknownError



Terraform Source Code

resource "fabric_data_pipeline" "bronze_pipelines" {
  for_each = toset(var.bronze_datapipelines)
  display_name              = each.value
  description               = "example with definition bootstrapping"
  workspace_id              = var.workspace_id
  format                    = "Default"
  definition = {
    "pipeline-content.json" = {
      source = "${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json"
      tokens = {
        "MyValue" = "World"
      }
    }
  }
}
Path : 

 

 

Data Pipeline
  • v-aatheeque's avatar
    v-aatheeque
    1 year ago

    Hi chetanhiwale ,
    Thanks for raising this query in Fabric Community Forum.

    Thanks for sharing the error and the context.

    The error you’re encountering could you try the below trouble shooting steps :

     

    • Ensure that the JSON file located at the specified path (${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json) is valid and correctly formatted. You can use a JSON validator to check for any syntax errors. Confirm that the path to the JSON file is correct and accessible.
    •  Ensure that the Terraform process has the necessary permissions to read the file.
    • Double-check the values of " var.bronze_datapipelines " and " var.workspace_id "to ensure they are correctly defined and populated. Incorrect or empty values could lead to errors during resource creation.
    • Ensure that the fabric_data_pipeline resource configuration is correct.
    • Ensure you're using the latest version of the Microsoft Fabric provider. Older versions may not support all features or may have unresolved bugs.

    for more info prefer this blog :  Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric

    Hope this helps !!

     

    If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

    If you continue to face issues, feel free to reach out to us for further assistance!



7 Replies

  • v-aatheeque's avatar
    v-aatheeque
    Community Support
    1 year ago

    Hi chetanhiwale ,
    Thanks for raising this query in Fabric Community Forum.

    Thanks for sharing the error and the context.

    The error you’re encountering could you try the below trouble shooting steps :

     

    • Ensure that the JSON file located at the specified path (${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json) is valid and correctly formatted. You can use a JSON validator to check for any syntax errors. Confirm that the path to the JSON file is correct and accessible.
    •  Ensure that the Terraform process has the necessary permissions to read the file.
    • Double-check the values of " var.bronze_datapipelines " and " var.workspace_id "to ensure they are correctly defined and populated. Incorrect or empty values could lead to errors during resource creation.
    • Ensure that the fabric_data_pipeline resource configuration is correct.
    • Ensure you're using the latest version of the Microsoft Fabric provider. Older versions may not support all features or may have unresolved bugs.

    for more info prefer this blog :  Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric

    Hope this helps !!

     

    If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

    If you continue to face issues, feel free to reach out to us for further assistance!



    • v-aatheeque's avatar
      v-aatheeque
      Community Support
      1 year ago

      Hi  chetanhiwale 

      I wanted to follow up regarding your inquiry about the "UnknownError" you encountered while creating data pipelines using Terraform. I provided some troubleshooting steps in my previous response, but I haven't received any feedback from you yet.

      To ensure we can assist you effectively, please take a moment to review the suggestions and let us know if you have made any progress or if you need further assistance.

      If the issue is resolved accept the solution.
      Your input is valuable, and we want to ensure you receive the support you need.

       

      Thank you for your understanding!

      • v-aatheeque's avatar
        v-aatheeque
        Community Support
        1 year ago

        Hi chetanhiwale 

        Just a quick reminder to check the troubleshooting steps I shared earlier regarding the error you encountered:

        If you’ve already tried these and still face issues, please share more details so I can help further!

        It would be great if you could share an update on your progress or let me know about any challenges you’re encountering. Your feedback helps us ensure you get the right support.

        If your issue has been resolved, please consider marking the response as the accepted solution so it can help others as well.

        I look forward to hearing back from you!

        Thank You!!

    • hale's avatar
      hale
      Helper II
      1 year ago

      Hi,
      I'm having the same issue and have been going through your debug list but no luck:

      1. json file is definitely picked up, I tried with a deliberate wrong path then TRF throws error straight away with the config.
      2. I even copy the json file to the same folder as TRF so permission should not be an issue.
      3. The json file is copied directly from a working pipeline in another workspace so it should be valid and working.
      4. I am using 1.3.0 version which is latest according to TRF registry page.

      Is there anything that I miss to debug this? Also would be very helpful if there is more information than unknown error, is there a log file that I can export and have a look into?

       

      Thanks,

      Ha

      • v-aatheeque's avatar
        v-aatheeque
        Community Support
        1 year ago

        Hi hale 

        In some deployment failure scenarios, you may need to collect Terraform logs for troubleshooting. When running a Terraform command, the default output in the terminal is minimal, showing only basic information. If you require detailed logs, such as API requests, HTTP properties (including Correlation IDs and Activity IDs), and Azure responses, you should enable advanced logging.

        To enable advanced logging, set the TF_LOG environment variable to DEBUG before running your Terraform command:

        export TF_LOG=DEBUG
        terraform apply

        Reference : Enable logs to debug Terraform | Terraform | HashiCorp Developer

        Hope this helps !!