Forum Discussion

chetanhiwale's avatar
chetanhiwale
Resolver I
1 year ago
Solved

Create Data Pipeline with existing definition using Terraform

Hi Folks, I am trying to create data pipelines using Terraform where  I already have defined definition at a path. In path, the defination is in JSON format so no need to decode the defination fro...
data pipeline
  • v-aatheeque's avatar
    v-aatheeque
    1 year ago

    Hi chetanhiwale ,
    Thanks for raising this query in Fabric Community Forum.

    Thanks for sharing the error and the context.

    The error you’re encountering could you try the below trouble shooting steps :

     

    • Ensure that the JSON file located at the specified path (${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json) is valid and correctly formatted. You can use a JSON validator to check for any syntax errors. Confirm that the path to the JSON file is correct and accessible.
    •  Ensure that the Terraform process has the necessary permissions to read the file.
    • Double-check the values of " var.bronze_datapipelines " and " var.workspace_id "to ensure they are correctly defined and populated. Incorrect or empty values could lead to errors during resource creation.
    • Ensure that the fabric_data_pipeline resource configuration is correct.
    • Ensure you're using the latest version of the Microsoft Fabric provider. Older versions may not support all features or may have unresolved bugs.

    for more info prefer this blog :  Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric

    Hope this helps !!

     

    If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

    If you continue to face issues, feel free to reach out to us for further assistance!



hale's avatar
hale
Helper II
1 year ago

Hi,
I'm having the same issue and have been going through your debug list but no luck:

  1. json file is definitely picked up, I tried with a deliberate wrong path then TRF throws error straight away with the config.
  2. I even copy the json file to the same folder as TRF so permission should not be an issue.
  3. The json file is copied directly from a working pipeline in another workspace so it should be valid and working.
  4. I am using 1.3.0 version which is latest according to TRF registry page.

Is there anything that I miss to debug this? Also would be very helpful if there is more information than unknown error, is there a log file that I can export and have a look into?

 

Thanks,

Ha

v-aatheeque's avatar
v-aatheeque
Community Support
1 year ago

Hi hale 

In some deployment failure scenarios, you may need to collect Terraform logs for troubleshooting. When running a Terraform command, the default output in the terminal is minimal, showing only basic information. If you require detailed logs, such as API requests, HTTP properties (including Correlation IDs and Activity IDs), and Azure responses, you should enable advanced logging.

To enable advanced logging, set the TF_LOG environment variable to DEBUG before running your Terraform command:

export TF_LOG=DEBUG
terraform apply

Reference : Enable logs to debug Terraform | Terraform | HashiCorp Developer

Hope this helps !!

  • hale's avatar
    hale
    Helper II
    1 year ago

    Thanks, I have tried that but not giving me helpful information or at least to me.
    I did manage to fix my issue by stripping off the data pipeline so that it can be as the most basic as possible, later I found that a connection to a Lakehouse is the culprit for the issue, once I change that connection to something else then it works. My guess is I need to have a dependency that lakehouse is being created first. Newbie mistake, thanks for your help.