topic Re: Create Data Pipeline with existing definition using Terraform in Data Engineering

Create Data Pipeline with existing definition using Terraform

chetanhiwale — Fri, 20 Jun 2025 06:09:50 GMT

Hi Folks,

I am trying to create data pipelines using Terraform where I already have defined definition at a path. In path, the defination is in JSON format so no need to decode the defination from Base64 format. I am getting following error, any one has faced similar error.

Error :

Error: Create operation
│
│ with fabric_data_pipeline.bronze_pipelines["PL_BRONZE_"],
│ on items.tf line 5, in resource "fabric_data_pipeline" "bronze_pipelines":
│ 5: resource "fabric_data_pipeline" "bronze_pipelines" {
│
│ Could not create resource: The request could not be processed due to an error
│
│ Error Code: UnknownError

Terraform Source Code

resource "fabric_data_pipeline" "bronze_pipelines" {

for_each = toset(var.bronze_datapipelines)

display_name = each.value

description = "example with definition bootstrapping"

workspace_id = var.workspace_id

format = "Default"

definition = {

"pipeline-content.json" = {

source = "${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json"

tokens = {

"MyValue" = "World"

}

Path :

Re: Create Data Pipeline with existing definition using Terraform

v-aatheeque — Fri, 20 Jun 2025 10:42:49 GMT

Hi @chetanhiwale ,
Thanks for raising this query in Fabric Community Forum.

Thanks for sharing the error and the context.

The error you’re encountering could you try the below trouble shooting steps :

Ensure that the JSON file located at the specified path (${var.bronze_base_path}/datapipelines/${each.value}.DataPipeline/pipeline-content.json) is valid and correctly formatted. You can use a JSON validator to check for any syntax errors. Confirm that the path to the JSON file is correct and accessible.
Ensure that the Terraform process has the necessary permissions to read the file.
Double-check the values of " var.bronze_datapipelines " and " var.workspace_id "to ensure they are correctly defined and populated. Incorrect or empty values could lead to errors during resource creation.
Ensure that the fabric_data_pipeline resource configuration is correct.
Ensure you're using the latest version of the Microsoft Fabric provider. Older versions may not support all features or may have unresolved bugs.

for more info prefer this blog : Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric

Hope this helps !!

If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

If you continue to face issues, feel free to reach out to us for further assistance!

Re: Create Data Pipeline with existing definition using Terraform

v-aatheeque — Tue, 24 Jun 2025 17:42:54 GMT

Hi @chetanhiwale

I wanted to follow up regarding your inquiry about the "UnknownError" you encountered while creating data pipelines using Terraform. I provided some troubleshooting steps in my previous response, but I haven't received any feedback from you yet.

To ensure we can assist you effectively, please take a moment to review the suggestions and let us know if you have made any progress or if you need further assistance.

If the issue is resolved accept the solution.
Your input is valuable, and we want to ensure you receive the support you need.

Thank you for your understanding!

Re: Create Data Pipeline with existing definition using Terraform

v-aatheeque — Fri, 27 Jun 2025 17:52:12 GMT

Hi @chetanhiwale

Just a quick reminder to check the troubleshooting steps I shared earlier regarding the error you encountered:

If you’ve already tried these and still face issues, please share more details so I can help further!

It would be great if you could share an update on your progress or let me know about any challenges you’re encountering. Your feedback helps us ensure you get the right support.

If your issue has been resolved, please consider marking the response as the accepted solution so it can help others as well.

I look forward to hearing back from you!

Thank You!!

Re: Create Data Pipeline with existing definition using Terraform

v-aatheeque — Mon, 30 Jun 2025 16:51:39 GMT

Hi @chetanhiwale

I wanted to check back in regarding the “ UnknownError ” you encountered while creating data pipelines with Terraform. I previously shared some troubleshooting suggestions, but I haven’t heard back from you yet.

Could you let us know if you had a chance to try those steps and whether they helped resolve the issue? If you’re still experiencing problems or need additional support, please feel free to share more details we’re here to help!

If the issue has already been resolved, please consider marking the response as the accepted solution so it can assist others facing a similar challenge.

Your feedback is important, and we want to make sure you get the help you need.

Looking forward to your update!

Re: Create Data Pipeline with existing definition using Terraform

hale — Tue, 05 Aug 2025 01:54:48 GMT

Hi,
I'm having the same issue and have been going through your debug list but no luck:

json file is definitely picked up, I tried with a deliberate wrong path then TRF throws error straight away with the config.
I even copy the json file to the same folder as TRF so permission should not be an issue.
The json file is copied directly from a working pipeline in another workspace so it should be valid and working.
I am using 1.3.0 version which is latest according to TRF registry page.

Is there anything that I miss to debug this? Also would be very helpful if there is more information than unknown error, is there a log file that I can export and have a look into?

Thanks,

Re: Create Data Pipeline with existing definition using Terraform

v-aatheeque — Thu, 07 Aug 2025 07:34:13 GMT

Hi @hale

In some deployment failure scenarios, you may need to collect Terraform logs for troubleshooting. When running a Terraform command, the default output in the terminal is minimal, showing only basic information. If you require detailed logs, such as API requests, HTTP properties (including Correlation IDs and Activity IDs), and Azure responses, you should enable advanced logging.

To enable advanced logging, set the TF_LOG environment variable to DEBUG before running your Terraform command:

export TF_LOG=DEBUG
terraform apply

Reference : Enable logs to debug Terraform | Terraform | HashiCorp Developer

Hope this helps !!

Re: Create Data Pipeline with existing definition using Terraform

hale — Fri, 08 Aug 2025 01:43:15 GMT

Thanks, I have tried that but not giving me helpful information or at least to me.
I did manage to fix my issue by stripping off the data pipeline so that it can be as the most basic as possible, later I found that a connection to a Lakehouse is the culprit for the issue, once I change that connection to something else then it works. My guess is I need to have a dependency that lakehouse is being created first. Newbie mistake, thanks for your help.