Forum Discussion
Parameterize a data connection using copy data activity when extracting data from GBQ to Lakehouse
Hello,
I am trying to extract the data from google BQ to lakeshoue, however i have parametarized the google BD SQL, in the same way i would like to parametarize the connection name as well, how can i parametarize the connection name, i tried this below and getting error, any help is greatly appriciated.
In source connection if i select my connection from dropdown list its working as expected, but not with the parametarized connection.
Hi sumanthkakarla,
Thank you for reaching out to the Microsoft Fabric Forum Community. Please try with below code and refine as per your requirements.
$filePath = "$(Build.SourcesDirectory)/Pipeline/pipeline1.json"
# Define the original and replacement connection strings
$original = "Test-w3prc4skt6eu3pib5o7d4pbqqm.datawarehouse.fabric.microsoft.com"
$replacement = "test.datawarehouse.fabric.microsoft.com"
# Check if file exists
if (Test-Path $filePath) {
$content = Get-Content $filePath -Raw
if ($content -like "*$original*") {
Write-Host "Found target string in $filePath. Replacing..."
$updated = $content -replace [Regex]::Escape($original), $replacement
Set-Content -Path $filePath -Value $updated
Write-Host "Replacement complete in $filePath"
}
else {
Write-Host "Target string not found. No changes made."
}
}
else {
Write-Error "File not found: $filePath"
}Please make sure to update the artifactId, workspaceId, and endpoint with the correct values according to your requirements if you are using the warehouse. For lakehouse deployments using the deployment pipelines in fabrics, the system will automatically point to the lakehouse in the target workspace if it has the same name.
Best regards,
Prasanna Kumar
13 Replies
As far as I'm aware, currently parameterized connections are not supported. You can create or vote for the idea here. https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas
Thank you GilbertQ , its real surprising that it wont support parameterized connections, so always there is some manual intervenion is needed after deployment, hum... there should be another way...
Thank you GilbertQ , the parametrised connection is working for me as expected, we can close the thread.
The only problem i am facing right now is creating a deployment rules for data pipeline, what i heard from Microsoft was deployment rules are not supported for data pipelines, its a manual effort once we migrate the data pipeline to stage or prod workspace to update all the pipeline parameter values.
Thank you Anonymous , like you said may be i can give it a try using power shell script to update the pipeline parameter values.
Hi sumanthkakarla,
Thank you for reaching out to the Microsoft Fabric Forum Community.
Currently, fabric pipelines do not support direct parameterization. As an alternative, you can use a PowerShell script within your DevOps pipeline to update the environment connection string with the new value before deploying.
Best regards,
Prasanna KumarHi sumanthkakarla,
Just checking in to see if the suggested workaround using a PowerShell script in your DevOps pipeline helped with parameterizing the connection name in Fabric pipelines. Let me know if you need any further support or clarification.Best regards,
Prasanna KumarThank you Anonymous
Do you have any sample powershell script that i can use for this request ? i can refine it as per my requirement.
Hi sumanthkakarla,
Thank you for reaching out to the Microsoft Fabric Forum Community. Please try with below code and refine as per your requirements.
$filePath = "$(Build.SourcesDirectory)/Pipeline/pipeline1.json"
# Define the original and replacement connection strings
$original = "Test-w3prc4skt6eu3pib5o7d4pbqqm.datawarehouse.fabric.microsoft.com"
$replacement = "test.datawarehouse.fabric.microsoft.com"
# Check if file exists
if (Test-Path $filePath) {
$content = Get-Content $filePath -Raw
if ($content -like "*$original*") {
Write-Host "Found target string in $filePath. Replacing..."
$updated = $content -replace [Regex]::Escape($original), $replacement
Set-Content -Path $filePath -Value $updated
Write-Host "Replacement complete in $filePath"
}
else {
Write-Host "Target string not found. No changes made."
}
}
else {
Write-Error "File not found: $filePath"
}Please make sure to update the artifactId, workspaceId, and endpoint with the correct values according to your requirements if you are using the warehouse. For lakehouse deployments using the deployment pipelines in fabrics, the system will automatically point to the lakehouse in the target workspace if it has the same name.
Best regards,
Prasanna Kumar
Hi sumanthkakarla,
Just checking in to see if the information shared was helpful in addressing your query regarding parameterizing the connection name in your Google BigQuery to Lakehouse setup. As noted, parameterized connections aren’t currently supported in Fabric pipelines.Best regards,
Prasanna KumarHi sumanthkakarla,
We wanted to kindly check in to see if everything is working as expected after trying the suggested solution. If there’s anything else we can assist with, please don’t hesitate to ask.
Warm regards,
Prasanna Kumar
Thank you Anonymous , i did not try that, distracted with other stuff, now i am facing another issue after migrating to test. i have used a release pipline and created a deployment rule to update the connection to test lakehosue, however when i downloaded the report to desktop i am not seeing any actie conections, Why, i can create a new thread if you would like.
Hi sumanthkakarla,
Could you please provide more detailed information along with screenshots? That would help us understand the issue better and assist you more effectively.
Thanks & Regards,Prasanna Kumar
Hi sumanthkakarla,
Could you please provide more detailed information along with screenshots? That would help us understand the issue better and assist you more effectively.
Thanks & Regards,Prasanna Kumar
Hi @sumanthkakarla,
Could you please provide more detailed information along with screenshots? That would help us understand the issue better and assist you more effectively.
Thanks & Regards,Prasanna Kumar
