Forum Discussion
Fabric Warehouse component Deployment using Azure Repo(DevOps)
- 1 year ago
Thanks for sharing your YAML and more details! You're on the right track, and it's great to see your pipeline taking shape. Let me help clarify a few points and share some practical steps and sample YAML to get you going, especially around subscription handling and modular deployments.
1. Azure Subscription in YAML:
To use Azure CLI tasks in your pipeline, you need to provide a service connection that has access to the right Azure subscription. In Azure DevOps, this is handled via a “service connection” (not just the subscription ID).- In your YAML, instead of manually specifying the subscription ID as a variable, use the azureSubscription input, which references a service connection you create in Azure DevOps (Project Settings > Service connections).
- Example:YAML
- task: AzureCLI@2 inputs: azureSubscription: 'Your-Service-Connection-Name' scriptType: 'bash' scriptLocation: 'inlineScript' inlineScript: | # Your scripts here
- The service connection handles authentication and securely stores credentials.
2. Modular/Single & Multi-Select Deployments:
You can use pipeline variables or parameters to control which tables or components deploy, making your pipeline flexible for single or multi-table deployment.- Define a parameter at the top of your YAML:YAML
parameters: - name: tablesToDeploy type: object default: ['Table1', 'Table2']
- Reference this parameter in your deployment scripts to loop through and deploy only the selected tables.
3. Sample Step for Validating Deployment:
Here's how you might add a validation step for your Fabric SQL endpoint: yaml - task: AzureCLI@2 inputs: azureSubscription: 'Your-Service-Connection-Name' scriptType: 'bash' scriptLocation: 'inlineScript' inlineScript: | az login --service-principal -u $(clientId) -p $(clientSecret) --tenant $(tenantId) sqlcmd -S $(sqlEndpoint) -d $(databaseName) -U $(sqlUser) -P $(sqlPass) -Q "SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES"- Make sure your pipeline variables (like clientId, clientSecret, etc.) are set using Azure DevOps secrets for security.
4. Step-by-Step:
- Set up an Azure Service Connection in Azure DevOps for your subscription.
- Reference this connection in your YAML (azureSubscription: ...).
- Use parameters/variables to control which components get deployed.
- Add validation steps as needed (as above).
- For multi-table or environment-based deployment, loop through parameterized table names or environment settings.
If you want, I can share a more complete YAML template that includes parameterized deployment and best practices for secrets/connection handling. Just let me know your exact scenario (e.g., number of environments, single vs. multiple tables, etc.), and I can tailor it for you.
Summary:
- Use Azure DevOps “service connections” for subscription/auth.
- Parameterize your pipeline for flexible deployment.
- Use inline scripts for validation and deployment logic.
- Secure your secrets with pipeline variables.
Let me know if you’d like a full example YAML or need help with a specific step! You’re making great progress—just a couple of tweaks and you’ll have a robust deployment pipeline.
Hi @marvelbites06,
As we haven’t heard back from you, we wanted to kindly follow up to check if the issue got resolved? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
I was not able to getback as my Service Connection was not created and I was not able tomove ahead. Now I have the service connection created and hte Pipeline is validated. It will be really helpful if you can guide me though the paramter creation and how to Map the branches. I am pasting the YAML below.
I have Main Branch and I have ADS_DEV as Testing Branch. I want to move the data from Test to Prod with Example that I want to Move Sales_Document_Fact to Prod. How will I have to Parameterise it.
Let me kow if we ca connect over teams as well I will be happy to send you the invite.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
vmImage: 'windows-latest'
variables:
sqlEndpoint: 'cji6xqkgpgpcxiacaz7r2se-lol6ltv7evohd5tzhxy6e.datawarehouse.fabric.microsoft.com'
databaseName: 'ADS_GSC_DATAMART_WH' # Replace with your actual Fabric warehouse name
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'GSC-NONP-->PRO-ADSCLOUD-001-D-SC' # Replace with your Azure DevOps service connection
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
echo "Logging in with service principal..."
az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey --tenant $tenantId
echo "Testing connection to Fabric SQL endpoint..."
sqlcmd -S $sqlEndpoint -d $databaseName -G -Q "SELECT TOP 1 * FROM INFORMATION_SCHEMA.TABLES"
env:
servicePrincipalId: $(clientId)
servicePrincipalKey: $(clientSecret)
tenantId: $(tenantId)
sqlEndpoint: $(sqlEndpoint)
databaseName: $(databaseName)