How to Migrate An Existing DataGateway with PowerShell

[buraktas]

Hi there,

I have a PowerShell script that downloads and installs an On-Premise Data Gateway on an EC2 AWS instance. The script runs each time when this instance is restarted or a new instance is provisioned. So, it checks if the DataGateway exists, and if not it tries to create a new one. The existing DataGateway always stays offline when the instance is totally replaced even though Get-DataGatewayCluster api returns the existing data gateway information. I assume I need to migrate existing data gateway into the instance but I couldn't find a related API for that even though the Data Gateway Windows Application has that option. Do you happen to know if there is a way to migrate an existing data gateway through a REST API or PowerShell cmdlet?

The script is very much like this one https://github.com/daltskin/PowerBIGateway/blob/52743a6bcb67d52d8e494f5c94f7e12f3ab1cc28/scripts/pbi...

[lbendlin]

Out of curiosity - what is your reasoning for using a single cluster gateway?

[buraktas]

I am pretty new to Microsoft environment and PowerBI, and I have to setup a DataGateway. Not because I mainly prefer to proceed with single cluster gateway. But as I understand, I will face the same issue anyway even though I have setup a multi cluster gateway too.

[lbendlin]

I don't know the background or the cost considerations of "each time when this instance is restarted or a new instance is provisioned"  but I would look for something more permanent to avoid having to redo the gateway all the time.  If you have multiple cluster members the load balancer will give the job to whichever cluster member is available, without a business interruption.

[buraktas]

So, yes having a cluster mode with a load balancer is the best solution for resiliency and high-availability for sure. However, same problem will occur if we would like to have a code pipeline with a basic blue/green deployment. Instances might get replaced when there is a software update or even die. What I see microsoft doesn't provide a simple solution to fully automate this process easily which always requires a manual intervention.

[lbendlin]

You can always add/remove cluster members from the logical configuration (regardless of their physical status).  Where it gets (extremely) hairy is when you remove the last ("anchor") cluster member.  That's when your entire gateway goes "poof".

[buraktas]

Yeap, adding/removing members came into my mind after you introduced cluster mode idea, but that poof moment will put me in a hard situation. I think that can be avoided easily though by removing an offline member after adding the new one. Nevertheless, the annoying part is there is no any REST api or cmdlet that allows migrating a data gateway...

[lbendlin]

If this is important to you please consider voting for an existing idea or raising a new one at https://ideas.fabric.microsoft.com

[tharunkumarRTK]

@buraktas 

Normally, while taking over an existing gateway and setting it up in a new server, we disable it from the service and then we take it over in the new server while installing. This approach worked. 
So I would suggest you to add the following command to disable it

Set-OnPremisesDataGateway

and then for installation your can leverage -overwriteexsitinggateway flag

ConvertFrom-SecureString -SecureString (Read-Host "Enter Recovery Key" -AsSecureString) | Out-File -FilePath .\encryptedRecoveryKey.txt
$secureRecoveryKey = (cat .\encryptedRecoveryKey.txt | ConvertTo-SecureString)
Add-DataGatewayCluster -RecoveryKey $secureRecoveryKey -Name "MyNewGateway" -RegionKey brazilsouth -OverwriteExistingGateway

I have not tried it from poweshell but I believe it works.

Need a Power BI Consultation? Hire me on Upwork

Connect on LinkedIn

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

[buraktas]

hey @tharunkumarRTK thanks for help. I tried the Add-DataGatewayCluster cmdlet before with -OverwriteExistingGateway option before which returns Add-DataGatewayCluster: Gateway data-gateway-test already exists error. From my understanding, overwrite flag is only for overriding the current gateway with a new name which is not the automated case that I am looking for. On the other hand, I think Set-DataGatewayCluster is not the cmdlet I am looking for because it only provides updating a couple of attributes. By the way, I am using PowerShell 7.4.4 which has different cmdlets than 5.1.

[tharunkumarRTK]

@buraktas 

Good to know that you have already tried Add-DataGatewayCluster. But I am wondering if you have tried it after disabling the gateway. As per the documentation below command allows to disable the gateway. I understand that it only works with PowerShell 5, but as long as it solves the problem I believe you are ok to use PowerShell 5. 

https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-powershell-support

Need a Power BI Consultation? Hire me on Upwork

Connect on LinkedIn

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

[buraktas]

hey @tharunkumarRTK I installed the OnPremisesDataGatewayHAMgmt module in PowerShell 7.4 too, but I can't find any reference documentation about the example usage anywhere. get-help command returns the description of the API. There is nothing specified for disabling a gateway. Maybe it is MemberStatus field? And even if it is I can't find acceptable MemberStatus values.

PS C:\Windows\System32> get-help Set-OnPremisesDataGateway -Full

NAME
Set-OnPremisesDataGateway

SYNOPSIS
Set gateway info.


SYNTAX
Set-OnPremisesDataGateway [-ClusterObjectId] <Guid> [-GatewayObjectId] <Guid> [[-MemberStatus] <String>] [[-GatewayContactInformation] <String>] [[-Name] <String>] [<CommonParameters>]


DESCRIPTION
This function set gateway info and return the updated info, note it won't update the field if it's value is null.

[lbendlin]

Is this a single cluster gateway?  You  need to distinguish between the logical configuration on the service (which doesn't care about the presence or absence of physical entities) and the physical configuration of the cluser members.  Sounds like in your scenario you need to take over as anchor each time you recreate your instance.

[buraktas]

True, it is a single cluster gateway. I am aware of the fact that I need to take over each time the server is replaced, and I can do that from the On-Premises Data Gateway UI manually. But my purpose is automating this via PowerShell if possible.