Data Gateway PowerShell module issue with `Add-DataGatewayClusterMember` command `-RegionKey` param

[mikeeq]

Hi all,

I recently encountered an issue with the latest version of DataGateway PowerShell module (latest version of PS7 and DataGateway service): https://www.powershellgallery.com/packages/DataGateway/3000.196.145

PS C:\> (Get-Module DataGateway).version

Major Minor Build Revision
----- ----- ----- --------
3000 196 145 -1

PS C:\> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.3

PS C:\> (Get-Item "C:\Program Files\On-premises data gateway\Microsoft.PowerBI.EnterpriseGateway.exe").VersionInfo.FileVersion
3000.226.5

`Add-DataGatewayClusterMember` command doesn't seem to respect `-RegionKey` parameter, whenever it's passed to the command, it cannot find the GatewayClusterId in the specified region. 

# Info: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx is created in uksouth region
PS C:\> Add-DataGatewayClusterMember -GatewayClusterId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -GatewayName datagatewaypoc-2 -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -RegionKey uksouth

Add-DataGatewayClusterMember: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx was not found

When the cluster is deployed in a default PowerBI tenant region and when `-RegionKey` parameter is not passed then it works as it should.

# Info: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx is created in PowerBI default tenant (northeurope in that case)
PS C:\> Add-DataGatewayClusterMember -GatewayClusterId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -GatewayName datagatewaypoc-2 -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway


GatewayId GatewayObjectId GatewayName GatewayType
--------- --------------- ----------- -----------
xxxxxx xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx datagatewaypoc-2 Resource

Also, `Add-DataGatewayClusterMember` is not documented at all, the only mention is in the blog post regarding release notes from 2023.10:
https://learn.microsoft.com/en-us/powershell/module/datagateway/remove-datagatewayclustermember?view...

https://powerbi.microsoft.com/en-in/blog/on-premises-data-gateway-october-2023-release/

@dazhang

[v-jianpeng-msft]

Thank you @hackcrr 

Hi, @mikeeq 

I apologize for my late reply. I did a complete test based on the code you are using. During the test, the gateway module honors the region parameter. Here are the results of my tests:

First, I updated my PowerShell to version 7.4.3 using the following commands:

winget source reset --force
winget source update
winget install --id Microsoft.Powershell --source winget

I installed the gateway module using the following command:

Install-Module -Name OnPremisesDataGatewayHAMgmt
Install-Module -Name DataGateway

Import the gateway module in PowerShell and view the gateway version:

Import-Module DataGateway
(Get-Module DataGateway).version

Next log into my gateway account:

Login-DataGatewayServiceAccount

(Get-Item "C:\Program Files\On-premises data gateway\Microsoft.PowerBI.EnterpriseGateway.exe").VersionInfo.FileVersion

Use the following command to get the zone of the gateway to see which zone my current gateway is installed in:

Get-DataGatewayRegion

Add a gateway cluster member to the gateway region:

Add-DataGatewayClusterMember -GatewayClusterId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -GatewayName datagatewaypoc-2 -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -RegionKey uksouth

You can see that we were also able to add the gateway member without any problems when we specified the RegionKey uksouth.

Use your debug command below to test that the Power BI API automatically redirects to the specified region when you add a gateway cluster member:

PS C:\> Add-DataGatewayClusterMember -RegionKey uksouth -GatewayName "DataGatewayPoC-4" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -Debug -Verbose -GatewayClusterId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Use the following command to get the just added gateway cluster:

 get-dataGatewayCluster -RegionKey uksouth -Verbose -Debug

During this entire test, the gateway module worked fine. You can try to add the gateway cluster members again using PowerShell commands. Or you can perform the operation in another computer.
In addition to this, you can choose an alternative solution: add the gateway cluster using the local data gateway software.

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[mikeeq]

Hi @v-jianpeng-msft,

The problem is when you are trying to add a member to the cluster which is deployed in non-default region, as far as I can see you are adding the member to the cluster which is your default one - UK South, try another region 😉

[v-jianpeng-msft]

Hi, @mikeeq 

Thanks for your reply. As you mentioned, when I add my gateway member to another region, it fails to add.

Since my current main gateway is located in westus3, when I add members to this main gateway, I specify other regions, such as westeurope.
Then when we execute the current command, Power BI will try to search for the current main gateway in the westeurope region, but in fact, our main gateway does not exist in the current region, so 404 not found appears.

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[mikeeq]

Seems like API requests are missing "&region=westeurope" parameter at the end of URI, or region specific API endpoint is not filtering data (gateways in that case) per region.

PS C:\Users\vagrant> add-datagatewaycluster -RecoveryKey $secureRecoveryKey -Name DataGatewayPoC -OverwriteExistingGateway -RegionKey westeurope -Verbose -Debug
DEBUG: 12:18:57 - Add-DataGatewayCluster begin processing with ParameterSet __AllParameterSets.
DEBUG: 12:18:57 - Cmdlet version: 3000.196.145
VERBOSE: Request Uri: https://api.powerbi.com/v2.0/myorg/me/gatewayRegions
VERBOSE: Status Code: OK (200)
VERBOSE: Request Uri: https://wabi-west-europe-redirect.analysis.windows.net/v2.0/myorg/me/gatewayClusters?$expand=permissions,memberGateways
VERBOSE: Status Code: OK (200)
VERBOSE: Request Uri: https://wabi-west-europe-redirect.analysis.windows.net/unifiedgateway/gateways/CreateGatewayWithApps
VERBOSE: Status Code: OK (200)
DEBUG: Microsoft.PowerBI.DataMovement.ExternalContracts.API.CreateGatewayResponse

DEBUG: 12:19:02 - Add-DataGatewayCluster end processing.
GatewayId GatewayObjectId                      GatewayName    GatewayType
--------- ---------------                      -----------    -----------
xxxxxxx   xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx DataGatewayPoC Resource

PS C:\> Add-DataGatewayClusterMember -RegionKey westeurope -GatewayName "DataGatewayPoC-2" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -Debug -Verbose -GatewayClusterId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
DEBUG: 12:19:51 - Add-DataGatewayClusterMember begin processing with ParameterSet __AllParameterSets.
DEBUG: 12:19:51 - Cmdlet version: 3000.196.145
VERBOSE: Request Uri: https://api.powerbi.com/v2.0/myorg/me/gatewayRegions
VERBOSE: Status Code: OK (200)
VERBOSE: Request Uri: https://wabi-west-europe-redirect.analysis.windows.net/v2.0/myorg/me/gatewayClusters?$expand=permissions,memberGateways
VERBOSE: Status Code: OK (200)
Add-DataGatewayClusterMember: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx was not found
DEBUG: System.ArgumentException: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx was not found
   at Microsoft.DataMovement.Powershell.Commands.DataGateway.AddDataGatewayClusterMember.ExecuteCmdletAsync()
   at Microsoft.DataMovement.Powershell.Commands.Common.PowerBICmdlet.ProcessRecordAsync()
DEBUG: 12:19:52 - Add-DataGatewayClusterMember end processing.

PS C:\Users\vagrant> get-dataGatewayCluster -RegionKey westeurope -Verbose -Debug
DEBUG: 12:29:37 - Get-DataGatewayCluster begin processing with ParameterSet __AllParameterSets.
DEBUG: 12:29:37 - Cmdlet version: 3000.196.145
VERBOSE: Request Uri: https://api.powerbi.com/v2.0/myorg/me/gatewayClusters?$expand=permissions,memberGateways&region=westeurope
VERBOSE: Status Code: OK (200)

Id                    : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Name                  : DataGatewayPoC
Description           :
Permissions           : {app-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, app-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,
                        xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}
Type                  : Resource
MemberGateways        : {DataGatewayPoC}
Datasources           : {}
LoadBalancingSettings :
AllowableOptions      : Microsoft.PowerBI.ServiceContracts.Api.ODataDictionary`1[System.Boolean]
StaticCapabilities    : 0
MaxPermission         : Microsoft.PowerBI.ServiceContracts.Api.Permission

DEBUG: 12:29:38 - Get-DataGatewayCluster end processing.

[hackcrr]

Hi, @mikeeq 

Debug logs indicate that the Add-DataGatewayClusterMember command does not region to include this parameter in its API request URI, while the Get-DataGatewayCluster command does.
RegionKey As a temporary workaround, you might consider deploying the gateway in a default region that doesn't require the parameter, as you mentioned, it works fine in this case. Since this Get-DataGatewayCluster command region correctly includes the parameter, you can use it to verify the gateway cluster and its region. Then, try adding cluster members without specifying a region.
This is a potential scripted workaround:

# Get Gateway Cluster Details with RegionKey
$gatewayClusterId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
$gatewayCluster = Get-DataGatewayCluster -RegionKey "westeurope" -Verbose -Debug

if ($gatewayCluster) {
    Write-Output "Gateway cluster found: $($gatewayCluster.Name) in region $($gatewayCluster.Region)"
} else {
    Write-Output "Gateway cluster not found."
}

# Attempt to add a cluster member without RegionKey
try {
    Add-DataGatewayClusterMember -GatewayClusterId $gatewayClusterId -GatewayName "DataGatewayPoC-2" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -Verbose -Debug
} catch {
    Write-Output "Error encountered: $_"
    # Log error details for further analysis
}

If this workaround doesn't work or you don't want to use it, you can just report the error. Below is a sample template you can use to report a problem:

Subject: Bug Report: Add-DataGatewayClusterMember Command Ignoring -RegionKey Parameter in DataGateway Module

Description:
I am encountering an issue with the Add-DataGatewayClusterMember command in the latest DataGateway PowerShell module (version 3000.196.145). When using the -RegionKey parameter, the command fails to find the specified GatewayClusterId in the region. The -RegionKey parameter works correctly with other commands, such as Get-DataGatewayCluster.

Steps to Reproduce:
Create a Gateway Cluster in the westeurope region.
Attempt to add a cluster member using the Add-DataGatewayClusterMember command with the -RegionKey parameter.

# Add Gateway Cluster
Add-DataGatewayCluster -RecoveryKey $secureRecoveryKey -Name DataGatewayPoC -OverwriteExistingGateway -RegionKey westeurope -Verbose -Debug

# Add Cluster Member
Add-DataGatewayClusterMember -RegionKey westeurope -GatewayName "DataGatewayPoC-2" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -Debug -Verbose -GatewayClusterId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Debug Logs:

# Excerpt from Add-DataGatewayClusterMember debug logs
VERBOSE: Request Uri: https://api.powerbi.com/v2.0/myorg/me/gatewayRegions
VERBOSE: Status Code: OK (200)
VERBOSE: Request Uri: https://wabi-west-europe-redirect.analysis.windows.net/v2.0/myorg/me/gatewayClusters?$expand=permissions,memberGateways
VERBOSE: Status Code: OK (200)
Add-DataGatewayClusterMember: Gateway cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx was not found

Expected Behavior:
The Add-DataGatewayClusterMember command should respect the -RegionKey parameter and correctly find the GatewayClusterId in the specified region.

Actual Behavior:
The command fails with an error message indicating that the GatewayClusterId was not found.

hackcrr

If this post helps, then please consider Accept it as the solution and kudos to this post to help the other members find it more quickly

[hackcrr]

Hi, @mikeeq 

First, ensure that you are using the latest version of the DataGateway module. You can update the module using the following command:

Update-Module DataGateway

Double-check the syntax and ensure that the

-RegionKey

 parameter is correctly specified. PowerShell parameters are case-sensitive, so make sure you're using the exact region key.

As a temporary workaround, you can try deploying the gateway cluster in the default Power BI tenant region if possible. If this is not feasible, continue using the manual process without the RegionKey parameter until a fix is available.

Here are examples of how you might use the commands with and without the -RegionKey parameter:

Without -RegionKey Parameter (Default Region)

$secureRecoveryKey = ConvertTo-SecureString "YourRecoveryKey" -AsPlainText -Force
Add-DataGatewayClusterMember -GatewayClusterId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -GatewayName "datagatewaypoc-2" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway

With -RegionKey Parameter (Specific Region)

$secureRecoveryKey = ConvertTo-SecureString "YourRecoveryKey" -AsPlainText -Force
Add-DataGatewayClusterMember -GatewayClusterId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -GatewayName "datagatewaypoc-2" -RecoveryKey $secureRecoveryKey -OverwriteExistingGateway -RegionKey "uksouth"

Try using the Get-DataGatewayCluster command with the -RegionKey parameter to see if it works correctly:

Get-DataGatewayCluster -RegionKey "uksouth"

If the -RegionKey parameter works with other commands, it indicates that the issue is specific to the Add-DataGatewayClusterMember command.

hackcrr

If this post helps, then please consider Accept it as the solution and kudos to this post to help the other members find it more quickly

[mikeeq]

"ensure that you are using the latest version of the DataGateway module."

It's mentioned in the first sentence that I'm using the latest version of the DataGateway module.

"If the -RegionKey parameter works with other commands, it indicates that the issue is specific to the Add-DataGatewayClusterMember command."

`-RegionKey` parameter works with other commands without any issues, there's only a problem with `Add-DataGatewayClusterMember` and maybe wth `Remove-DataGatewayClusterMember`, but as `add member` command doesn't work, I wasn't able to test the `remove member` command

[hackcrr]

Hi, @mikeeq 

You can create a ticket by clicking the link below, which will have a dedicated engineer testing the current bug.

https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

hackcrr

If this post helps, then please consider Accept it as the solution and kudos to this post to help the other members find it more quickly

[v-jianpeng-msft]

Hi, @mikeeq 

Do you have the GatewayClusterId used by the check, and the gateway cluster is indeed deployed in that region. An incorrect GatewayClusterId is causing the current issue.

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[mikeeq]

Hi @v-jianpeng-msft,

GatewayClusterId is for sure correct, I double-checked that multiple times, I'm pretty sure it's a bug in the PS module itself.