Forum Discussion

c_lovestrom's avatar
c_lovestrom
Frequent Visitor
9 months ago
Solved

What if a gateway update causes errors?

My manager is concerned that one day we'll update our on-premise data gateway that the majority of our power bi service connections use, and the update will have errors that will cause all our Semant...
Need Help
  • lbendlin's avatar
    lbendlin
    9 months ago

    As you can see from my screenshot there can be many, many corrections issued between regular releases.  September 2025 had five versions, for example, and the latest August version was sneakily inserted AFTER the first September version was released. 

     

    Of course you can take your own conclusions from that but we have been badly burned in the past by this, and are now extra extra cautious, updating a canary first and then waiting how it behaves.  There are tons of feature changes between versions, nearly none of them documented (the latest 0.20 Spark drama for example).  

     

    There be dragons.

Sukan1719's avatar
Sukan1719
Regular Visitor
8 months ago

c_lovestrom  Your manager is right. If an update introduces a bug (which has happened in the past), there is no simple "Undo" button in the interface to revert to the previous state.

However, there is a very viable, standard industry approach to solving this. You don't need to wait for Microsoft to fix it if you implement a Staging Strategy and a Disaster Recovery Plan.

Here is the breakdown of how to architect this safely.

1. The "Canary" Method: A Dev/Test Gateway

The most effective way to test updates before they hit production is to establish a separate, non-production Gateway.

  • How it works: You set up a Gateway on a separate server (or VM) specifically for development or testing.

  • ** The Process:**

    1. When a new Gateway version is released, install it on the Dev Gateway first.

    2. Have a specific Power BI workspace configured to use this Dev Gateway with a few critical Semantic Models (datasets) that represent your typical data sources (SQL, Oracle, etc.).

    3. Run refreshes on these test models.

    4. If the refreshes succeed and the data looks correct, wait 24–48 hours to ensure stability.

    5. Only then apply the update to the Production Gateway.

2. The Rollback Plan (The "Time Machine")

Your manager is doubtful about reversing updates. He is technically correct that there is no "Rollback" button, but you can manually revert if you prepare correctly.

The Golden Rule: Always keep the installation file (GatewayInstall.exe) of your current working version before you update.

How to revert if an update fails:

  1. Uninstall: You must completely uninstall the broken, new version of the Gateway from the server via Windows "Add or remove programs."

  2. Reinstall: Run the installer for the previous version (which you saved).

  3. Restore: During installation, it will ask if you want to register a new gateway or Restore / Takeover an existing gateway. Choose Restore.

  4. Key Entry: You will be prompted for your Gateway Recovery Key. Note: If you do not have this key saved, you cannot restore the gateway, and you will have to create a new one from scratch and re-map all data sources in Power BI Service.

Crucial Requirement: Ensure your Recovery Key is saved in a secure password manager. Without it, a rollback is impossible.

3. High Availability (HA) Clusters

To prevent downtime during the update process itself, you should use Gateway Clustering.

  • Concept: You install the Gateway on two or more servers and join them to the same Gateway "Cluster."

  • The Update Workflow:

    1. Take Node A out of the load balancer (or just let the cluster handle it).

    2. Update Node A.

    3. If Node A comes back online and works, update Node B.

    4. If Node A fails, Node B is still running the old version and handling the traffic.

Note: HA protects against the server going down, but it is less effective against "bad updates" than the Dev/Test Gateway method. If you update Node A and it reports "Healthy" to the service but creates data errors due to a bug, the Cluster might still route queries to it. Therefore, Strategy #1 (Dev/Test Gateway) is superior for detecting bugs.

Summary Recommendation for Your Manager

To satisfy your manager's request for safety, propose this standard operating procedure (SOP):

The "Safe-Update" Protocol

  1. Maintain an Archive: Create a folder on your network drive containing the installer files for the last 3 months of Gateway releases. (e.g., Gateway_Install_Sept2024.exe, Gateway_Install_Oct2024.exe).

  2. Verify Recovery Keys: Confirm that the Gateway Recovery Key is known and accessible.

  3. Implement the 3-Day Rule:

    • Day 1: Microsoft releases update. Do not install. Download it and install it on the Test Gateway.

    • Day 2: Check Test Gateway refreshes. Check the Power BI Community forums to see if other users are reporting massive failures.

    • Day 3: If tests pass and forums are quiet, install the update on the Production Gateway.