Forum Discussion

cdj's avatar
cdj
Frequent Visitor
6 months ago

Fabric Pipeline Warehouse connection problem

In my data pipeline I have a look up activity that runs a stored procedure on my Warehouse. About once every 14 days i get this error:

 

ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: '????.datawarehouse.fabric.microsoft.com', Database: '???', User: ''. Check the connection configuration is correct, and make sure the SQL Database firewall allows the Data Factory runtime to access.,Source=Microsoft.DataTransfer.Connectors.MSSQL,''Type=Microsoft.Data.SqlClient.SqlException,Message=The operation could not be completed because the system was updated or access to the encryption key was lost. Please confirm that your customer-managed key (CMK) in key vault is active, then sign in again and retry.,Source=Framework Microsoft SqlClient Data Provider,'

 

I have tried taken ownership of the warehouse and I have tried making a new connector. So far nothing seems to work. Has anyone else run into this problem? Did you manage to fix it?

7 Replies

  • Hi cdj,

    This issue occurs when Microsoft Fabric Warehouse is temporarily unavailable due to system updates or backend reconfiguration, rather than from permissions or connector settings. During these times, the SQL endpoint may briefly lose connectivity, leading to temporary errors like system update or shutdown in progress. Because Lookup activities do not retry automatically, enabling retry logic (3–5 retries with 2–5 minute intervals) can help the Warehouse recover on its own.

    Warehouse Connectivity - Microsoft Fabric | Microsoft Learn

    Troubleshoot the Warehouse - Microsoft Fabric | Microsoft Learn

     

    Thank you.

  • HI cdj,

    Checking in to see if your issue has been resolved. let us know if you still need any assistance.

     

    Thank you.

  • Hi,

    Based on the error details you shared, the main issue appears to be related to temporary connectivity loss between the Fabric Pipeline runtime and the Warehouse SQL endpoint, rather than a permanent connector or permission configuration problem.

    From the error message:

    • SqlFailedToConnect – Cannot connect to SQL Database

    • Inner SQL exception:
      “The operation could not be completed because the system was updated or access to the encryption key was lost…”

    This typically indicates one of the following scenarios:

    1. Temporary Warehouse unavailability due to backend update or service reconfiguration on the Fabric platform.

    2. If you are using Customer-Managed Keys (CMK), there may be a temporary loss of access to the encryption key (for example: Key Vault access policy issue, key disabled, identity permission issue, or key rotation timing).

    Since you mentioned this happens roughly once every ~14 days, this strongly suggests a platform-side maintenance or reconfiguration window, rather than a persistent configuration problem.


    Recommended Mitigation Steps

    1️⃣ Add Retry Logic to the Lookup Activity

    Lookup activities do not always handle transient failures automatically.
    I recommend configuring:

    • Retry count: 3–5

    • Retry interval: 2–5 minutes

    This usually allows the Warehouse endpoint to recover automatically if the issue is caused by temporary backend operations.


    2️⃣ If CMK Is Enabled – Validate Key Vault Configuration

    If your Warehouse is encrypted with CMK, verify:

    • CMK is Enabled / Active in Azure Key Vault

    • No recent key rotation caused permission drift

    • Managed Identity or Service Principal used by Fabric still has:

      • Get

      • Unwrap Key

      • Wrap Key
        permissions


    3️⃣ If Issue Persists – Open a Microsoft Support Ticket

    Since the issue is periodic, providing timestamps helps backend engineers correlate with platform events.

    You can create a support ticket here:
    Create a Fabric and Power BI Support Ticket
    https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket


    Useful Microsoft Documentation

    Fabric Warehouse Connectivity
    https://learn.microsoft.com/en-us/fabric/data-warehouse/connectivity

    Fabric Warehouse Troubleshooting Guide
    https://learn.microsoft.com/en-us/fabric/data-warehouse/troubleshoot-fabric-data-warehouse


    If you can confirm whether CMK is enabled on your Warehouse and whether retries are already configured on the Lookup activity, we can narrow this down further and validate whether this is fully transient or configuration-related.

    Hope this helps.