Virtual Network Gateways not publishing data into Lakehouse

Hi Fabric Community,

I have a virtual gateway which is able to read data from an Azure SQL database through Gen2 Dataflow however will not then publish the data into a destination, which is a Lakehouse. I can view the data inside the workflow but cannot get it to the destination inside fabric. If I open up the dataflow itself and try to add the connection, this is the error.

When I click the red error triangle outside of the dataflow (in the workspace where the dataflow lives), this is the error I get.

Dataflows

pochuang
2 years ago
Hi LouisSh ,

We checked trace with the ReqeustId you provided and noticed that your gateway cannot resolve api.powerbi.com. It seems that there is an incorrect DNS record in the private DNS zone probably becauase you had a PowerBI privatelink setup that hasn't being cleaned up correctly.

In the future, you can get more insight of the failure by checking the diagnostic logs (Download logs on the VNet data gateway | Microsoft Learn), and use the network troubleshooting tool for networking problems (Troubleshoot a virtual network (VNet) network | Microsoft Learn).

Thanks.

8 Replies

v-cboorla-msft
Microsoft Employee
2 years ago
Hi LouisSh

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

The error message suggests that the dataflow is having trouble communicating with the Fabric endpoint via the on-premises data gateway. Is your on-premises data gateway machine behind a firewall, and, if so, can you check your firewall logs to see if there was a failed connection attempt made to https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails?

For more details you can refer to this thread: Problem to import on premise data onto a lakehouse using a DataflowGen2

I hope this information helps. Please do let us know if you have any further questions.

Thanks
- LouisSh
  Frequent Visitor
  2 years ago
  Hi v-cboorla-msft
  
  Just to confirm, this is a virtual gateway that is being used and not an on premise. There is no firewall for this. How can I check the firewall logs anyways for virtual gateways, is this in the Azure portal?
  - v-cboorla-msft
    Microsoft Employee
    2 years ago
    Hi LouisSh
    
    I have tried to repro the same scenario but didn't find any difficulty in publishing the data into a destination which is a Lakehouse using Dataflow Gen2. Please find the screenshots for reference.
    
    You can also try connecting to the Azure SQL database uisng onprem-gateway.
    
    For more details please refer to the below mentioned links:
    
    Install an on-premises data gateway.
    
    How to access on-premises data sources in Data Factory for Microsoft Fabric.
    
    On-premises data gateway considerations for data destinations in Dataflow Gen2.
    
    I also tried connecting using on-prem gateway:
    
    Please try connecting as stated above and let us know if you have any further issues.
    
    This might seems to be an intermittent issue. Can you please try again after a day and let me know if you are still facing this issue?
    
    Hope this helps.
    
    Thanks.
SidJay
Microsoft Employee
2 years ago
Hi LouisSh,

Could you please confirm that your VNet is set up to allow these endpoints?

Protocol: TCP
Endpoints: *.api.powerbi.com, *.datawarehouse.pbidedicated.windows.net, *.datawarehouse.fabric.microsoft.com, *.dfs.fabric.microsoft.com
Port: 1433
Thanks
- v-cboorla-msft
  Microsoft Employee
  2 years ago
  Hi LouisSh
  
  We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
  In case if you have any resolution please do share that same with the community as it can be helpful to others.
  Otherwise, will respond back with the more details and we will try to help.
  
  Thanks.
  - v-cboorla-msft
    Microsoft Employee
    2 years ago
    Hi LouisSh
    
    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
    If you have any question relating to the current thread, please do let us know and we will try out best to help you.
    In case if you have any other question on a different issue, we request you to open a new thread.
    
    Thanks.