Copy activity from Oracle to Lakehouse fails without staging

[vietng]

Hello everyone,

I have Oracle on-prem → Gateway → Fabric Lakehouse setup. I try to create a simple Copy Activity in Data pipeline to read 1 table (just 3 records) from Oracle and write to Lakehouse table.

This pipeline always fails with this error: "ErrorCode=LakehouseOperationFailed An existing connection was forcibly closed by the remote host". Then I turn the "Enable staging" on with the option Data store type is Workspace; the pipeline works perfectly fine.

Question: Why does direct copy fail (even with only 3 records), but staging works?

P/s: Gateway already opened these endpoints/ports:

*.download.microsoft.com | 80
*.powerbi.com | 443
*.analysis.windows.net | 443
*.login.windows.net, login.live.com, and aadcdn.msauth.net | 443
*.servicebus.windows.net | 5671–5672
*.servicebus.windows.net | 443 and 9350–9354
*.frontend.clouddatahub.net | 443
*.core.windows.net | 443
login.microsoftonline.com | 443
*.msftncsi.com | 80
*.microsoftonline-p.com | 443
*.dc.services.visualstudio.com | 443

[deborshi_nag · Accepted solution]

Hi @vietng Even if you're choosing a Lakehouse it may still be using the SQL Endpoint behind the scenes, since I'm guessing you're writing in the /Tables area. Trying opening the 1433 port and see if it works. 

[v-prasare]

Hi @vietng,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

[v-prasare]

Hi @vietng,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

[tayloramy · Accepted solution]

Hi @vietng, 

One small correction, if you're targeting a lakehouse, it wouldn't be using the SQL endpoint to write data as the sql endpoint is read only, but @deborshi_nag is correct that it might still be using the SQL ports to do the data movement.  

I always recommend opening all the ports found on this page when doing gateway maintenance: 
Adjust communication settings for the on-premises data gateway | Microsoft Learn

[deborshi_nag · Accepted solution]

Hi @vietng Even if you're choosing a Lakehouse it may still be using the SQL Endpoint behind the scenes, since I'm guessing you're writing in the /Tables area. Trying opening the 1433 port and see if it works. 

[deborshi_nag]

Hello @vietng 

A Staged Copy is necessary when port 1433 isn't open for SQL connections. Since ticking "Enable Staging" resolves the issue, it suggests that port 1433 is needed for your data copy. Usually, copying data to a Fabric Lakehouse or OneLake doesn't require outbound port 1433. Are you working with a SQL Warehouse or using a SQL Endpoint instead of a Lakehouse ? That might be why you're experiencing this issue.

You might want to whitelist this and see if that works. 

*.datawarehouse.pbidedicated.windows.net | 1433

[vietng]

Hi @deborshi_nag 
Thank you for your support. About your question, I use the Lakehouse connection in the Destination settings (select Browse all, then choose my Lakehouse in the list), so it can't be the SQL Warehouse or SQL Endpoint right?