VScode Synapse PROXY

[SnackSeeker91]

Hello,

Could someone tell me how to set a proxy for PySpark Lighter? I have everything installed correctly (VSCode, Synapse kernels, Python, Miniconda3), but I can't bypass the network settings with the proxy in PySpark Lighter. With the proxy set in VSCode settings, I can see tables, download files, and download notebooks, but I can't use the notebook via Synapse.

I know I have to set the proxy there because when I make an HTTP request without the proxy, I see a similar error, HTTPSConnectionPool. With the proxy settings, I can even manually ping the SparkLighter server but with the wrong response of course.

I have another Synapse Fabric that is working well because there is no firm VPN but in this case i can work only with firm VPN.

Thank you

11:30:23,724 root INFO Using synapse remote kernel ...
11:30:23,724 root INFO Should attach session in dev mode False
11:30:24,267 root INFO Starting session 8e46...
11:30:24,896 root INFO Getting refresh token...
11:30:25,312 root ERROR Failed to initialize Spark Lighter variables. HTTPSConnectionPool(host='wabi-north-europe-k-primary-redirect.analysis.windows.net', port=443): Max retries exceeded with url: /metadata/v201606/generatemwctoken (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
11:30:25,312 root INFO Registering Spark Lighter magics for IPython...

[SnackSeeker91]

Hello @v-shex-msft, thank you very much for your reply. I have set the global proxy with setx for HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. The connection and session are now working, but there's another problem with authentication. I can't read the tables in my OneLake.

Here is the log from the SparkLighter after session is connected.

Caused by: com.microsoft.aad.msal4j.MsalInteractionRequiredException: AADSTS50076: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '00000009-....'. Trace ID: 6f682a... Correlation ID: 7a7d9c... Timestamp: 2024-10-28 12:22:25Z
at com.microsoft.aad.msal4j.MsalServiceExceptionFactory.fromHttpResponse(MsalServiceExceptionFactory.java:39) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.TokenRequestExecutor.createAuthenticationResultFromOauthHttpResponse(TokenRequestExecutor.java:96) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.TokenRequestExecutor.executeTokenRequest(TokenRequestExecutor.java:37) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.AbstractClientApplicationBase.acquireTokenCommon(AbstractClientApplicationBase.java:128) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:63) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:59) ~[msal4j-1.10.1.jar:1.10.1]
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:17) ~[msal4j-1.10.1.jar:1.10.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:1.8.0_311]
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source) ~[?:1.8.0_311]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:1.8.0_311]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source) ~[?:1.8.0_311]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:1.8.0_311]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:1.8.0_311]

and notebook exception:

(Querry and lakehouse are in the same Worspaces)

Thanks in advance for any help. Best regards.

[v-shex-msft · Accepted solution]

HI @SnackSeeker91,

It seems mention that you are tried to use this data in multiple data regions. For this scenario, you can take a look the following link about multi-geo support of fabric capacity:

Multi-Geo support for Fabric - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

[SnackSeeker91]

Hi @v-shex-msft ,

You are right we have different locations for Datalke and Azure so we need to migrate to the same location. After that it should work without hitting MSAL. Thank you very much for you help.

Best regards.

[v-shex-msft]

Hi @SnackSeeker91,

Have you tied to enable global proxy on your system instead of specific app? They should simple to use than manually setting program level proxy.
BTW, I check the official document but not found some of them mention about enable proxy in the vscode extension.

Regards,

Xiaoxin Sheng

[SnackSeeker91]

Hello @v-shex-msft, thank you very much for your reply. I have set the global proxy with setx for HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. The connection and session are now working, but there's another problem with authentication. I can't read the tables in my OneLake.

Here is a log from the SparkLighter:

and here is the error from the jupyter notebook:

(Querry and lakehouse are in the same workspace)

Thanks in advance for any help. Best regards.