How get connection string in Spark for on premise/local db connection by gateway?

[marcoG]

I tried:

mssparkutils.credentials.getFullConnectionString("<connectionName>")

but i have:

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getFullConnectionString. : java.util.NoSuchElementException: spark.arcadia.session.token at org.apache.spark.SparkConf.$anonfun$get$1(SparkConf.scala:266) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.SparkConf.get(SparkConf.scala:266) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.$anonfun$getAccessTokenInternal$3(TokenLibrary.scala:271) at scala.util.Try$.apply(Try.scala:213) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getAccessTokenInternal(TokenLibrary.scala:270) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.$anonfun$getAccessTokenInternal$4(TokenLibrary.scala:303) at scala.util.Try$.apply(Try.scala:213) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getAccessTokenInternal(TokenLibrary.scala:291) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getAccessTokenAsync(TokenLibrary.scala:216) at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$.getFullConnectionString(TokenLibrary.scala:1216) at mssparkutils.credentials$.getFullConnectionString(credentials.scala:49) at mssparkutils.credentials.getFullConnectionString(credentials.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CalCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750)

[marcoG]

Hi @HimanshuS-msft ,

I was trying this solution as dataflow gen2 from on premise db through gateway is slow and not working.

Dataflow gen1 works but doesn't bring data to the lakehouse.

On-premise pipelines are not yet available

Do you have any other alternatives?

Thanks

[HimanshuS-msft]

HI @marcoG 

Thanks for using the Fabric community .

As I understand the ask here is to access the loacaldb from the Spark code . I don't thiink its possible as of now .

I think you should try out to use Dtaflow to copy the data to Lakehouse and then reference that in the notebook .
This document should help .
https://learn.microsoft.com/en-us/fabric/data-engineering/load-data-lakehouse#considerations-when-ch...

Thanks
HImanshu

[marcoG]

There is some update?