Forum Discussion
Helper IError in VS Code notebook when trying spark.sql()
Hi GammaRamma ,
I have the following suggestions for code that can be executed perfectly directly in Fabric, but fails to execute in VS Code:
Ensure that the environment configuration in VS Code matches the environment in the fabric. This includes versions of the VS Code extensions for JDK, Miniconda, and Synapse.
Ensure that users have access to lakehouse and tables.
Why use Run time 1.1 instead of 1.3?
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support Team
If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
GammaRammaAnonymous thanks for your reply.
I don't have synapse 1-3 runtime in VS code, but I do have 1-2, which gives another error for the same code:
Py4JJavaError: An error occurred while calling o33.sql. : org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace
Since posting my first message, I have tried other Fabric notebooks in VS Code that use spark.sql() and found that a couple of them actually work. So I compared the config files of the notebooks that work with the ones of the notebooks that don't work, but I found no significant differences. I compared .ressource-info, .artifact-info and lighter-config.json.
The notebooks are all in the same Fabric Workspace.I copied and pasted the code from the notebooks that don't work into the ones that work and it worked, so it's not some particularity of the code that's the problem either.
I created new notebooks but they didn't work with the same code.
It seems pretty random.
I don't know what else to look at.
Thank you