Forum Discussion
Notebook Resources | Spark not recognized in local machine | File not recognized in local machine
Serogio
hey..sorry for replying a little bit late.
again, really appriecate the feedback of the Synapse VS Code experience and I am more than happy to run into a call to discuss further, feel free to reach out to me via: [email protected].
for the "spark" variable issue, there is the quick workaround that might help to address the warning.
The pre-run code I mentiond actually stay in the local desktop, too. It is sub-folder under your Home directory as: .ipython/profile_default/startup/init_lighter.py.
so in your own .py file, you can import that "init_lighter.py" module.
import sys
import os
home_directory = os.path.expanduser("~")
directory_to_append = os.path.join(home_directory, ".ipython/profile_default/startup/")
sys.path.append(directory_to_append)
from init_lighter import spark
print("Spark version: " + spark.version)
now within you own .py file, you can call the method from the "Spark" variable without any warning from Python interperter.
for the second issue, one thing need to call out in current release, ONLY the spark code will be posted/synced to the remote workspace for the execution, the pure python code would be still execute locally in your desktop, that explain why the you will need a local folder path macth to the lakehosue log.
qixiao
Hi QixiaoWang !
No worries about the late reply. I’m glad I can contribute for the VS code experience in Fabric and, thanks for the time you put into trying to figure out these issues.
Regarding Spark session invocation:
As a quick test, I tried to invoke the spark session inside my notebook, before even going into my modules, but apparently the init_lighter.py module is not being recognized, as you can see in image below:
although it was initialized when I opened the notebook in my local machine, as you can see in the output console in above image, and also it does exist in my local machine as below image demonstrates:
Maybe, at this point, it’s just me being bad with Python, but you happen to have any clue on how to solve this one?
Regarding the logging file:
I understand what you mean, but it is still a bit odd to me that the Tables are recognized but the Files are not. I am just wondering, right now, what would be the best/advised practice to have in this case. Perhaps instead of a logging file, I should have a logging table, so I don’t have to replicate too many artifacts in my local machine, in order to avoid mistakes or maintenance complexity/overhead.
Again, thanks for the input and for your time. 🙂
Hi QixiaoWang !
I wonder if you had some time to have a look at my questions above?Please let me know.
Thank you.