Forum Discussion
Notebook Resources | Spark not recognized in local machine | File not recognized in local machine
Serogio
for the first question related to the spark variable, in fabric notebook, before its execution, there would be some pre-run code executed by the system to define this "spark" variable as the runtime context of spark, so you dont need to manually define any spark session. given this pre-run code only execute when running the notebook, that explain why it is notrecognized in your own .py file. I think you can just ignore this error.
for your second quesiton about the file path, we might need some time to work on the solution, as workground to unblock you, could you please manually create a folder as the error require?
thanks
qixiao
Hi QixiaoWang !
Thanks for your input.
Correct me if I'm wrong, but I beleive you are one of the authors os the documentation about Notebook Resources and. If you are, then it makes you one of the best persons to answer my questions (not diminuishing the importance of the work of your other collegues, of course).
Am I right? =]
Anyway, to the points:1 - Spark session question:
I think one of the main purposes of having VS Code associated to notebooks, is to be able to develop my solutions using my local VS code development environment. Using this development environment is more convient than using MS Fabric UI to directly develop in there and, also, I am able to create modules that are part of my solution not having to resort to MS Fabric environments due to the general overhead they represent when managing packages (and I am refering directly to the mandatory rule of having to define wheel files to deal with that).
I think ignoring the error is not an option, because ideally I would like to develop things in my machine, provided I am synchronized with my data lake tables and files, as it makes me more productive.
2 - 'No such file or directory' error:
With your input, I came to realize that, indeed, the file does not exist physically in my machine but it does exist on my data lake, as you can see in image below:
By reading through the VS code integration documentation, I understood that the notebook was synched with my datalake and, therefore, I wouldn't need to have the tables and files form my data lake in my local machine.
Perhaps I interpreted that wrongly and I know that Fabric is on it's early stages and continuously evolving, but would'nt it make sense to not have the files physically present in my machine?
3 - Bonus question:
All in all, I guess one of the perks of working with VS code and data lake synchronization, is to be able to have a familiar and productive environment to develop our solutions. Question here is: reading through my questions, please let me know if I'm doing something wrong in what the development setup is concerned, because my general purpose is to find the ideal setup to develop with Fabric.
Worth noting that I have been in contact with your support teams, which are also looking into this (questions 1 and 2 only).
I also have previously provided the support ticket number (please see message 9).
Thank you.
Serogio
hey..sorry for replying a little bit late.
again, really appriecate the feedback of the Synapse VS Code experience and I am more than happy to run into a call to discuss further, feel free to reach out to me via: [email protected].
for the "spark" variable issue, there is the quick workaround that might help to address the warning.
The pre-run code I mentiond actually stay in the local desktop, too. It is sub-folder under your Home directory as: .ipython/profile_default/startup/init_lighter.py.
so in your own .py file, you can import that "init_lighter.py" module.
import sys import os home_directory = os.path.expanduser("~") directory_to_append = os.path.join(home_directory, ".ipython/profile_default/startup/") sys.path.append(directory_to_append) from init_lighter import spark print("Spark version: " + spark.version)now within you own .py file, you can call the method from the "Spark" variable without any warning from Python interperter.
for the second issue, one thing need to call out in current release, ONLY the spark code will be posted/synced to the remote workspace for the execution, the pure python code would be still execute locally in your desktop, that explain why the you will need a local folder path macth to the lakehosue log.
qixiao
Hi QixiaoWang !
No worries about the late reply. I’m glad I can contribute for the VS code experience in Fabric and, thanks for the time you put into trying to figure out these issues.
Regarding Spark session invocation:
As a quick test, I tried to invoke the spark session inside my notebook, before even going into my modules, but apparently the init_lighter.py module is not being recognized, as you can see in image below:
although it was initialized when I opened the notebook in my local machine, as you can see in the output console in above image, and also it does exist in my local machine as below image demonstrates:
Maybe, at this point, it’s just me being bad with Python, but you happen to have any clue on how to solve this one?
Regarding the logging file:
I understand what you mean, but it is still a bit odd to me that the Tables are recognized but the Files are not. I am just wondering, right now, what would be the best/advised practice to have in this case. Perhaps instead of a logging file, I should have a logging table, so I don’t have to replicate too many artifacts in my local machine, in order to avoid mistakes or maintenance complexity/overhead.
Again, thanks for the input and for your time. 🙂