Can't run pipeline with notebook which tries to access anything from built-in resources

[shubhamg]

I have a python script stored in built-in resources:

I'm running this script from notebook using %run as below and it runs successfully:

But when I run the same notebook through pipeline I get error:
Fetch notebook content for NB_BRZ_SHPO_List_Ingest failed with exception: Request to https://tokenservice1.australiasoutheast.trident.azuresynapse.net/api/v1/proxy/preprocessorApi/versi... failed with status code: 400, response:Failed to get file "workspace.py" from notebook file system resource manager with error: "file workspace.py was not found in storage.".Notebook path: workspace.py, response headers: Array(Content-Type: text/plain; charset=utf-8, Date: Wed, 22 May 2024 12:13:49 GMT, Server: Kestrel, Transfer-Encoding: chunked, Request-Context: appId=, x-ms-gateway-request-id: xx-cdf0-4527-a264-xx| client-request-id : xx-d48b-4363-9682-xx, x-ms-workspace-name: xx-195f-42dc-87aa-xx, x-ms-activity-id: xx-cdf0-4527-a264-xx, x-ms-client-request-id: xx-d48b-4363-9682-xx).

I cannot access any other resource form built-in example json either:
[Errno 2] No such file or directory: '/synfs/resource/xx-86ca-499f-ad6c-xx/nb_resource/builtin/workspace.json' ---------------------------------------------------------------------------FileNotFoundError Traceback (most recent call last)Cell In[7], line 7 5 ws_details = workspace_id_map[worspace_id] 6 else: ----> 7 with open(f'{mssparkutils.nbResPath}/builtin/workspace.json', 'r') as file: 8 ws_details = json.load(file) 10 print(ws_details) File ~/cluster-env/trident_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py:284, in _modified_open(file, *args, **kwargs) 277 if file in {0, 1, 2}: 278 raise ValueError( 279 f"IPython won't let you open fd={file} by default " 280 "as it is likely to crash IPython. If you know what you are doing, " 281 "you can use builtins' open." 282 ) --> 284 return io_open(file, *args, **kwargs) FileNotFoundError: [Errno 2] No such file or directory: '/synfs/resource/xx-499f-ad6c-xx/nb_resource/builtin/workspace.json'

[Anonymous · Accepted solution]

Hi @shubhamg ,

Thanks for using Fabric Community.
When I tried to repro your scenario, I am able execute the magic command from both the places (from - Notebook & Pipeline).

Magic Command:

%run -b workspace.py

From Notebooks:

From Pipeline:

Click on Spark Monitoring Url

Can you please check your notebook code and try again,  if you are still facing an issue to execute in pipeline please share the necessary screenshot to understand the issue in detail?

Hope this is helpful. Do let me know incase of further queries.

[Anonymous]

Hi @shubhamg ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[Anonymous]

Hi @shubhamg ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks