topic Re: Can't run pipeline with notebook which tries to access anything from built-in resources in Data Engineering

Can't run pipeline with notebook which tries to access anything from built-in resources

shubhamg — Wed, 22 May 2024 13:29:18 GMT

I have a python script stored in built-in resources:

I'm running this script from notebook using %run as below and it runs successfully:

But when I run the same notebook through pipeline I get error:
Fetch notebook content for NB_BRZ_SHPO_List_Ingest failed with exception: Request to https://tokenservice1.australiasoutheast.trident.azuresynapse.net/api/v1/proxy/preprocessorApi/versions/2019-01-01/productTypes/trident/capacities/xxxx failed with status code: 400, response:Failed to get file "workspace.py" from notebook file system resource manager with error: "file workspace.py was not found in storage.".Notebook path: workspace.py, response headers: Array(Content-Type: text/plain; charset=utf-8, Date: Wed, 22 May 2024 12:13:49 GMT, Server: Kestrel, Transfer-Encoding: chunked, Request-Context: appId=, x-ms-gateway-request-id: xx-cdf0-4527-a264-xx| client-request-id : xx-d48b-4363-9682-xx, x-ms-workspace-name: xx-195f-42dc-87aa-xx, x-ms-activity-id: xx-cdf0-4527-a264-xx, x-ms-client-request-id: xx-d48b-4363-9682-xx).

I cannot access any other resource form built-in example json either:
[Errno 2] No such file or directory: '/synfs/resource/xx-86ca-499f-ad6c-xx/nb_resource/builtin/workspace.json' ---------------------------------------------------------------------------FileNotFoundError Traceback (most recent call last)Cell In[7], line 7 5 ws_details = workspace_id_map[worspace_id] 6 else: ----> 7 with open(f'{mssparkutils.nbResPath}/builtin/workspace.json', 'r') as file: 8 ws_details = json.load(file) 10 print(ws_details) File ~/cluster-env/trident_env/lib/python3.10/site-packages/IPython/core/interactiveshell.py:284, in _modified_open(file, *args, **kwargs) 277 if file in {0, 1, 2}: 278 raise ValueError( 279 f"IPython won't let you open fd={file} by default " 280 "as it is likely to crash IPython. If you know what you are doing, " 281 "you can use builtins' open." 282 ) --> 284 return io_open(file, *args, **kwargs) FileNotFoundError: [Errno 2] No such file or directory: '/synfs/resource/xx-499f-ad6c-xx/nb_resource/builtin/workspace.json'

Re: Can't run pipeline with notebook which tries to access anything from built-in resources

Anonymous — Thu, 23 May 2024 09:42:19 GMT

Hi @shubhamg ,

Thanks for using Fabric Community.
When I tried to repro your scenario, I am able execute the magic command from both the places (from - Notebook & Pipeline).

Magic Command:

%run -b workspace.py

From Notebooks:

From Pipeline:

Click on Spark Monitoring Url

Can you please check your notebook code and try again, if you are still facing an issue to execute in pipeline please share the necessary screenshot to understand the issue in detail?

Hope this is helpful. Do let me know incase of further queries.

Re: Can't run pipeline with notebook which tries to access anything from built-in resources

Anonymous — Fri, 24 May 2024 10:56:37 GMT

Hi @shubhamg ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Can't run pipeline with notebook which tries to access anything from built-in resources

Anonymous — Mon, 27 May 2024 09:25:52 GMT