Concurrent notebooks in a pipeline

[Ostrzak]

Hi everyone,

I try to use two notebooks in a concurrent session to be able to pass global temp tables in between them (without writing and reading every single intermediary value I have in my workflow). When I do it manually on notebooks, I am able to use global_temp tables, but when I run everything in a pipeline, they seem to no be concurred and I get an error.

Is there a way to dynamically use notebook values (as some sort of a cache) in a pipeline without explicitly saving them in a lakehouse? I feel like there will be a lot of mess with so many intermediary files. I will be grateful for every answer, even remotely associated with the issue.

[v-cboorla-msft]

Hi @Ostrzak 

Apologies for the delay in response.

We haven’t heard from you on the last response and was just checking back to see whether your query was answered. Otherwise, will respond back with the more details and we will try to help.

[v-cboorla-msft]

Hi @Ostrzak 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.

[ajarora]

Pipelines allow activities to pass control data to the subsequent activities. In your notebook you can set custom return values, and have them available to the next notebook activity, and passed down to the actual notebook through notebook activity params

[Ostrzak]

Hi @ajarora 

Could you elaborate? The only solution I came around is to use exit values for the notebook, but it is not flexible (here is my other post explaining the issues I have: Re: Referencing notebook exit value as a variable ... - Microsoft Fabric Community)

Thank you in advance for any help.

[v-cboorla-msft]

Hi @Ostrzak 

Thanks for using Microsoft Fabric Community.

Apologies for the issue you have been facing. 

Could you please share me the error details which you are getting? And also can you share me screenshot of the pipeline so that I can help you in better way.

Thanks