topic Re: Referencing notebook exit value as a variable in a data pipeline in Pipelines

Referencing notebook exit value as a variable in a data pipeline

Ostrzak — Tue, 31 Oct 2023 11:43:24 GMT

Hi everyone,

In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeline.

I checked several pages concerning the topic and consensus was to use Exit Value in the notebook:

mssparkutils.notebook.exit(variable)

And then create a pipeline variable, and then in Set variable pipeline activity use a dynamic expression:

@activity('Notebook_name').output.status.Output.result.exitValue

For some reason I get such error in Set Variable activity:

The expression 'activity('Notebook1').output.status.Output.result.exitValue' cannot be evaluated because property 'Output' cannot be selected. Property selection is not supported on values of type 'String'.

I can't find anything on it's origin and what I should change. I tried formatting exit value as an number or string and it doesn't help.

I would appreciate the help. If there is another way to "pass" values from the notebook to the pipeline, please let me know; I need it to work further.

Re: Referencing notebook exit value as a variable in a data pipeline

Ostrzak — Tue, 31 Oct 2023 22:24:13 GMT

An update: I found out that changing Set variable activity to:

@activity('Notebook_name').output.result.exitValue

... solves the main problem.

I still haven't figured out how to pass multiple notebook values to pipeline variables, so if anyone could help me I will be grateful.

Re: Referencing notebook exit value as a variable in a data pipeline

v-cboorla-msft — Wed, 01 Nov 2023 10:36:08 GMT

Hi Ostrzak

Thanks for using Fabric Community.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Re: Referencing notebook exit value as a variable in a data pipeline

v-cboorla-msft — Wed, 01 Nov 2023 13:27:37 GMT

Hi Ostrzak

We can pass multiple parameters from Notebook to a pipeline using list of parameters. I have attached the screenshots along with the steps for your reference.

1) Create a list of parameters inside a notebook.

2) Using notebook and set variable activites we can pass the parameters to a pipeline.

I hope this information is helpful. Please do let us know if you have any questions.

Re: Referencing notebook exit value as a variable in a data pipeline

Ostrzak — Thu, 02 Nov 2023 10:28:59 GMT

Hi @v-cboorla-msft

Thank you for your answer. I got to that point independently but I can't parse the exit value so that I get several independent variables out of it. It would be great to know how to tackle that problem.

Additionally, is there a more advanced documentation for expression builder? I found this page and it's pretty fine, but it doesn't go beyond the functions listed in the builder. Even the expression I used in Set Variable (see my second post) was found through trial and error, after an extensive web search; it was not efficient.

Thank you in advance for any help.

Re: Referencing notebook exit value as a variable in a data pipeline

v-cboorla-msft — Fri, 10 Nov 2023 09:55:57 GMT

Hi Ostrzak

Apologies for the delay in reply from our side. Did you got a resolution for this?
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Referencing notebook exit value as a variable in a data pipeline

v-cboorla-msft — Wed, 15 Nov 2023 06:41:08 GMT

Hi Ostrzak

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.

Re: Referencing notebook exit value as a variable in a data pipeline

Ostrzak — Fri, 24 Nov 2023 17:17:59 GMT

Hi v-cboorla-msft

I haven't found a solution to this problem yet.

Re: Referencing notebook exit value as a variable in a data pipeline

Nero — Mon, 19 Feb 2024 06:51:09 GMT

I also struggled, until I tried ignoring the expression builder syntax errors and just ran my pipeline. I was able to get the following to work by using a delimited string as the notebook return value.

Notebook pyspark:

lResult="returnValueOne=25|returnValueTwo=abc|returnValueThree=19" mssparkutils.notebook.exit(str(lResult))

Expression for varAllReturnValuesArray

@split(activity('Test Notebook').output.result.exitValue,'|')

Expression for varReturnValueOneString

@last(split(first(split(activity('Test Notebook').output.result.exitValue,'|')),'='))

Expression for varReturnValueTwoString

@last(split(split(activity('Test Notebook').output.result.exitValue,'|')[1],'='))

Re: Referencing notebook exit value as a variable in a data pipeline

Nero — Wed, 21 Feb 2024 04:42:55 GMT

For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value:

Notebook pyspark:

lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]} mssparkutils.notebook.exit(str(lResult))

Expression for varMessageStr:

Anonymous(activity('Test Notebook').output.result.exitValue).message

Expression for varNumUpdatedInt:

INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

Expression for varFruitArray:

Anonymous(activity('Test Notebook').output.result.exitValue).fruit

Re: Referencing notebook exit value as a variable in a data pipeline

AdamFry — Fri, 12 Apr 2024 15:06:11 GMT

Just want to say that this post was EXTREMELY helpful to me and I want to say thank you for sharing this really nice approach.

Re: Referencing notebook exit value as a variable in a data pipeline

sam_wise — Wed, 31 Jul 2024 21:02:11 GMT

Bravo! This was extremely helpful!

Many thanks!

Re: Referencing notebook exit value as a variable in a data pipeline

prabhatnath — Sat, 21 Sep 2024 05:14:42 GMT

Thank you for this infomation v-cboorla-msft .
Can you also help specifying what is the type specified for the variable and what is the expression used in the set variable please?

Thanks,

Prabhat

Re: Referencing notebook exit value as a variable in a data pipeline

tlanza89_RWL — Tue, 15 Oct 2024 19:17:51 GMT

This is exactly what I needed. Thank you!

Re: Referencing notebook exit value as a variable in a data pipeline

jwinchell40 — Wed, 16 Oct 2024 03:32:02 GMT

Ostrzak - This request is really similiar to one I replied to a few months back. (https://community.fabric.microsoft.com/t5/Data-Pipeline/Use-Notebook-Variable-in-Email-Activity/m-p/3973679#M3786)

If you send back a JSON object you can convert the string in the pipeline use @Json() and then reference values within the JSON object.

@Json(activity(<activity_name>).output.result.exitValue).<property>

Re: Referencing notebook exit value as a variable in a data pipeline

blossom_dataeng — Tue, 04 Feb 2025 17:12:19 GMT

Thank you so much for this solution - curious question for you: I was able to get it working only with

@json(activity('Test Notebook).output.status.Output.result.exitValue).param1

rather than

@json(activity('Test Notebook').output.result.exitValue).year

Do you know why this could be?

Re: Referencing notebook exit value as a variable in a data pipeline

blossom_dataeng — Tue, 04 Feb 2025 17:12:55 GMT

Apologies, the bottom should read: @json(activity('Test Notebook').output.result.exitValue).param1