Referencing notebook exit value as a variable in a data pipeline

Hi everyone, In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeli...

Need Help

pipelines

Nero

2 years ago

For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value:

Notebook pyspark:

lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

Expression for varMessageStr:

Anonymous(activity('Test Notebook').output.result.exitValue).message

Expression for varNumUpdatedInt:

INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

Expression for varFruitArray:

Anonymous(activity('Test Notebook').output.result.exitValue).fruit

v-cboorla-msft
Microsoft Employee
2 years ago
Hi Ostrzak

We can pass multiple parameters from Notebook to a pipeline using list of parameters. I have attached the screenshots along with the steps for your reference.

1) Create a list of parameters inside a notebook.

2) Using notebook and set variable activites we can pass the parameters to a pipeline.

I hope this information is helpful. Please do let us know if you have any questions.
- Ostrzak
  Helper II
  2 years ago
  Hi @v-cboorla-msft
  
  Thank you for your answer. I got to that point independently but I can't parse the exit value so that I get several independent variables out of it. It would be great to know how to tackle that problem.
  
  Additionally, is there a more advanced documentation for expression builder? I found this page and it's pretty fine, but it doesn't go beyond the functions listed in the builder. Even the expression I used in Set Variable (see my second post) was found through trial and error, after an extensive web search; it was not efficient.
  
  Thank you in advance for any help.
  - jwinchell40
    Resolver III
    1 year ago
    Ostrzak - This request is really similiar to one I replied to a few months back. (https://community.fabric.microsoft.com/t5/Data-Pipeline/Use-Notebook-Variable-in-Email-Activity/m-p/3973679#M3786)
    
    If you send back a JSON object you can convert the string in the pipeline use @Json() and then reference values within the JSON object.
    
    @Json(activity(<activity_name>).output.result.exitValue).<property>
- prabhatnath
  Advocate III
  1 year ago
  Thank you for this infomation v-cboorla-msft .
  Can you also help specifying what is the type specified for the variable and what is the expression used in the set variable please?
  
  Thanks,
  Prabhat
v-cboorla-msft
Microsoft Employee
2 years ago
Hi Ostrzak

Thanks for using Fabric Community.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.