Forum Discussion

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
3 years ago
Solved

Referencing notebook exit value as a variable in a data pipeline

Hi everyone,   In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeli...
Need Help
pipelines
  • Nero's avatar
    Nero
    2 years ago

    For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

     

    Notebook pyspark:

     

    lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

     

    Expression for varMessageStr:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).message

     

    Expression for varNumUpdatedInt:

     

    INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

     

    Expression for varFruitArray:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).fruit

     

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
3 years ago

Hi Ostrzak 

 

We can pass multiple parameters from Notebook to a pipeline using list of parameters. I have attached the screenshots along with the steps for your reference.

1) Create a list of parameters inside a notebook.

2) Using notebook and set variable activites we can pass the parameters to a pipeline.

 

I hope this information is helpful. Please do let us know if you have any questions.

 

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
3 years ago

Hi @v-cboorla-msft 

 

Thank you for your answer. I got to that point independently but I can't parse the exit value so that I get several independent variables out of it. It would be great to know how to tackle that problem.

 

Additionally, is there a more advanced documentation for expression builder? I found this page and it's pretty fine, but it doesn't go beyond the functions listed in the builder. Even the expression I used in Set Variable (see my second post) was found through trial and error, after an extensive web search; it was not efficient.

 

Thank you in advance for any help.