Forum Discussion

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
2 years ago
Solved

Referencing notebook exit value as a variable in a data pipeline

Hi everyone,   In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeli...
Need Help
pipelines
  • Nero's avatar
    Nero
    2 years ago

    For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

     

    Notebook pyspark:

     

    lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

     

    Expression for varMessageStr:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).message

     

    Expression for varNumUpdatedInt:

     

    INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

     

    Expression for varFruitArray:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).fruit

     

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
2 years ago

An update: I found out that changing Set variable activity to:

@activity('Notebook_name').output.result.exitValue

... solves the main problem.

 

I still haven't figured out how to pass multiple notebook values to pipeline variables, so if anyone could help me I will be grateful.

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi Ostrzak 

 

We can pass multiple parameters from Notebook to a pipeline using list of parameters. I have attached the screenshots along with the steps for your reference.

1) Create a list of parameters inside a notebook.

2) Using notebook and set variable activites we can pass the parameters to a pipeline.

 

I hope this information is helpful. Please do let us know if you have any questions.