Forum Discussion

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
2 years ago
Solved

Referencing notebook exit value as a variable in a data pipeline

Hi everyone,   In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeli...
Need Help
pipelines
  • Nero's avatar
    Nero
    2 years ago

    For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

     

    Notebook pyspark:

     

    lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

     

    Expression for varMessageStr:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).message

     

    Expression for varNumUpdatedInt:

     

    INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

     

    Expression for varFruitArray:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).fruit

     

Nero's avatar
Nero
Icon for Advocate II rankAdvocate II
2 years ago

I also struggled, until I tried ignoring the expression builder syntax errors and just ran my pipeline. I was able to get the following to work by using a delimited string as the notebook return value.

Notebook pyspark:

 

lResult="returnValueOne=25|returnValueTwo=abc|returnValueThree=19"
mssparkutils.notebook.exit(str(lResult))

 

Expression for varAllReturnValuesArray

 

@split(activity('Test Notebook').output.result.exitValue,'|')

 

Expression for varReturnValueOneString

 

@last(split(first(split(activity('Test Notebook').output.result.exitValue,'|')),'='))

 

Expression for varReturnValueTwoString

 

@last(split(split(activity('Test Notebook').output.result.exitValue,'|')[1],'='))

 


 

Nero's avatar
Nero
Icon for Advocate II rankAdvocate II
2 years ago

For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

 

Notebook pyspark:

 

lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

 

Expression for varMessageStr:

 

Anonymous(activity('Test Notebook').output.result.exitValue).message

 

Expression for varNumUpdatedInt:

 

INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

 

Expression for varFruitArray:

 

Anonymous(activity('Test Notebook').output.result.exitValue).fruit