Forum Discussion

Ostrzak's avatar
Ostrzak
Icon for Helper II rankHelper II
2 years ago
Solved

Referencing notebook exit value as a variable in a data pipeline

Hi everyone,   In general: I am trying to build a pipeline around multiple PySpark notebooks, with some additional pipeline features. To achieve it I need to pass values from a notebook to a pipeli...
Need Help
pipelines
  • Nero's avatar
    Nero
    2 years ago

    For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

     

    Notebook pyspark:

     

    lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

     

    Expression for varMessageStr:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).message

     

    Expression for varNumUpdatedInt:

     

    INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

     

    Expression for varFruitArray:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).fruit

     

Nero's avatar
Nero
Icon for Advocate II rankAdvocate II
2 years ago

I also struggled, until I tried ignoring the expression builder syntax errors and just ran my pipeline. I was able to get the following to work by using a delimited string as the notebook return value.

Notebook pyspark:

 

lResult="returnValueOne=25|returnValueTwo=abc|returnValueThree=19"
mssparkutils.notebook.exit(str(lResult))

 

Expression for varAllReturnValuesArray

 

@split(activity('Test Notebook').output.result.exitValue,'|')

 

Expression for varReturnValueOneString

 

@last(split(first(split(activity('Test Notebook').output.result.exitValue,'|')),'='))

 

Expression for varReturnValueTwoString

 

@last(split(split(activity('Test Notebook').output.result.exitValue,'|')[1],'='))

 


 

  • Nero's avatar
    Nero
    Icon for Advocate II rankAdvocate II
    2 years ago

    For my own implementation the delimited string just wasnt flexible enough - so here's a version using a JSON object as the notebook return value: 

     

    Notebook pyspark:

     

    lResult={"numUpdated":10,"message":"success","fruit":["apple","orange","grapes"]}
mssparkutils.notebook.exit(str(lResult))

     

    Expression for varMessageStr:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).message

     

    Expression for varNumUpdatedInt:

     

    INT(json(activity('Test Notebook').output.result.exitValue).numUpdated)

     

    Expression for varFruitArray:

     

    Anonymous(activity('Test Notebook').output.result.exitValue).fruit

     

    • AdamFry's avatar
      AdamFry
      Icon for Advocate I rankAdvocate I
      2 years ago

      Just want to say that this post was EXTREMELY helpful to me and I want to say thank you for sharing this really nice approach.  

    • sam_wise's avatar
      sam_wise
      Frequent Visitor
      2 years ago

      Bravo! This was extremely helpful!

       

      Many thanks!

    • tlanza89_RWL's avatar
      tlanza89_RWL
      Icon for Advocate I rankAdvocate I
      1 year ago

      This is exactly what I needed. Thank you!