Notebook in Pipeline to trigger next flow, based on condition in the dataframe's column

[tan_thiamhuat]

I want to implement something like this:

In a Notebook, there is a dataframe DF with its column name 'Success'. If the sum(DF$success)>0, then I want to trigger the next flow.

Can someone show how to implement this?

[burakkaragoz · Accepted solution]

Hi @tan_thiamhuat ,

Yep, you can do this in your notebook by checking the sum of the 'Success' column and then using mssparkutils.notebook.exit to pass a value back to the pipeline. Here’s an example in PySpark:

Python

success_sum = DF.selectExpr("sum(Success)").collect()[0][0]
if success_sum > 0:
    mssparkutils.notebook.exit("trigger_next")
else:
    mssparkutils.notebook.exit("do_not_trigger")

In your pipeline flow, you can use the output value from the notebook activity to decide whether to trigger the next activity/flow or not. Just set up a conditional check in the pipeline based on the notebook’s output string.

Let me know if you need more details or some other way to handle it.

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.

[tan_thiamhuat]

I have this two conditions:

## when there is CSV file to be processed

mssparkutils.notebook.exit("Cleaned CSV File generated at /primary/CleanedData in SFTP Server.")

## when there is NO CSV file to be processed

mssparkutils.notebook.exit("No CSV files to be processed.")

And I put the condition in the if loop as below:

It always go to the False loop, even if there is CSV file available to be processed. Is my logic correct?