Get the output of a pipeline with a notebook

[BendeLyon_69]

Hello 

i need to do the opposite of this ? How can i get the output of a pipeline data factory in my notebook ?

Thank you for your help

[BendeLyon_69]

Hello

i succeed to get the output of my activity. Now i need to write this output on a table in my lakehouse.

Any ideas ? 

Thank you

[Hofpower]

Hi @BendeLyon_69 

I assume you want to persist calculated tables from the notebook to your lakehouse? 

In that case the easiest way would be to write these tables directly within your notebook using somehing like 

df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name)

See also the documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-notebook-load-data#load-data-wit... 

BR
Martin

[BendeLyon_69]

Hi @Hofpower 

Thank you for your answer, it is okay for me to write in a table of my lakehouse. But how can you get the value of the "Set Variable" activity in my notebook ? Perhaps using mssparkutils ? 

Thank for you help

[Hofpower]

Hi @BendeLyon_69 

As mentioned in my first reply below I advise you to use parameters. You can create a "parameter cell" (see https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#designate-a-parame... ) in the notebook with some kind of default parameters so the notebook logically would also work without executing it within a pipeline.

Then within the pipeline you need to set the parameter you defined in the parameter cell of the notebook within the notebook call in the pipeline (same article, a bit below https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#assign-parameters-.... Set the value of the parameter to the value of the variable and then the value of the variable will be used when executing the notebook from the pipeline. 

Did you try to use this and if yes why did it not work? Could you provide your code (anonymized if necessary) or a similar examplary code? 

BR

Martin

[Hofpower]

Hi @BendeLyon_69 

I am not 100% sure what you actually want to do but I assume you want to set or calculate some sort of values within a data factory pipeline and then hand these values over to an executed notebook. 

If this is the fact I think the easiest way is to use  parameter cells within the notebook, see the MSFT documentation https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#designate-a-parame... and e.g. this blog post https://community.fabric.microsoft.com/t5/Data-Pipelines/notebook-parameter/m-p/3907708#:~:text=Step.... 

Once these parameter cells are set in your notebook you can execute the notebook within the pipeline and programmatically describe the parameters that should be used for this specific run of the notebook within the pipeline.

Hope this answer solves your problem. 🙂
If not feel free to describe your problem a bit more detailed.

BR

Martin 

[Anonymous]

Hi @BendeLyon_69 ，

You can use the  pipeline and create the destination to a lakehouse and then use notebook to query the results.

Create your first data pipeline to copy data - Microsoft Fabric | Microsoft Learn

How to use notebooks - Microsoft Fabric | Microsoft Learn

Best Regards,
Gao
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly --  How to provide sample data in the Power BI Forum