topic Re: Get the output of a pipeline with a notebook in Data Engineering

Get the output of a pipeline with a notebook

Anonymous — Tue, 20 Aug 2024 07:06:36 GMT

Hello

i need to do the opposite of this ? How can i get the output of a pipeline data factory in my notebook ?

Thank you for your help

Re: Get the output of a pipeline with a notebook

Anonymous — Tue, 20 Aug 2024 08:36:28 GMT

Hi @Anonymous ，

You can use the pipeline and create the destination to a lakehouse and then use notebook to query the results.

Create your first data pipeline to copy data - Microsoft Fabric | Microsoft Learn

How to use notebooks - Microsoft Fabric | Microsoft Learn

Best Regards,
Gao
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly -- How to provide sample data in the Power BI Forum

Re: Get the output of a pipeline with a notebook

Hofpower — Tue, 20 Aug 2024 08:39:37 GMT

Hi @Anonymous

I am not 100% sure what you actually want to do but I assume you want to set or calculate some sort of values within a data factory pipeline and then hand these values over to an executed notebook.

If this is the fact I think the easiest way is to use parameter cells within the notebook, see the MSFT documentation https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#designate-a-parameters-cell and e.g. this blog post https://community.fabric.microsoft.com/t5/Data-Pipelines/notebook-parameter/m-p/3907708#:~:text=Step1%3A%20Create%20a%20notebook%20with,parameters%20under%20%E2%80%9Cbase%20parameters%E2%80%9D.

Once these parameter cells are set in your notebook you can execute the notebook within the pipeline and programmatically describe the parameters that should be used for this specific run of the notebook within the pipeline.

Hope this answer solves your problem. 🙂
If not feel free to describe your problem a bit more detailed.

Martin

Re: Get the output of a pipeline with a notebook

Anonymous — Tue, 20 Aug 2024 14:49:53 GMT

Hello

i succeed to get the output of my activity. Now i need to write this output on a table in my lakehouse.

Any ideas ?

Thank you

Re: Get the output of a pipeline with a notebook

Hofpower — Wed, 21 Aug 2024 07:33:36 GMT

Hi @Anonymous

I assume you want to persist calculated tables from the notebook to your lakehouse?

In that case the easiest way would be to write these tables directly within your notebook using somehing like

df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name)

BR
Martin

Re: Get the output of a pipeline with a notebook

Anonymous — Wed, 21 Aug 2024 08:43:55 GMT

Hi @Hofpower

Thank you for your answer, it is okay for me to write in a table of my lakehouse. But how can you get the value of the "Set Variable" activity in my notebook ? Perhaps using mssparkutils ?

Thank for you help

Re: Get the output of a pipeline with a notebook

Hofpower — Wed, 21 Aug 2024 12:14:35 GMT

Hi @Anonymous

As mentioned in my first reply below I advise you to use parameters. You can create a "parameter cell" (see https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#designate-a-parameters-cell ) in the notebook with some kind of default parameters so the notebook logically would also work without executing it within a pipeline.

Then within the pipeline you need to set the parameter you defined in the parameter cell of the notebook within the notebook call in the pipeline (same article, a bit below https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#assign-parameters-values-from-a-pipeline). Set the value of the parameter to the value of the variable and then the value of the variable will be used when executing the notebook from the pipeline.

Did you try to use this and if yes why did it not work? Could you provide your code (anonymized if necessary) or a similar examplary code?

Martin