Spark notebook output

[krishn]

Hi, we are trying to create pipeline flow like copy data from one lake house to another lake house. Copying daily data from external source (RestAPI) to Raw lake house table. As a subsequent step we wanted to query data (dynamically for specific day) in spark notebook and pass result data frame (SQL Query output) to following copy activity. Is it possible to pass spark notebook data frame out put to any activity in Microsoft Fabric ? We are restricted to use Fabric warehouse so we cannot use SQL, StoredProcedure and Lookup activities to query data. Appreciate for help. Krishn

[v-cboorla-msft]

Hi @krishn 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience.

Directly passing a Spark DataFrame output from a Microsoft Fabric notebook (likely a Spark Notebook activity in Data Factory) to another activity isn't possible due to format incompatibility. This is because Data Factory activities primarily work with structured data formats like Parquet, CSV, Avro, and JSON, while Spark DataFrames are in-memory representations of distributed datasets on Spark clusters.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[krishn]

Thanks for the reply, is there any alternatives we can consider? all we are trying to do is query table for specific data (from Lake house table) and load it into another lake house table. Krishn

[v-cboorla-msft]

Hi @krishn 

Apologies for the delay in response from my end.

Following up to see whether your query got resolved, if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @krishn 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.