Forum Discussion

krishn's avatar
krishn
Icon for Helper I rankHelper I
2 years ago

Spark notebook output

Hi, we are trying to create pipeline flow like copy data from one lake house to another lake house. Copying daily data from external source (RestAPI) to Raw lake house table. As a subsequent step we ...
Need Help
v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi krishn 

 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience.

Directly passing a Spark DataFrame output from a Microsoft Fabric notebook (likely a Spark Notebook activity in Data Factory) to another activity isn't possible due to format incompatibility. This is because Data Factory activities primarily work with structured data formats like Parquet, CSV, Avro, and JSON, while Spark DataFrames are in-memory representations of distributed datasets on Spark clusters.

 

I hope this information helps. Please do let us know if you have any further questions.

 

Thanks.

  • krishn's avatar
    krishn
    Icon for Helper I rankHelper I
    2 years ago

    Thanks for the reply, is there any alternatives we can consider? all we are trying to do is query table for specific data (from Lake house table) and load it into another lake house table. Krishn

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi krishn 

       

      Apologies for the delay in response from my end.

      Following up to see whether your query got resolved, if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
      Otherwise, will respond back with the more details and we will try to help.


      Thanks.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Icon for Microsoft Employee rankMicrosoft Employee
        2 years ago

        Hi krishn 

         

        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
        If you have any question relating to the current thread, please do let us know and we will try out best to help you.
        In case if you have any other question on a different issue, we request you to open a new thread.


        Thanks.