Fabric Data Flow Gen2 Destination as Lakehouse Files

[Sathish133]

Hi,

I have tried to load the data into Microsoft Fabric Lakehouse as files using Data Flow Gen2, but there is no option to select as files. It default to load as tables into Lakehouse using Data Flow Gen2.

I can able to see the options in Copy Activity to select either File or Table.

Since many source connectors (Example : Salesforce) are not available in Copy Activity, so we have to dependent on Data Flow Gen2.

Any idea to load as files into Lakehouse using Data Flow Gen2 or it is limitation of Data Flow Gen2.

COPY Activity :

DataFlow Gen2 :

Regards,

Sathish Kumar

[Anonymous]

Hi @Sathish133 ,

Thanks for using Fabric Community.
As I understand you are tying to load files from lakehouse. Can you please check below screenshot -

Please expand Files option during connection.

Hope this is helpful. Do let me know incase of further queries.

[Sathish133]

Hi,

I am not loading from Lakehouse, My case is loading into Lakehouse.

Your screenshot describes Source as Lakehouse which is showing tables and Files.

But I want to use Lakehouse as Destination (Target) and store as files.

Regards,

Sathish Kumar

[Anonymous]

Hi @Sathish133 ,

Apologize for the confusion. You're right, Dataflow Gen2 currently has limitations when it comes to directly loading data as files into the Fabric Lakehouse. Dataflow Gen2 focuses on transforming and loading data into tables within the Lakehouse.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this is helpful. Please let me know incase of further queries.

[Sathish133]

Thanks for the confirmation, as it is Data Flow Gen2 limitation.

This case how we can extract the data from Salesforce and ingest as files into Lakehouse using Fabric.

Regards,

Sathish Kumar

[miguel · Accepted solution]

You could use Notebooks as well in Fabric. Currently Dataflow Gen2 doesn't support the load operation as files to a destination, but you can request that as new feature idea from aka.ms/FabricIdeas 

You could alternatively load the data first as a table and then use something like Data Pipeline to copy the data and then save it as a file using the "Copy" activity. You could also take advantage of the "Copy" activity in pipelines and the REST API for it, or raise a new idea to have new explicit connectors to a source of your choice that might not be available today in Data Pipeline.

The most explicit way would be to use a Notebook. 

Hope this helps!

[Anonymous]

Hi @Sathish133 ,

If you are looking for options to load them as files in lakehouse. At present the alternative way is to use Azure Data Factory and can load data from Salesforce to Fabric Lakehouse.

Docs to refer -
Copy data from and to Salesforce - Azure Data Factory & Azure Synapse | Microsoft Learn
Copy and transform data in Microsoft Fabric Lakehouse - Azure Data Factory & Azure Synapse | Microso...

Hope this is helpful. Do let me know incase of further queries.