Dataflow Gen2 - Fails to write data

[webchris]

Hi

I need to open this again and I hope to get answers...

In my Dataflow Gen2 I want copy 8 tables from an external source into a Fabric Warehouse. Flow outputs all columns as text, target tables contain only text columns. The yellow marked tables have approx 80'000 records. The green ones < 1'000 records.

After 8 hours, the Dataflow fails and suggests to reduce complexity.

In case 80'000 records are too complex for a Dataflow Gen2, is there an alternative option?

Thanks

Christian

[webchris]

Hi, thanks for your Feedback @miguel - I created a support ticket but am waiting for 2 days now and did not get a reply...

@v-cboorla-msft 
I ran the huge flow and reduce the number of records to 11 mio - that worked. Therefore I doubt that this issue is related to firewalls or proxy!?

[miguel]

Please share your case number with us here and ping us if you don't hear from a support engineer by the end of the week.

[webchris]

Hi @miguel 

#2310170050002695 is the case number. Thanks!

[v-cboorla-msft]

Hi @webchris 

Thank you for your response and raising support ticket.

[webchris]

Update from my side:

It is possible, to load the data into the Lakehouse. At least for these small to medium files it worked.

I am still running another Flow that is supposed to write 50 Mio records. That - so far - did not succeed loading to lakehouse...

[v-cboorla-msft]

Hi @webchris 

Welcome to Fabric Community and thanks for posting your question here.

As I understand you are trying to load data from external source to Lakehouse using Dataflow Gen2. The below documentation might help you. 

Please refer to this link: Link1 

Hope this helps. Please let me know in case of any queries.

[miguel]

Hi Christian,

Please reach out to our support team so they can do a more thorough investigation on why some of these refreshes fail.

https://support.fabric.microsoft.com/support