topic Dataflow Gen2 - Fails to write data in Dataflow

Dataflow Gen2 - Fails to write data

webchris — Wed, 18 Oct 2023 07:15:18 GMT

I need to open this again and I hope to get answers...

In my Dataflow Gen2 I want copy 8 tables from an external source into a Fabric Warehouse. Flow outputs all columns as text, target tables contain only text columns. The yellow marked tables have approx 80'000 records. The green ones < 1'000 records.

After 8 hours, the Dataflow fails and suggests to reduce complexity.

In case 80'000 records are too complex for a Dataflow Gen2, is there an alternative option?

Thanks

Christian

Re: Dataflow Gen2 - Fails to write data

webchris — Wed, 18 Oct 2023 11:17:41 GMT

Update from my side:

It is possible, to load the data into the Lakehouse. At least for these small to medium files it worked.

I am still running another Flow that is supposed to write 50 Mio records. That - so far - did not succeed loading to lakehouse...

Re: Dataflow Gen2 - Fails to write data

miguel — Wed, 18 Oct 2023 15:45:52 GMT

Hi Christian,

Please reach out to our support team so they can do a more thorough investigation on why some of these refreshes fail.

https://support.fabric.microsoft.com/support

Re: Dataflow Gen2 - Fails to write data

v-cboorla-msft — Wed, 18 Oct 2023 17:16:09 GMT

Hi @webchris

Welcome to Fabric Community and thanks for posting your question here.

As I understand you are trying to load data from external source to Lakehouse using Dataflow Gen2. The below documentation might help you.

Please refer to this link: Link1

Hope this helps. Please let me know in case of any queries.

Re: Dataflow Gen2 - Fails to write data

webchris — Thu, 19 Oct 2023 11:30:32 GMT

Hi, thanks for your Feedback @miguel - I created a support ticket but am waiting for 2 days now and did not get a reply...

@v-cboorla-msft
I ran the huge flow and reduce the number of records to 11 mio - that worked. Therefore I doubt that this issue is related to firewalls or proxy!?

Re: Dataflow Gen2 - Fails to write data

miguel — Thu, 19 Oct 2023 16:06:34 GMT

Please share your case number with us here and ping us if you don't hear from a support engineer by the end of the week.

Re: Dataflow Gen2 - Fails to write data

webchris — Thu, 19 Oct 2023 16:08:56 GMT

Hi @miguel

#2310170050002695 is the case number. Thanks!

Re: Dataflow Gen2 - Fails to write data

v-cboorla-msft — Thu, 19 Oct 2023 17:02:14 GMT

Hi @webchris

Thank you for your response and raising support ticket.