topic Re: Dataflow to bring data from Google Bigquery takes forever and fails often in Dataflow

Dataflow to bring data from Google Bigquery takes forever and fails often

sathishsam — Mon, 11 Dec 2023 10:42:31 GMT

We have our data currently in Google BigQuery and using import model to build our reports and now want to explore Fabric to see if the reports would become more faster.

I tried to get data from BigQuery to Fabric Lakehouse using Dataflow Gen2, however, to get 30 million rows it is taking 4 to 5 hours and many a times it fails. Not sure if I am doing something incorrect.

- Using Google Bigquery connector

- No transformation at all

This is very important for us to further explore Fabric, please help

Re: Dataflow to bring data from Google Bigquery takes forever and fails often

Anonymous — Tue, 12 Dec 2023 07:34:43 GMT

Hi sathishsam ,

Apologies for the issue that you are facing.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Re: Dataflow to bring data from Google Bigquery takes forever and fails often

Anonymous — Wed, 13 Dec 2023 11:13:28 GMT

Hi sathishsam ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes, please provide the ticket number here as we can keep an eye on it.

Re: Dataflow to bring data from Google Bigquery takes forever and fails often

sathishsam — Wed, 13 Dec 2023 11:22:08 GMT

Raised a ticket and working with the engineer. however, it looks like there is nothing much the Dataflow Gen 2 will do here..
SR 2312120010002318