Dataflow refresh error for Google Big Query view

[Mukesh_Patel]

Receiving the following error while refreshing a dataflow which fetches data from a Google Cloud view.
"Error: Data Source Error : DataSource.Error: ERROR [HY000] [Microsoft][BigQuery] (310) Storage API Error: Error: { code: INTERNAL message: Received RST_STREAM with error code 2 }."
I am using the Google Big Query connector and native query to fetch data. The view has around 70 million records and I have setup incremental refresh for last 15 days.
The dataflow was refreshing for a week prior to receiving this error.

Please share solution to resolve this issue. Thanks.

[Anonymous]

Hi @Mukesh_Patel 

This error usually indicates that an internal error was encountered during the dataflow refresh process when using the Google BigQuery connector. Here are some of the possible solutions:

1. An unstable or interrupted internet connection may cause this error, make sure your internet connection is stable.

2. Increase BigQuery quota limits.

3. Adjust the incremental refresh policy from last 15 days to a shorter time period to see if this error can be avoided.

Please refer to the following troubleshooting documentation to see if it helps:

cloud.google.com/dataflow/docs/guides/common-errors 

cloud.google.com/bigquery/docs/troubleshoot-queries 

cloud.google.com/dataflow/docs/guides/troubleshoot-slow-jobs 

In the meantime, it is recommended that you get support from the BigQuery community to see if they have any good suggestions: cloud.google.com/bigquery/docs/getting-support 

Best Regards,
Jarvis Tang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.