Error in Mirrored google Big query

[Bharath_Kumar_S]

Hi, 

Q1. I am trying to mirror tables in GCP Big Query, once I set up everything its giving below error. 
I couldn't find any specific error message.

Q2. Our data has 15 tables few crore rows in each. Is it better to use Mirrored google Big query or a custom pipeline which will do CDC per day basis to optimize cost from both GCP and Fabric perspective?
Any help will be appreciated.

[ajarora]

We are announcing support for CDC from GBQ in CopyJob as well (Ignite). If you need finer control on schedule, or polling, you can consider CopyJob along with Mirror

[v-sshirivolu]

Hi @Bharath_Kumar_S ,
Thanks for reaching out to Microsoft Fabric Community forum.

I understand you're encountering errors with Mirrored Google BigQuery tables in Fabric and are evaluating whether mirroring or a custom CDC pipeline is more suitable for your scenario involving approximately 15 tables with crores of rows. Here’s an overview: Mirroring Errors - The Internal system error occurred (ArtifactId …) may result from: Transient issues : Reseed by navigating to Configure replication, deselecting and saving the affected tables, then re-enabling them. Gateway version/connectivity : Confirm your On-Premises Data Gateway is at least version 3000.286.6. Permissions : Ensure the service account has roles/bigquery.admin and roles/storage.admin. Unsupported objects : Only tables are supported; replace any views with materialized tables. Lag : Mirroring introduces a delay (10–15 minutes for CDC); polling slows if there is no new data. If these steps do not resolve the issue, please collect the ArtifactId, table name, timestamp, and mirrored item URL and submit a Fabric support ticket.

Mirroring vs. Custom Pipeline For large tables: Mirroring: Offers easy setup and automated CDC, suitable for near-real-time requirements, but is less flexible and currently in preview, which may lead to errors and higher costs with frequent changes. Custom CDC pipeline: Requires more initial configuration, enables scheduled loads and selective data management, and may be more cost-effective if real-time sync is not required.

Recommendation: For near real-time analytics, proceed with Mirroring after resolving any gateway or permission issues. For daily or less frequent refreshes and optimized cost management, a custom CDC pipeline is advisable.

[v-sshirivolu]

Hi @Bharath_Kumar_S ,

I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.

[v-sshirivolu]

Hi @Bharath_Kumar_S ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

[v-sshirivolu]

Hi @Bharath_Kumar_S ,
i wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

[tayloramy · Accepted solution]

Hi @Bharath_Kumar_S, 

Here’s what that message usually means and how to get unblocked, plus some cost/design guidance for BigQuery vs a custom CDC pipeline.

Troubleshooting steps:

1. Treat “Running with warnings -> Internal system error occurred (ArtifactId …)” as transient first. Reseed the affected table(s): in the Mirroring item, open Configure replication, uncheck the problem table(s), Save, then re-check and Mirror database. You can also stop and start mirroring if multiple tables show it. See status meanings in Monitor replication.
2. Verify gateway + permissions. BigQuery mirroring requires OPDG 3000.286.6+ and a service account with the documented BigQuery and GCS permissions (BigQueryAdmin + StorageAdmin covers them). Security requirements. 
3. Only tables are supported. Replace views with materialized tables before mirroring. Troubleshooting: views not supported. 
4. Expect some lag. BigQuery CDC has an inherent ~10–15 minute delay, and Fabric increases polling intervals when there are no changes. Tutorial and limitations. 
5. If it persists, open a ticket with the ArtifactId, mirrored item URL, failing table, and timestamp. BigQuery mirroring is in preview. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.