Forum Discussion
Helper IIntermittent Failures during Fabric Warehouse Table Updates with dbt on Azure Container Instances
Helper IThank you for your response. Unfortunately, the issue is not yet resolved. I did not quite understand how your reply could lead to a resolution. It seems there might be some misunderstanding about our current situation.
As I mentioned in my initial post, we are using the Fabric Warehouse database. It seems unlikely that the server-side firewall is blocking communication. Also, as stated before, it's not that we can't update at all - only about 0 to 4 out of 50 attempts randomly fail. This means that over 90% are processed correctly. If the server is not configured to accept remote connections, we would expect a 0% success rate.
Given this random behavior, it's challenging to investigate further details, which is why I posted on this forum hoping to gather information from others facing similar issues.
If anyone has experience with these types of issues in Fabric Warehouse and can share the conditions under which they occur, your information would be greatly appreciated.
Hi ShunOshidari ,
Apologies that you are still facing this issue.
Please make sure that all the pre-requisites mentioned in this section has been taken care and also make sure to follow the step by step guide to set up dbt in microsoft fabric as mentioned in this tutorial: Link1
If 'dbt test' step fails, try to jump to next step 'dbt run' before test as discussed in this video: Link2
Additionally, if issue still persists, then try the following workaround:
We have several options to mitigate this issue:
- Enable connection pooling in Linux has to be enabled both by the application and in the driver manager, and is off by default might and also we are going to improve the connection at client side.
- Change to redirect, Link3 improving the connection at server side and remove idle connection threshold.
- Implement the retry logic as follow in Python.
Hope this helps. Please let us know if you have any further queries.
- ShunOshidari
Thank you for your advice.
I looked at the "Link3" page about "Change to redirect", but does this also apply to Fabric Warehouse? It looked like a server-side setting, but if it's something that can be set on the user side in Fabric Warehouse, could you please tell me if there are any documents or procedures written about it?Hi ShunOshidari ,
Please refer these documents for your help.- Microsoft Fabric Synapse Data Warehouse setup | dbt Developer Hub (getdbt.com)
- Microsoft Fabric DWH configurations | dbt Developer Hub (getdbt.com)
- Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further queries. Glad to help.