Forum Discussion
Fabric SQL Database SQL endpoint is not refreshing (metadata sync lag)
Hi DKIDV,
Table count exceeds limit is the cause. It is not just a UI message. May be The issue is not caused by permissions or refresh problems the main reason your tables are missing in the SQL Analytics Endpoint is because your database has hit the internal table count limit. Once this limit is reached, the SQL Analytics Endpoint simply stops registering new tables, even though replication still shows as “Running” and the tables exist in the SQL Database. This is why older schemas appear while newer ones don’t, even when the tables are identical. The refreshMetadata API and small table changes won’t fix it because the endpoint is no longer accepting new metadata. The only real solutions are to reduce the number of tables in the database (drop unused or duplicate tables) or split your data across multiple databases using Fabric Pipelines to move data between them. This avoids the limit and ensures all tables become visible and sharable again.
Thanks & Regards,
Prasanna Kumar
Hi Nasif_Azam , Anonymous, MFelix
thanks for the reply.
I now understand the limitations of fabric sql db why some tables are not visible in sql analytics endpoint. e.g. I have some tables with PK's with unsupported data types. Fixing this will not solve 100% my initial question.
Since I want to share different schema's with different tenants e.g. schema A external data share with tenant A, schema B external data share with tenant B. We now have all data in one database, because I think it is not possible to use cross-database with several fabric sql database to operate ETL from database A to database B.
I use pipelines in fabric and not data flow gen 2. If you know a solution for this, I can operate with multiple databases in stead of 1. There are more tenants, because of client data that we want to seperate.
As you can see in the screenshot below: in the 'IM' schema you can see the DimDate table, in sql analytics endpoint you can't see the DimDate table. For some reason the other schema's I can see the IM_xxxxxx.DimDate table in sql analytics endpoint. These tables are 100% the same.
In this example it is just a simple table, no PK's whatsoever.
So unfortunately I have this problem for quite a lot of tables in different schema's.
Other tables that I do not see at all in the sql analyitcs endpoint doesn't have a 'Failed', 'Stopped', 'RunningWithWarnings' status.
Minor table changes also doens't trigger the visibility in sql analytics endpoint.
The error that you see in the screenshot: 'Table count exceeds limit', is that a cause? or just a UI thing?
Thank you in advance!
Hi DKIDV,
Table count exceeds limit is the cause. It is not just a UI message. May be The issue is not caused by permissions or refresh problems the main reason your tables are missing in the SQL Analytics Endpoint is because your database has hit the internal table count limit. Once this limit is reached, the SQL Analytics Endpoint simply stops registering new tables, even though replication still shows as “Running” and the tables exist in the SQL Database. This is why older schemas appear while newer ones don’t, even when the tables are identical. The refreshMetadata API and small table changes won’t fix it because the endpoint is no longer accepting new metadata. The only real solutions are to reduce the number of tables in the database (drop unused or duplicate tables) or split your data across multiple databases using Fabric Pipelines to move data between them. This avoids the limit and ensures all tables become visible and sharable again.
Thanks & Regards,
Prasanna Kumar
Hi Anonymous ,
Thank you for the reply!
That's a pity if the internal table count limit will cause the problem. I will to create more fabric sql databases to orchestrate the data once more, so it will be available in sql analytics endpoint and eligible to do external data sharing!
Is it possible to get data from database A en store it in database B? Like SSIS?Hi DKIDV,
You can move or copy data from Database A to Database B in Microsoft Fabric using a Data Pipeline, which works like SSIS. The pipeline uses a Copy Activity that reads tables from Database A and writes them into Database B. Once the tables are copied, they appear as new tables in Database B and become visible again in the SQL Analytics Endpoint, as long as the table limit is not exceeded. For quick or small transfers, you can also use SQL commands such as INSERT INTO. SELECT or CREATE TABLE AS SELECT (CTAS) to copy data between databases, if permissions allow.
How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
Thanks & Regards,
Prasanna Kumar