Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
amaaiia
Super User
Super User

Data Pipeline says Invalid object name even though it exists

I have a Data Pipeline that performs several steps one after the other:

amaaiia_0-1720436974866.png

 

1. I read some data from a SQL Server and write it in a table t1 of a lakehouse L1.
2. (Yellow) I wait 2 seconds (I will explain why I do this later).
3. (Red) With a notebook I read this table t1 that I have just written and I write it in another lakehouse L2 in a table t2.
4. (Blue) I run a query on t1 to get the smallest value of one of its columns (select min(...) from .....)
5. Other operations.

 

I run this Data Pipeline every day, and some days it works correctly, but other days it fails in step 4 (blue). It gives me an error that the table t1 on which I am performing the query does not exist, which I don't understand because in the previous step (red) I have read via notebook that table t1 and it has not given me any error.

 

amaaiia_2-1720437331315.png

 

I put a wait of 2 seconds to give the flow time to load the table correctly, but sometimes it doesn't even find it.

 

I don't understand how in step 4 it doesn't find the table t1 if in step 3 it has found it. I don't understand how in a step after step 1 (step in which the table is written) t1 is not found. I put a timeout but I don't understand why I put it, as it should not be necessary.

1 ACCEPTED SOLUTION
frithjof_v
Super User
Super User

Is step 4 reading from the SQL Analytics Endpoint?

 

If so, my guess is that there is some (varying) delay from the creation of the table in Spark until it is synced to the SQL Analytics Endpoint ("metadata sync").

 

https://community.fabric.microsoft.com/t5/General-Discussion/Delayed-data-refresh-in-SQL-Analytical-...

 

https://community.fabric.microsoft.com/t5/Data-Engineering/SQL-Endpoint-Slow-To-Reflect-Changes-In-L...

 

However, these docs say that the new table should be available immediately in the SQL Analytics Endpoint: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-sql-analytics-endpoint

View solution in original post

1 REPLY 1
frithjof_v
Super User
Super User

Is step 4 reading from the SQL Analytics Endpoint?

 

If so, my guess is that there is some (varying) delay from the creation of the table in Spark until it is synced to the SQL Analytics Endpoint ("metadata sync").

 

https://community.fabric.microsoft.com/t5/General-Discussion/Delayed-data-refresh-in-SQL-Analytical-...

 

https://community.fabric.microsoft.com/t5/Data-Engineering/SQL-Endpoint-Slow-To-Reflect-Changes-In-L...

 

However, these docs say that the new table should be available immediately in the SQL Analytics Endpoint: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-sql-analytics-endpoint

Helpful resources

Announcements
Fabric July 2025 Monthly Update Carousel

Fabric Monthly Update - July 2025

Check out the July 2025 Fabric update to learn about new features.

August 2025 community update carousel

Fabric Community Update - August 2025

Find out what's new and trending in the Fabric community.