Forum Discussion

amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago
Solved

Data Pipeline says Invalid object name even though it exists

I have a Data Pipeline that performs several steps one after the other:

 

1. I read some data from a SQL Server and write it in a table t1 of a lakehouse L1.
2. (Yellow) I wait 2 seconds (I will explain why I do this later).
3. (Red) With a notebook I read this table t1 that I have just written and I write it in another lakehouse L2 in a table t2.
4. (Blue) I run a query on t1 to get the smallest value of one of its columns (select min(...) from .....)
5. Other operations.

 

I run this Data Pipeline every day, and some days it works correctly, but other days it fails in step 4 (blue). It gives me an error that the table t1 on which I am performing the query does not exist, which I don't understand because in the previous step (red) I have read via notebook that table t1 and it has not given me any error.

 

 

I put a wait of 2 seconds to give the flow time to load the table correctly, but sometimes it doesn't even find it.

 

I don't understand how in step 4 it doesn't find the table t1 if in step 3 it has found it. I don't understand how in a step after step 1 (step in which the table is written) t1 is not found. I put a timeout but I don't understand why I put it, as it should not be necessary.

1 Reply