In pipeline, Lookup activity does not query updated version of delta table in previous notebook

[pshepunde]

Hello,

In Fabric pipeline, I have first activity, a notebook which updates list of files to be processed. Second activity, a LookUp which query the same table to get a list of files to be processed. But it gets 0 records so pipeline stops. When I use LookUp query for the same table in SQL Endpoint, I find many rows matching conditions. I hope, my question is clear and you can help me resolve this issue. 

[pshepunde]

@v-nikhilan-msft 

[v-nikhilan-msft]

Hi @pshepunde 

Thanks for using Fabric Community.

Can you please send the me the screenshot of the lookup activity details? What is the query which you are using in SQL endpoint?

These details will help me to understand the issue better.

Thanks

[pshepunde]

select distinct date_folder
from deltaload_info
where is_processed = 'N'
and schema_name='UNICPROD'
and date_folder >= '2024/05/08'
order by date_folder

[pshepunde]

Here is the screenshot of Advance settings. I tried with isolation level as both Read committed and uncommitted. But its not working

[v-nikhilan-msft]

Hi @pshepunde 
I tried to repro the above scenario :

I did not face any issue here. The SQL endpoint data matches with the output of the lookup. Hence I am unable to understand what might be the root cause for this.

I request you to create a support ticket, so that our internal team can take a close look on this to find out the reason behind this.

https://support.fabric.microsoft.com/en-US/support/

After creating the ticket please provide the details here so that we can track it internally.

Thanks

[v-nikhilan-msft]

Hi @pshepunde 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks

[pshepunde]

I am yet to create a ticket as you have mentioned. Issue is still open and I am seeking help to get it resolved. For more details, please note that notebook has a user defined function in Pyspark which is merging into the table. This table is subsequently used in next lookup activity. If possible, please try this as well

[v-nikhilan-msft]

Hi @pshepunde 
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please share the details here.
Thanks