Multi Step Pipelines not seeing table updates from prior steps

[kdelrio]

(noob) I have a Fabric pipeline that calls multiple dataflows. The first dataflow Gen2 writes its output to a Datalake table, and the second dataflow uses that same table to continue processing the data. However, when I run the flow, it seems that the subsequent dataflow does not see the updated table rows. If I run the pipeline multiple times, it works fine, but it is very slow. I tried adding a "wait" step to give the table time to update, but that did not improve the performance. Is there any way to ensure that all database table updates are complete and committed prior to proceeding to the next step? 

[kdelrio]

Hi @v-cboorla-msft
Thank you for checking on me! No, I was not able to resolve the issue. While using the trigger of 'upon successful completion' of the previous one should work, there is still some delay before the data is written/available for the next activity. I tried using the lookup activity, but it only returns a full table (see screenshot below) in Fabric, which is way too much data in this circumstance. Is there a way to write an SQL for the lookup to only return a Select Distinct query against one column? Any other thoughts on how to pause until the updated data is available? (Right now the only way i can get it to work is by putting a 300-sec wait between each step)

[v-cboorla-msft]

Hi @kdelrio 

Apologize for the delay in reponse from my end.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[v-cboorla-msft]

Hi @kdelrio 

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thanks.

[Ayybee]

Since the trigger of the preceding activities is upon completion of the previous one. So the data is written, you can confirm it by looking into the output of activity in the progress box.

What you can do is use lookup activity to specify the data you want to work on before implementing the next step.

[v-cboorla-msft]

Hi @kdelrio 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.