Pipeline - Set Variable activity not working properly

[narendranj]

Trying to update a pipeline variable value through `Set Variable` activity and it is getting success but not reflected the new value in the variables tab or when accessing this variable on any other activity. Always showing the default value that I set when creating that variable.

Variable - The `last_run_date` is the variable and it's defualt value is `2024-01-01`:

Pipeline activities:

Web activity - Using that variable in the URL by setting dynamic content:

Set Variable activity - After the successful run of web activity, updating that variable value as current date:

Run results of FIRST time:

Web activity - The defult value of `last_run_date` is set correctly:

 Set Variable activity - The current date `2024-05-21` is set correctly:

Variable - The default vaule is not being updated:

Run results of SECOND time:
Web activity - The updated value (2024-05-21) of `last_run_date` is not set correctly, still it is taking the default value (2024-01-01) that I set initially:

[Ray_Minds · Accepted solution]

Based on my understanding, variables values are only available for current session. we couldn't be used after the session closed. so you will need to store variable "Last_run_date" value in some locations like (blob storage, data lake etc.. as per your convenience).
Please follow the below step :-
1. Choose the lookup activity then configure the connection where have been to store location of variable "Last_run_date".

2. Now you choose the webactivity and configure the "Relative URL" as per mentioned in the below:-
api/v1/employees? date = @activity('Lookup').output.value.

3. Now you add a variable(Last_run_date) as you mentioned.

4. Then choose the copy activity that help to store "last_run_date" variable. please follow below step to config.
a). Create a dummy dataset as source. go to "additional column" >> add column name like ("Last_run_date")
>>>> go the column value and click on "add dynamically content" then select the variables.
b). Create a sink dataset where to store "Last_run_date" value.

I hope that would be helpful for you.

[narendranj]

Noted, thanks! Yes, this is how I already implemented the workaround.

Regarding the behavior of the Set Variable activity: I have a different view from its current functionality. I don't see the benefit of holding variable values only for the current session. I believe it should be designed to dynamically update pipeline variable values.

[v-cboorla-msft]

Hi @narendranj 

Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.

If the issue still persist, please do let us know. We will try to help.

Thank you.

[v-cboorla-msft]

Hi @narendranj 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

[v-cboorla-msft]

Hi @narendranj 

Thanks for using Microsoft Fabric Community.

Could you please try to interchange the connection between Web and Set variable and then try to execute the pipeline. I have tried to repro the scenario, please find the screenshots for reference.

If the issue still persists, please do let us know. Glad to help.

I hope this information helps.

Thank you.

[narendranj]

@v-cboorla-msft Thanks for the response. My scenario is a bit different: I need to set `last_run_date` to the current date once all API calls are completed. This current date will be used by my API to fetch records from after this date the next time it runs. My pipeline will be running on a scheduled basis. Could you please suggest how I can achieve this?