Forum Discussion

narendranj's avatar
narendranj
Frequent Visitor
2 years ago
Solved

Pipeline - Set Variable activity not working properly

Trying to update a pipeline variable value through `Set Variable` activity and it is getting success but not reflected the new value in the variables tab or when accessing this variable on any other activity. Always showing the default value that I set when creating that variable.
Variable - The `last_run_date` is the variable and it's defualt value is `2024-01-01`:

Pipeline activities:

Web activity - Using that variable in the URL by setting dynamic content:

Set Variable activity - After the successful run of web activity, updating that variable value as current date:

Run results of FIRST time:

Web activity - The defult value of `last_run_date` is set correctly:

 

 Set Variable activity - The current date `2024-05-21` is set correctly:


Variable - The default vaule is not being updated:

Run results of SECOND time:

 

Web activity - The updated value (2024-05-21) of `last_run_date` is not set correctly, still it is taking the default value (2024-01-01) that I set initially:
 
Data Pipeline
  • Ray_Minds's avatar
    Ray_Minds
    2 years ago

    Based on my understanding, variables values are only available for current session. we couldn't be used after the session closed. so you will need to store variable "Last_run_date" value in some locations like (blob storage, data lake etc.. as per your convenience).
    Please follow the below step :-
    1. Choose the lookup activity then configure the connection where have been to store location of variable "Last_run_date".

    2. Now you choose the webactivity and configure the "Relative URL" as per mentioned in the below:-
    api/v1/employees? date = @activity('Lookup').output.value.

    3. Now you add a variable(Last_run_date) as you mentioned.

    4. Then choose the copy activity that help to store "last_run_date" variable. please follow below step to config.
    a). Create a dummy dataset as source. go to "additional column" >> add column name like ("Last_run_date")
    >>>> go the column value and click on "add dynamically content" then select the variables.
    b). Create a sink dataset where to store "Last_run_date" value.

    I hope that would be helpful for you.

6 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi narendranj 

     

    Thanks for using Microsoft Fabric Community.

    Could you please try to interchange the connection between Web and Set variable and then try to execute the pipeline. I have tried to repro the scenario, please find the screenshots for reference.

    If the issue still persists, please do let us know. Glad to help.

     

    I hope this information helps.

     

    Thank you.

    • narendranj's avatar
      narendranj
      Frequent Visitor
      2 years ago

      v-cboorla-msft Thanks for the response. My scenario is a bit different: I need to set `last_run_date` to the current date once all API calls are completed. This current date will be used by my API to fetch records from after this date the next time it runs. My pipeline will be running on a scheduled basis. Could you please suggest how I can achieve this?

  • Ray_Minds's avatar
    Ray_Minds
    Solution Supplier
    2 years ago

    Based on my understanding, variables values are only available for current session. we couldn't be used after the session closed. so you will need to store variable "Last_run_date" value in some locations like (blob storage, data lake etc.. as per your convenience).
    Please follow the below step :-
    1. Choose the lookup activity then configure the connection where have been to store location of variable "Last_run_date".

    2. Now you choose the webactivity and configure the "Relative URL" as per mentioned in the below:-
    api/v1/employees? date = @activity('Lookup').output.value.

    3. Now you add a variable(Last_run_date) as you mentioned.

    4. Then choose the copy activity that help to store "last_run_date" variable. please follow below step to config.
    a). Create a dummy dataset as source. go to "additional column" >> add column name like ("Last_run_date")
    >>>> go the column value and click on "add dynamically content" then select the variables.
    b). Create a sink dataset where to store "Last_run_date" value.

    I hope that would be helpful for you.

    • narendranj's avatar
      narendranj
      Frequent Visitor
      2 years ago

      Noted, thanks! Yes, this is how I already implemented the workaround.


      Regarding the behavior of the Set Variable activity: I have a different view from its current functionality. I don't see the benefit of holding variable values only for the current session. I believe it should be designed to dynamically update pipeline variable values.

       

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi narendranj 

         

        Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.

        If the issue still persist, please do let us know. We will try to help.

         

        Thank you.