Forum Discussion

munindra's avatar
munindra
Frequent Visitor
1 year ago
Solved

passing the dynamically created file to notebook

hello all, i made the datapipeline while passing  the source from http and used @concat('user_1',utcNow(),'.txt') expression

to make the destination file in copy activity and i am trying to pass this filename to notebook as parameter but notebook execution getting failed.could any one help with the expression for accessing the filename of copy data activity dynamically to the notebook or suggest me any other ways to it.

 

data engineering
data pipeline
notebook
  • spencer_sa's avatar
    spencer_sa
    1 year ago

    Firstly, in order to avoid race conditions on using utcNow in multiple places and getting different answers, I'd probably create a pipeline variable for the filename and 'Set Variable activity' it to be @concat('user_1', utcNow(), '.txt')
    Then I'd use that variable as both the destination filename *and* a parameter in the Notebook activity.  And then in the Notebook I'd have a parameter cell that uses the filename parameter.
    So:
    Pipeline


    Variable setup

    Set Variable activity

    Copy Data Activity destination

    Notebook activity

    And the notebook itself

     

    If this helps, please consider Accepting as a Solution so others can find it more easily.

1 Reply

  • spencer_sa's avatar
    spencer_sa
    Impactful Individual
    1 year ago

    Firstly, in order to avoid race conditions on using utcNow in multiple places and getting different answers, I'd probably create a pipeline variable for the filename and 'Set Variable activity' it to be @concat('user_1', utcNow(), '.txt')
    Then I'd use that variable as both the destination filename *and* a parameter in the Notebook activity.  And then in the Notebook I'd have a parameter cell that uses the filename parameter.
    So:
    Pipeline


    Variable setup

    Set Variable activity

    Copy Data Activity destination

    Notebook activity

    And the notebook itself

     

    If this helps, please consider Accepting as a Solution so others can find it more easily.