Forum Discussion

JoergNeulist's avatar
JoergNeulist
Regular Visitor
1 year ago

Getting modification time in copy activity

Hello everyone! We are ingesting data from an Azure storage account. The pipeline has a copy activity to the OneLake/Files storage and a small notebook to translate the data into OneLake/Tables. Th...
Need Help
pipelines
frithjof_v's avatar
frithjof_v
Community Champion
1 year ago

Is it possible to use the Get Metadata activity?

  • JoergNeulist's avatar
    JoergNeulist
    Regular Visitor
    1 year ago

    I never considered this! It's a good pointer - but what next? I can't put the metadata into a variable in the notebook ("Error: Unable to convert metadata to string.")

    I see that Set Variable would accept arrays, but how do I get that from inside the Notebook?

    • frithjof_v's avatar
      frithjof_v
      Community Champion
      1 year ago

      You can pass variables from the pipeline into the Notebook:

       

      https://learn.microsoft.com/en-us/fabric/data-factory/notebook-activity#notebook-settings

       

      https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#integrate-a-notebook

       

      How do you do the copy, btw? Are you using a loop to run through the files in the Azure Storage and copy every one of them?

       

      If so, you could perhaps use the outputs from the Get Metadata activity to select which files to copy before copying them with the Copy Activity?

       

      EDIT: If you need to convert between array and string, or other manipulations, you could have a look and see if some functions can work for you: 

       

      https://learn.microsoft.com/en-us/fabric/data-factory/expression-language#functions

       

      E.g.

       

      • join
      • split
      • array
      • string

       

      and Activities

      https://learn.microsoft.com/en-us/fabric/data-factory/activity-overview#control-flow-activities,

       

      e.g.

      • Filter
      • JoergNeulist's avatar
        JoergNeulist
        Regular Visitor
        1 year ago

        So, the original pipeline is Copy -> Notebook1 -> Notebook2 which is basically the ELT steps. The Copy gets all files as binary from the Azure source using flatten hierarchy. Notebook1 interprets the files (which are basically CSV, but with some weird details that dataflows can't handle) and adds their content to the table, and Notebook2 does some aggregations on all of the data (fresh and old).

        Splitting this using Foreach would create quite some overhead, because the notebooks do have some setting up that is only done once. Also, I'm really not keen on rewriting the whole thing.

         

        So I tried experimenting with Get Metadata -> Notebook to try interpreting the data.

        Weirdly enough, it takes the combination

        @join(array(activity('Get Metadata1').output), '$')

        to actually work. Unfortunately, that's still not what I'm looking for. It's some telemetry of the pipeline step, not the metadata I was looking for.

         

        I really appreciate you taking the time to walk me through this, but I would much prefer some more detailed documentation or a debugging mode.