topic Re: Get Metadata of All Tables in Pipelines

Get Metadata of All Tables

Anonymous — Wed, 13 Nov 2024 02:56:34 GMT

Hi everyone, I have experience taking metadata info of all Files in a folder in the Get Metadata activity but I'm not sure how I can do the same for Tables.

I want to take the name of all of the tables available in Get Metadata, filter for everything except certain tables in Filter and then pass them to be copied from one lakehouse to another through a For Each activity with a Copy Data activity.

I think I have to write a dynamic expression for this but not sure what it would be.

Does anyone know how to do this?

Re: Get Metadata of All Tables

spencer_sa — Wed, 13 Nov 2024 09:37:07 GMT

It appears that the Get Metadata activity when set to look at tables requires there to be a table name supplied, so you can't just say 'enumerate all tables'.
Given that limitation, the way I'd probably handle this is a notebook that (either via API calls or via sempy-labs) gets a list of tables and outputs them via notebookutils.notebook.exit(<JSON string of list of table names>), convert to an array and then use that list in your filter.

Re: Get Metadata of All Tables

ifistic — Tue, 19 Nov 2024 01:07:39 GMT

my metadata pipeline say taht my inputfile is a folder. how can i solve it

Re: Get Metadata of All Tables

spencer_sa — Tue, 19 Nov 2024 10:34:38 GMT

Sounds like a different problem to the OP's one - they were after enumerating Tables.
If you're after enumerating Files inside Folders, then it'll depend on whether you always know the Folder it lives in or not.
If you know the folder, then just add that to the metadata step;

If you don't know the folder name until runtime, then you're into a ForEach with an If condition on FileType = 'Folder', and then feed the item name (which is the folder name) into a nested Invoke Pipeline step as a Parameter. This sub-pipeline then uses Get Metadata on the subfolder.
You could (and it's not necessarily a good idea) make this recursive to traverse any depth of folder structure.

Re: Get Metadata of All Tables

NandanHegde — Tue, 19 Nov 2024 12:21:43 GMT

Can you please be more clear what you are trying to achieve? Do you want to get list of all tables in lakehouse? And what do you want to do