Copy tables and files (quickly) between two lakehouses with same name

[Scott_Powell]

Hi, we're experimenting with CI/CD and using the whole "branch out" git functionality. As part of this, I'll want the ability to copy any and all files and tables from one workspace into another. Is there an easy way to do this?

The copy activity in a pipeline seems to require knowing the list of tables and files beforehand, making it inflexible as things change. 

Using a notebooks seems the best way - but I'm uncertain how to "attach" both Lakehouses when they have the same name. 

Any idea how to do this? CoPilot hasn't been much help...

Thanks,

Scott

[Anonymous]

Hi @Scott_Powell 

Another CI/CD experience in Fabric is Deployment pipelines. If you want to copy lakehouses from one workspace to another workspace directly, you can create a deployment pipeline to do this task. The files and tables in a lakehouse can still remain their original names, and the lakehouse in the destination stage worksapce can also have the same name as the source one. This might be an alternative. 

Get started using deployment pipelines, the Fabric Application lifecycle management (ALM) tool - Mic...

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

[frithjof_v · Accepted solution]

Regarding Notebook: I think you can use a combination of abfss path and mounting (which relies on abfss path). Notebookutils can also be a helpful tool for such use cases (getting abfss paths dynamically, mounting lakehouses, file system utilities, etc.). NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

For transferring tables, you could use the abfss path of the lakehouses.

You could mount the lakehouses, this is useful for transferring files, I think (using the local directories, i.e. relative paths). How To Mount A Lakehouse and Identify Mounted Lakehouses in Fabric

Or perhaps you can use the abfss path directly for transferring files also.