topic Re: Copy job, Lakehouse changing ids causing code to break. in Data Engineering

Copy job, Lakehouse changing ids causing code to break.

yllsuarez76 — Thu, 21 Nov 2024 20:24:45 GMT

I am working on my feature branch using a copyjob to push data from a remote SQL into a lakehouse.

Once I merge my code to the main branch and open the production workspace the copyjob is broke, for obvious reasons, the lakehouse id changed.

Question is, how to mantain the code for jumping between branches and not breaking during merges.

Re: Copy job, Lakehouse changing ids causing code to break.

spencer_sa — Thu, 21 Nov 2024 23:05:30 GMT

We maintain a table of workspace/lakehouse ids by name and environment and look these up at pipeline run time.
You can automate the creation of this table using sempy-labs, the API, and/or notebookutils.lakehouse.get.
Technically you could also dynamically look these up in a notebook that is ran before any other job that outputs in its exitValue the list of lakehouses and ids.

Re: Copy job, Lakehouse changing ids causing code to break.

Anonymous — Fri, 22 Nov 2024 07:58:29 GMT

HI @yllsuarez76,

AFAIK, current it seems not support these feature, perhaps you can submit an idea to improve the 'copy job' usage:

Microsoft Fabric Ideas

Regards

Xiaoxin Sheng

Re: Copy job, Lakehouse changing ids causing code to break.

Anonymous — Fri, 03 Jan 2025 06:20:10 GMT

Hi @yllsuarez76 ,

Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

Regards,

Xiaoxin Sheng