topic Re: working with a Bronze Delta Lake and a Silver dataLake in Data Engineering

working with a Bronze Delta Lake and a Silver dataLake

DebbieE — Fri, 28 Jun 2024 14:03:48 GMT

I have created a Notebook and added the Bronze Delta Lake to transform the data in there

Now I want to create Delta tables from them

df.write.mode("overwrite").parquet('Files/Data/Silver/datatransformed.parquet')

However, this just creates a Filver Folder in the Delta Lake and I want to save to another data lake? SilverTrainingDL

Is there any way of doing this?

jjaeger94 — Fri, 28 Jun 2024 16:23:59 GMT

If you add the Silver Lakehouse to your notebook and then

Copy the ABFS path to the files, that will be the path you want to use to write the file to.

frithjof_v — Fri, 28 Jun 2024 18:04:49 GMT

If you want to create delta table, you can use saveAsTable function.

DebbieE — Mon, 01 Jul 2024 10:01:10 GMT

I have the Bronze Lakehouse added but I dont know how to add another Lakehouse tob the notebook so i have two?

Ahhhh OK, You can add another one. Lets see if I can do the above......

When I right click on the Tables folder in the notebook all I get is refresh. No copy ABFS path because nothing is there yet.

DebbieE — Mon, 01 Jul 2024 10:27:59 GMT

Ah It worked when I did this

df.write.mode("overwrite").option("overwriteSchema", "true").format("delta").saveAsTable("GoldTraininglh.dimContestant")