Forum Discussion

holgergubbels's avatar
holgergubbels
Frequent Visitor
7 months ago
Solved

PySpark Update Warehouse

Hi community, I really have problems implementing a medallion architecture in fabric - I always struggle at some point with (I thought) easiest things... It is really frustrating - and Copilot/chatg...
environment
  • holgergubbels's avatar
    holgergubbels
    7 months ago

    Hi chetanhiwale ,

    thanks for your answer! I decided to use another approach:

    - Load to landing: pipeline using copy activity, logs/watermarks via SP in Warehouse

    - Raw to Bronze: Piepline for logging, watermark, call notebook for the load.

    - same with bronze to silver

     

    It is not the best approach I think, because the spark session is always startet and temrinated for every job. But it is the best I could find... 

     

    Holger

tayloramy's avatar
tayloramy
Super User
7 months ago

Hi holgergubbels, 

 

The way that I would typically approach this is to just work with the lakehouses natively. I store my metadata in a Lakehouse as it makes it nice and easy to interact with in notebooks. 

 

Then when updating records, I will filter on records where completed (or another similar flag) is null and update the most recent record (which in my case corresponds with the current run) for whatever table/process is currently running.