Forum Discussion

holgergubbels's avatar
holgergubbels
Frequent Visitor
7 months ago
Solved

PySpark Update Warehouse

Hi community, I really have problems implementing a medallion architecture in fabric - I always struggle at some point with (I thought) easiest things... It is really frustrating - and Copilot/chatg...
environment
  • holgergubbels's avatar
    holgergubbels
    7 months ago

    Hi chetanhiwale ,

    thanks for your answer! I decided to use another approach:

    - Load to landing: pipeline using copy activity, logs/watermarks via SP in Warehouse

    - Raw to Bronze: Piepline for logging, watermark, call notebook for the load.

    - same with bronze to silver

     

    It is not the best approach I think, because the spark session is always startet and temrinated for every job. But it is the best I could find... 

     

    Holger

Anonymous's avatar
Anonymous
Not applicable
7 months ago

Hi holgergubbels,

 

Thank you for reaching out to the Microsoft Fabric Forum Community, and special thanks to apturlov , tayloramy  deborshi_nag  for prompt and helpful responses.

Just following up to see if the Response provided by community members were helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.

 

Best regards,
Prasanna Kumar