Forum Discussion
PySpark Update Warehouse
Hi chetanhiwale ,
thanks for your answer! I decided to use another approach:
- Load to landing: pipeline using copy activity, logs/watermarks via SP in Warehouse
- Raw to Bronze: Piepline for logging, watermark, call notebook for the load.
- same with bronze to silver
It is not the best approach I think, because the spark session is always startet and temrinated for every job. But it is the best I could find...
Holger
Hi holgergubbels ,
We had the same issue during our implementation. These are the things which we implemented.
1. Checks the logs types as data and fabric items logs/audits.
Data logs/audits were written using JDBC.
Fabric Item logs/audits were done using stored proc in a pipeline.
2. For data logs/audits we used JDBC connection. This was easy to use and was able to run our stored proc. We were fetching our secrets from Keyvault.
Honestly, I dont recommend adding logs/audits using Synapse sql connector for writng any data. Although its best for reading the data.
For your case the best option will be
1. Create your stored proc in Warehouse
2. Add secrets of your warehouse in Azure Key vault
3. Fetch those secrets in runtime in notebook
4. Use those credentials to hit the stored proc using Pyodbc module.
Hope this helps. Lets me know if you have any doubts
Hi chetanhiwale ,
thanks for your answer! I decided to use another approach:
- Load to landing: pipeline using copy activity, logs/watermarks via SP in Warehouse
- Raw to Bronze: Piepline for logging, watermark, call notebook for the load.
- same with bronze to silver
It is not the best approach I think, because the spark session is always startet and temrinated for every job. But it is the best I could find...
Holger