Forum Discussion

ToddChitt's avatar
ToddChitt
Icon for Super User rankSuper User
2 years ago

Save PySpark datafram to Fabric Warehouse via T-SQL type MERGE operation

I am building out a solution that will need to process lots of JSON files. I initially set it up to save the JSON file to a known location and known file name in the Lakehouse, then use a Dataflow Ge...
data engineering
data warehouse
Element115's avatar
Element115
Icon for Memorable Member rankMemorable Member
2 years ago

Here's  a suggestion. Since you are using sp, I'd create all the necessary sp for the warehouse, then orchestrate everything from within one or more pipelines with the Stored procedure activity and any other activity required.  Pipelines are parametrizable.  Also, with a Script activity you can query the SQL endpoint of your LH, get all the metadata you need, and then use that with your sp and push it out to the WH.