Forum Discussion
DataLake to DataWarehouse
You could perhaps use Dataflows Gen2 with Append method.
Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
This should work similarly both for Lakehouse or Warehouse.
Or just use a cross-database query to get the Lakehouse data into your Warehouse, as shown in the previous comment (however it doesn't work for Lakehouse files, only Lakehouse tables).
I don't think Notebook can write data to Warehouse. So I don't think what you are asking is directly possible.
I think you will need to use some other tool than Notebook (Dataflows Gen2, T-SQL script or stored procedure, data pipeline, etc.).
Or you can use Notebook to enter data into a Lakehouse table, and then use cross-database queries to make the data available in the Warehouse.If you want to use Notebook to load data, why not just use Lakehouse instead of Warehouse? I am curious.
If someone knows how to use Notebook to write data to Warehouse, I am interested to know.
Right now we can't write to a Warehouse without going through the Warehouse SQL Endpoint (we can read via Spark as you know). So the only way to work with data is using a connectivit method like pyodbc and connecting to the Warehouse SQL Endpoint in the Notebook.
I'll dig up an example I did a whle ago to show this.
Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn
Hi reded1 ,
Transferring files from Lakehouse to Warehouse using a notebook can be a little difficult, I have a more convenient solution here.
Once you're in the warehouse in Fabric, select the place I've highlighted.
Select the Lakehouse you want to connect to.
Now it's time to use your connected Lakehouse.
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support Team
If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
So I am using mendelian architecture to file movement from the bronze lakehouse layer to the warehouse
So I do know that its possible with pipeline.
but I want to know the solution from the notebook
where I am trying to move raw selected files to warehouse delta
So whenever new incremental load happens the new data will gett append to the warehouse table on the incremental load bases.
if you have more optimized or better way to move data from lakehouse to warehouse using medallion architecture please let me know
I am loading data from on premises database to lakehouse it will be a full load (parquet) storing as a database/year/month/date/table name
now also getting incremental data from same database on daily basis.
now I want to add these incremental load to silver warehouse
please provide good way do I can run trigger daily and update my silver warehouse layer
thanks,
Red
frithjof_vCommunity Champion
You could perhaps use Dataflows Gen2 with Append method.
Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
This should work similarly both for Lakehouse or Warehouse.
Or just use a cross-database query to get the Lakehouse data into your Warehouse, as shown in the previous comment (however it doesn't work for Lakehouse files, only Lakehouse tables).
I don't think Notebook can write data to Warehouse. So I don't think what you are asking is directly possible.
I think you will need to use some other tool than Notebook (Dataflows Gen2, T-SQL script or stored procedure, data pipeline, etc.).
Or you can use Notebook to enter data into a Lakehouse table, and then use cross-database queries to make the data available in the Warehouse.If you want to use Notebook to load data, why not just use Lakehouse instead of Warehouse? I am curious.
If someone knows how to use Notebook to write data to Warehouse, I am interested to know.
- AndyDDC
Most Valuable Professional
Right now we can't write to a Warehouse without going through the Warehouse SQL Endpoint (we can read via Spark as you know). So the only way to work with data is using a connectivit method like pyodbc and connecting to the Warehouse SQL Endpoint in the Notebook.
I'll dig up an example I did a whle ago to show this.
Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn