Forum Discussion
DataLake to DataWarehouse
- 2 years ago
You could perhaps use Dataflows Gen2 with Append method.
Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
This should work similarly both for Lakehouse or Warehouse.
Or just use a cross-database query to get the Lakehouse data into your Warehouse, as shown in the previous comment (however it doesn't work for Lakehouse files, only Lakehouse tables).
I don't think Notebook can write data to Warehouse. So I don't think what you are asking is directly possible.
I think you will need to use some other tool than Notebook (Dataflows Gen2, T-SQL script or stored procedure, data pipeline, etc.).
Or you can use Notebook to enter data into a Lakehouse table, and then use cross-database queries to make the data available in the Warehouse.If you want to use Notebook to load data, why not just use Lakehouse instead of Warehouse? I am curious.
If someone knows how to use Notebook to write data to Warehouse, I am interested to know.
- 2 years ago
Right now we can't write to a Warehouse without going through the Warehouse SQL Endpoint (we can read via Spark as you know). So the only way to work with data is using a connectivit method like pyodbc and connecting to the Warehouse SQL Endpoint in the Notebook.
I'll dig up an example I did a whle ago to show this.
Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn
You could perhaps use Dataflows Gen2 with Append method.
Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
This should work similarly both for Lakehouse or Warehouse.
Or just use a cross-database query to get the Lakehouse data into your Warehouse, as shown in the previous comment (however it doesn't work for Lakehouse files, only Lakehouse tables).
I don't think Notebook can write data to Warehouse. So I don't think what you are asking is directly possible.
I think you will need to use some other tool than Notebook (Dataflows Gen2, T-SQL script or stored procedure, data pipeline, etc.).
Or you can use Notebook to enter data into a Lakehouse table, and then use cross-database queries to make the data available in the Warehouse.
If you want to use Notebook to load data, why not just use Lakehouse instead of Warehouse? I am curious.
If someone knows how to use Notebook to write data to Warehouse, I am interested to know.
Right now we can't write to a Warehouse without going through the Warehouse SQL Endpoint (we can read via Spark as you know). So the only way to work with data is using a connectivit method like pyodbc and connecting to the Warehouse SQL Endpoint in the Notebook.
I'll dig up an example I did a whle ago to show this.
Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn