Hello,
I'm working on a pipeline with a Notebook (Python) and a Dataflow Gen2, but I'm struggeling with a problem.
Here is the basic setup:
1. I've setup a Python-Notebook that runs every night an fetches data for the previous day from a web api. This data is in a flat format and consists of dim- and fact-data in one row.
2. The daily data is written to a Lakehouse table "RawData", the mode is "overwrite" to ensure, that only the data from the previous day is being stored in raw format and not all data from past days.
3. I've also setup a Dataflow Gen2 that reads the data from the table "RawData" and separates it in dim data and fact data.
4. Dim data is written to a table "DimData" in the Lakehouse, fact data is written to a table "FactData".
5. The update method in the dataflow is set to "Append".
Here are the results:
1. The daily data is written to table "RawData". RawData contains only the data from the previous day. This is the expected behaviour and works fine.
2. The Dataflow separates the RawData in dim data and fact data and writes this data to separate Lakehouse tables "DimData" and "FactData". This works in principle, but not as expected (see 3).
3. The daily data should be appended to "DimData" and "FactData", so that DimData and FactData contain the date from the previous day as well as existing data from past days. This doesn't work. Instead existing data in DimData and FactData is ovewritten, so that both tables contain only data from the preivious day.
I hope this was understandable. Does anyone have any ideas whats going wrong?
Thanks,
Udo
