Hi all,
I'm currently building a Fabric Finance Cube, and I have an architecture like so:
Pipelines for each: Dimension tables and Fact tables using Copy Data activity from Google Cloud + One DataFlow to bring another fact table from SQL Server (will adjust this to a Pipeline but our gateway is out of date and need to wait for IT to update). This data gets stored in an ingestion-layer lakehouse (all of this is one workspace).
Then, using DataFlows Gen2, I take the raw data from the ingestion Lakehouse and perform transformations (mostly consolidating the dimension and fact tables), and push the consolidated table to the final modeling lakehouse where I use the default semantic model to build up the cube.For testing purposes, I limited the data publishing to max of 5 rows and will set it to publish full data when I'm happy with it, so I don't need to wait 15 minutes just to see "error"
Problem is, this an interative process, and I build on things, I need to adjust the schemas of the consolidated dimension or fact tables (add a column here and there). But when I go in to the data destination, the new columns are never there. Neither in the destination lakehouse, nor in the destination settings (the cog in the DF). I've ensured that it's the correct data type, and my destination does not have any relationships or measures. I've tried swapping between new/existing table. The only solution to get the new columns to show up is to delete the table in the destination lakehouse manually, then re-do the destination settings. Very frustrating.
Am I just missing something here? Would be really grateful for any advice!
