Forum Discussion
[Fabric] How can I load incremental data from a PostgreSQL database using Dataflows or Data Factory?
I am currently working on loading a large database from PostgreSQL to several Lakehouses within a workspace. This is part of implementing the medallion lakehouse architecture, specifically the first ...
Thank you for the information. Unfortunately, it is not possible for us to implement this solution because several of our tables do not have a watermark. What we have found works best for loading data from PostgreSQL is using the WAL.
Not really an expert on the WAL of postgres, but would it make sense to try and leverage a native query within dataflow to the Postgresql WAL to discover what has changed and use that as an filter on the source data in combination with a append only data destination?