incremental loading
1 Topic
MS Fabric Incremental loading
I need help to understand the best practices for Incremental loading with AZURE and loading into Data Factory. I beleive there is two methods behind my use case but want to understand the route I should take and explore my options Use case: I'm in a smaller organization ( Around 400 people) and we want to build a central location for a reporting platform solution. We use Salesforce and QAD as our ERP. QAD is a little old but we are still able to connect to azure through ODBC. How and what prinicples should I be following for loading data from our ERP enviornment on a incremental cadence? Based off my understadning and knowledge, Should I (1) Use the copy data pipeline and use fucntions/ activities to update the table? how would I scheudle the job? How would this take into account for a heavy volume table? or (2) use notebooks throughout this process and set a datetime column as a water mark value to help with updates? How would I go around this method? I would apprecaite any information on how to handle this and best practices.... I do not have a SQL server available which I know which would help for the control table feature. Any suggestions, thanksSolved6.6KViews0likes1Comment