Forum Discussion
SSIS Lookup No Match Output
- 2 years ago
I'm not sure I followed everything in your description, but I am guessing you want to use the Dataflow Gen2 to insert only new rows into the Data Warehouse.
If so, I think it can be done the following way:
Bring in both your source system ("jobs") and the data warehouse table as two separate queries into your Dataflow Gen2 (i.e. Get Data).
Filter the data from the data warehouse table to only query the last 30 days (as per your description).
Next, I am assuming you have a Job ID column or another column/set of columns which uniquely identify each row.
Do an anti join between your source system query and the data warehouse query. The anti join will ensure that any rows in the source system query, which are already in the warehouse, will be filtered out of the source system query.
Then write the remaining rows to the data warehouse table. Use the warehouse table as the Data destination and choose Append method.
- 2 years ago
Good Afternoon,
Yes, I finally did it inside the Dataflow, I made it as you said,
1 - Via PowerQuery, in my case it was easier with the Advanced Editor I created a Variable Destination with the select of just the keys that are in the Destination
2 - I did a filtering doing the anti-Join, so PowerQuery just returns me the new data.
3 - On the data destination section I chosed Append, so I insert the rows.
Doing this it works, it's a bit complex than before, but it does the job, thanks for the help,
Best Regards,.
I'm not sure I followed everything in your description, but I am guessing you want to use the Dataflow Gen2 to insert only new rows into the Data Warehouse.
If so, I think it can be done the following way:
Bring in both your source system ("jobs") and the data warehouse table as two separate queries into your Dataflow Gen2 (i.e. Get Data).
Filter the data from the data warehouse table to only query the last 30 days (as per your description).
Next, I am assuming you have a Job ID column or another column/set of columns which uniquely identify each row.
Do an anti join between your source system query and the data warehouse query. The anti join will ensure that any rows in the source system query, which are already in the warehouse, will be filtered out of the source system query.
Then write the remaining rows to the data warehouse table. Use the warehouse table as the Data destination and choose Append method.
Good Afternoon,
Yes, I finally did it inside the Dataflow, I made it as you said,
1 - Via PowerQuery, in my case it was easier with the Advanced Editor I created a Variable Destination with the select of just the keys that are in the Destination
2 - I did a filtering doing the anti-Join, so PowerQuery just returns me the new data.
3 - On the data destination section I chosed Append, so I insert the rows.
Doing this it works, it's a bit complex than before, but it does the job, thanks for the help,
Best Regards,.