Hi everyone,
I'm currently working on a project in Azure Synapse where I'm using the SAP CDC Connector to connect to an S4Hana system. My goal is to filter data on the source side before storing it in my ADLS Gen2, as there are certain data restrictions that I need to adhere to.
I need to fetch multiple objects from SAP, and I typically use a parameterized approach for this. I have a JSON file that contains parameters and queries for each object I want to retrieve from the source. For instance, I define SQL queries in the JSON file to perform the filtering. This method works well with SQL Connectors.
However, with the SAP CDC Connector, I haven’t been able to find any functionality that allows me to apply such filtering directly at the source.
Here’s what I’m doing so far:
I’m currently using a dataflow in a for each loop. In the dataflow however, I cannot pass SQL queries and Im stuck with the expression builder. I cannot figure out how to dynamically pass query like filtering. So Im just getting the unfiltered objects, which is not an option. I have so many objects, that I cant maintain a non parameterized version.
I tried using a copy data activity as well, however when selecting it, I do not get the option to choose the SAP CDC Integration Dataset.
Has anyone successfully managed to filter tables at the source when using the SAP CDC linked service? Any insights or suggestions on how to achieve this would be greatly appreciated.
Thanks in advance for your help!
