How to get results of Salesforce upsert using Data Copy?

[markmal]

Hi,

I need to push data from DB into Salesforce. I can do it using Data Copy.
However it does not return row level results with possible errors for each row.

I know about logging, and it is probably possible to get errors from there.

However the format of logs is CSV and SF Id is not in separate column there. 

May be I should use some another component ?
Please advise.

thanks,
Mark

[v-sathmakuri]

Hi @markmal ,

I hope this information provided is helpful. Feel free to reach out if you have any further questions or would like to discuss this in more detail. If responses provided answers your question, please accept it as a solution so other community members with similar problems can find a solution faster.

Thank you!!

[v-sathmakuri]

Hi @markmal ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the responses has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you!!

[v-sathmakuri]

Hi @markmal ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-sathmakuri · Accepted solution]

Hi @markmal ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @nilendraFabric  for the prompt response.

One potential solution is to implement custom upsert logic either through Spark notebooks. This allows for more granular error handling and logging, and it provides a more robust mechanism to capture errors during the upsert process. You can implement row-level tracking and errors directly within these custom scripts, giving you the flexibility to log failures, retry records, and maintain detailed error reporting.

As a workaround, If you are restricted to using the Data Copy activity, a workaround would be to perform an additional step after the copy process to query Salesforce for recently modified records. By cross referencing records with Salesforce, you can identify which records were successfully upserted and which ones failed. While this approach doesn't provide row-level error reporting directly from the copy activity, it can still help you track the success or failure of upserts and perform necessary remediation steps.

Both options require some custom implementation, but they should provide more control and visibility into the process until native row-level error reporting is available in Data Copy.

If you feel row level error reporting should be available for data copy activity, I recommend submitting your detailed feedback and ideas through Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.  

Thank you!!

[nilendraFabric]

Hi @markmal 

There is currently no out-of-the-box, row-level error reporting for Salesforce upserts using Data Copy in Microsoft Fabric.

One idea could be

Implement additional activities in your pipeline to capture and process error outputs. For example, after a copy activity, you could query Salesforce for recently modified records to cross-reference which records were successfully upserted.

If your architecture allows, you could use Spark notebooks (for Lakehouse) or T-SQL (for Warehouse) to implement custom upsert logic, which can provide more granular error handling and logging

[markmal]

Hi Nirendra,

unfortunately this method does not show errors happen in previous step.

It is weird that taking the result with by row statuses and making it accessible as an activity output was not implemented from very beginning. 

I do not believe this feature was not requested.