Dataflow gen2 ends in sucess but does not refresh the data on table with security policy

[PowerNewUser]

I have a lakehouse secured using a security policy that controls the access by the user email. When I run the dataflow gen2 to refresh the tables, it runs successfully but the data is not updated.

The dataflow is setup to replace the table. I have found even deleting the table in the lake house does not work until the policy is first removed. When I remove the policy and refresh the dataflow - it works just fine. Any suggestions on how to refresh my data while keeping the policy in place?

----SECURITY POLICY SQL USED

CREATE FUNCTION Security.tvf_supervisorEmailPredicate(@Email AS varchar(150))

    RETURNS TABLE

    WITH SCHEMABINDING

    AS

    RETURN SELECT 1 AS tvf_supervisorEmailPredicate_result

WHERE @Email = USER_NAME();

GO

CREATE SECURITY POLICY Security.Active_SupervisorFilter

    ADD FILTER PREDICATE Security.tvf_supervisorEmailPredicate(Supervisor_Email)

    ON dbo.<tablename>

    WITH (STATE = ON);

GO

[Anonymous]

Hi @PowerNewUser 

You can troubleshoot the problem by following these steps:

• Ensure that the account or service principal running Dataflow Gen2 has the necessary permissions to write to the lake tables (even when a security policy is applied)
  
• You can try creating a policy that allows broader access to see if the problem is related to security policy stringency

The following document may help you:

Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Data Factory Dataflow Gen2 limitations - Microsoft Fabric | Microsoft Learn

Best Regards,
Yulia Xu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.