Run dataflow only if any file exists in sharepoint folder

[ShivaniTyagi01]

Hi,

Recently I ran into a scenario where in data pipeline  many dataflows are executing, but I want to run dataflow only if there is any file exists on sharepoint folder, because if any dataflow is running when no file exists on sharepoint folder it fails the execution.

thanks,

shivani

[v-pbandela-msft]

Hi @ShivaniTyagi01,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @ShivaniTyagi01,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

[v-pbandela-msft · Accepted solution]

Hi @ShivaniTyagi01,

Thank for reaching out in Microsoft Community Forum.

Based on your scenario, we're providing the solution in below points;

1. Add a Lookup Activity to Check File Existence;

Use a Lookup activity at the beginning of the pipeline.
Configure the Lookup activity to query the SharePoint folder using the Microsoft Graph API or a suitable connector.
Example query (via API or connector): List the files in the specific folder.
Use an Expression to Determine Execution
Example expression: @greater(length(activity('LookupActivityName').output.value), 0)

2. Add an If Condition activity after the Lookup activity.
Set the condition to the expression above.
In the "True" branch, include the Dataflow activity to run if files exist.
Leave the "False" branch empty.
Configure the Dataflow Activity in the True branch.

3. Place Dataflow activity in the True branch.
Ensure it processes files only when the If Condition activity passes.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.

Thank you,
Pavan.

[ShivaniTyagi01]

Hi Pavan,

Thanks for replying. How do I connect to sharepoint folder in data pipeline lookup activity , as sharepoint folder connector is not available in data pipeline but it is there when we create a data flow.

Thanks,

Shivani

[v-pbandela-msft]

Hi @ShivaniTyagi01,

Please follow the below steps to connect to SharePoint folder in data pipeline;
 

  Use HTTP Activity with Microsoft Graph API
You can use the HTTP activity in the pipeline to query the SharePoint folder using the Microsoft Graph API.

1.Register an Azure AD App:
a. Register an application in Azure Active Directory with the required permissions for accessing SharePoint.
b .Assign the Sites.Read.All permission (delegated or application).

2.Get Access Token:

Use the application/client ID, client secret, and tenant ID to fetch an OAuth token.

3.Configure HTTP Activity in Data Pipeline:

a. Add an HTTP activity in your pipeline.
Set the URL to query the SharePoint folder using Microsoft Graph API, 
e.g.: GET https://graph.microsoft.com/v1.0/sites/{site-id}/drives/{drive-id}/root:/path/to/folder:/children

4.Pass the OAuth token in the headers:

{
  "Authorization": "Bearer <token>"
}

5.Parse the Response:

a. The response will contain a list of files in the folder.
b. Use the output of the HTTP activity in next pipeline.

6.Add an If Condition activity after the HTTP activity.
Use the expression: @greater(length(activity('HTTPActivityName').output.value), 0)
If the condition comes true that means files exist, proceed to execute the Dataflow activity.

Note:
Use the following Graph API to get the Site ID of your SharePoint site: GET https://graph.microsoft.com/v1.0/sites?search=<Your SharePoint Site Name>

Once you have the Site ID, get the Drive ID for the document library: GET https://graph.microsoft.com/v1.0/sites/{site-id}/drives

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful.  help other members find it more easily.

Thank you,
Pavan.

[ShivaniTyagi01]

Hi,

Recently I ran into a scenario where in data pipeline  many dataflows are executing, but I want to run dataflow only if there is any file exists on sharepoint folder, because if any dataflow is running when no file exists on sharepoint folder it fails the execution.

thanks,

shivani