Forum Discussion

Sidhant's avatar
Sidhant
Icon for Advocate V rankAdvocate V
1 year ago

Managing and Exporting High-Volume Datasets (~1M Records) in Power BI

Hello everyone, I'm currently working on a use case where I need to manage and export high-volume datasets (~1 million records) using Power BI, and I'm exploring multiple approaches. I’d appreciate ...
Need Help
Sidhant's avatar
Sidhant
Icon for Advocate V rankAdvocate V
11 months ago

Hi folks,
So I tried to implement the next part of creating files and a folder with respect to user so once the file is created (after the do-until loop) I had few more actions and the flow looks like

In the Create new folder action (sharepoint) in the folder path I am using the previous two actions output to create the folder at the respective location: @{outputs('BaseFolderPath')}/@{outputs('UserFolderName')}
 So v-hashadapu were you referring to something like this?.

The only issue is somehow I can see in the Power Automate that the folder is created but when I checked in the Teams channel at the respective location I could see the file created (after the do-until loop) and even though the folder action was successful the next action 'Move file' could not find the folder and when I checked even I was not able to see the folder (but I had used the same path where the previously the csv file was created and it could be seen)

 

URI in Send an HTTP request to SharePoint:
_api/web/GetFolderByServerRelativeUrl(@{outputs('Create_new_folder')?['body/{FullPath}']})/ListItemAllFields/breakroleinheritance(copyroleassignments=false, clearlimitedaccessusers=true)


Any inputs what am I missing here and is this correct?.
Gabry , Poojara_D12 you can also share your inputs.

Regards,
Sidhant.

  • Sidhant's avatar
    Sidhant
    Icon for Advocate V rankAdvocate V
    11 months ago

    Hi folks,
    So a quick update so today I just observed the folder whcih was not visible under the Microsoft Teams channel when I checked at the same location on the web interface there I could see the respective folder been created
    (In Teams channel) File is created as expected

    But the folder isn't, so on web I was able to see the folder:

    So I looked out what might be the issue: Queried this issue on CoPilot which suggested to remove the invalid characters and spaces (so since there were not any invalid characters I removed the spaces) and added few more action and even some delays.
    To remove the white spacing between the words:

    Then in Create folder action used the base folder and the trimmed name

    Added few delays in between the actions and then just before move file action added get folder metadata using path: To cross-check if the folder exists but here is it failed again with same issue: status: 400 (could not find the folder)

    (Here the destinationFolderPath is a Compose action wherein I have just given the destination: @{outputs('BaseFolderPath')}/@{outputs('TrimmedUserFolderName')})
    Error: status: 404; message: Folder not found.
    I am not able to figure out why is there a discrepency even when the Create folder action is succedded.
    Any idea v-hashadapu , Gabry  and Poojara_D12  what is missing?
    Regards,
    Sidhant

    • Sidhant's avatar
      Sidhant
      Icon for Advocate V rankAdvocate V
      11 months ago

      Quick update: 
      So I had few observations (so I navigated on the web interface for the respective Sharepoint channel)
      The files were stored in: Documents/General/ExportLargeData

      Previously I had the Base Path as /Shared Documents/General/ExportLargeData

      So then I checked from the root level we have:

      (All my files are within General/ExportLargeData) and not Shared Documents
      Shared Documents also has its own General Folder and currently it creates the folder within it where I have no files

      So then I modified the Base path to: 

      So now this resolved the 1st half that the folder was created at the right path but again the flow failed at the same action which was added to get the folder metadata that it was not able to get the folder

       

      So then when create file action creates the file at the right location and when I had given the same path why did it create inside Shared Documents instead of General

      And now when the folder was created at the right path why is not able to get the folder?

      Regards,
      Sidhant.

      • Sidhant's avatar
        Sidhant
        Icon for Advocate V rankAdvocate V
        11 months ago

        So I figured out the issue: The thing is when we use Create new folder action (SharePoint) therein we specify the List or Library: Documents (which Internally corresponds to Shared Documents) so herein we just need to specify the sub-folder part so then in BaseFolderPath (Compose action) I just specified the same:

        This created the folder at the right location and then when we doing the cross -check 'Get Folder metadata using path' I used another Compose action (renamed it to ShareBasePathLocation) wherein I simply specified the complete path (along with SharedDocuments)

        So this resolved the folder not found issue, now once the file moves in the respective location we need to break the inheritance (default security) and impose new one so for that I have added the 'Send HTTP request to SharePoint' action

        uri:
_api/web/GetFolderByServerRelativeUrl('@{outputs('FolderServerRelativeUrl')}')/ListItemAllFields/breakroleinheritance(copyroleassignments=false,clearSubscopes=true

SiteServerRelativePrefix: (Compose action)
/sites/<SharePointSiteName>

FolderLibraryRelativeFullPath: (Compose action)
@{outputs('Create_new_folder')?['body/{FullPath}']}

FolderServerRelativeURL (Compose action):
Inputs: @{outputs('Create_new_folder')?['body/{FullPath}']}

        This is where the flow fails:

        "status": 400,
                "message": "{\"odata.error\":{\"code\":\"-1, Microsoft.SharePoint.Client.InvalidClientQueryException\",\"message\":{\"lang\":\"en-US\",\"value\":\"The expression \\\"web/GetFolderByServerRelativeUrl('/sites/<SiteName>/Shared Documents/General/ExportLargeData/<Foldername>')/ListItemAllFields/breakroleinheritance(copyroleassignments=false,clearSubscopes=true\\\" is not valid.\"}}}

        Any clue on this v-hashadapu , Gabry , Poojara_D12 

        Regards,
        Sidhant