Forum Discussion

Sidhant's avatar
Sidhant
Icon for Advocate V rankAdvocate V
1 year ago

Managing and Exporting High-Volume Datasets (~1M Records) in Power BI

Hello everyone, I'm currently working on a use case where I need to manage and export high-volume datasets (~1 million records) using Power BI, and I'm exploring multiple approaches. I’d appreciate ...
Need Help
Sidhant's avatar
Sidhant
Icon for Advocate V rankAdvocate V
11 months ago

Hi folks,
So a quick update so today I just observed the folder whcih was not visible under the Microsoft Teams channel when I checked at the same location on the web interface there I could see the respective folder been created
(In Teams channel) File is created as expected

But the folder isn't, so on web I was able to see the folder:

So I looked out what might be the issue: Queried this issue on CoPilot which suggested to remove the invalid characters and spaces (so since there were not any invalid characters I removed the spaces) and added few more action and even some delays.
To remove the white spacing between the words:

Then in Create folder action used the base folder and the trimmed name

Added few delays in between the actions and then just before move file action added get folder metadata using path: To cross-check if the folder exists but here is it failed again with same issue: status: 400 (could not find the folder)

(Here the destinationFolderPath is a Compose action wherein I have just given the destination: @{outputs('BaseFolderPath')}/@{outputs('TrimmedUserFolderName')})
Error: status: 404; message: Folder not found.
I am not able to figure out why is there a discrepency even when the Create folder action is succedded.
Any idea v-hashadapu , Gabry  and Poojara_D12  what is missing?
Regards,
Sidhant

Sidhant's avatar
Sidhant
Icon for Advocate V rankAdvocate V
11 months ago

Quick update: 
So I had few observations (so I navigated on the web interface for the respective Sharepoint channel)
The files were stored in: Documents/General/ExportLargeData

Previously I had the Base Path as /Shared Documents/General/ExportLargeData

So then I checked from the root level we have:

(All my files are within General/ExportLargeData) and not Shared Documents
Shared Documents also has its own General Folder and currently it creates the folder within it where I have no files

So then I modified the Base path to: 

So now this resolved the 1st half that the folder was created at the right path but again the flow failed at the same action which was added to get the folder metadata that it was not able to get the folder

 

So then when create file action creates the file at the right location and when I had given the same path why did it create inside Shared Documents instead of General

And now when the folder was created at the right path why is not able to get the folder?

Regards,
Sidhant.

  • Sidhant's avatar
    Sidhant
    Icon for Advocate V rankAdvocate V
    11 months ago

    So I figured out the issue: The thing is when we use Create new folder action (SharePoint) therein we specify the List or Library: Documents (which Internally corresponds to Shared Documents) so herein we just need to specify the sub-folder part so then in BaseFolderPath (Compose action) I just specified the same:

    This created the folder at the right location and then when we doing the cross -check 'Get Folder metadata using path' I used another Compose action (renamed it to ShareBasePathLocation) wherein I simply specified the complete path (along with SharedDocuments)

    So this resolved the folder not found issue, now once the file moves in the respective location we need to break the inheritance (default security) and impose new one so for that I have added the 'Send HTTP request to SharePoint' action

    uri:
_api/web/GetFolderByServerRelativeUrl('@{outputs('FolderServerRelativeUrl')}')/ListItemAllFields/breakroleinheritance(copyroleassignments=false,clearSubscopes=true

SiteServerRelativePrefix: (Compose action)
/sites/<SharePointSiteName>

FolderLibraryRelativeFullPath: (Compose action)
@{outputs('Create_new_folder')?['body/{FullPath}']}

FolderServerRelativeURL (Compose action):
Inputs: @{outputs('Create_new_folder')?['body/{FullPath}']}

    This is where the flow fails:

    "status": 400,
            "message": "{\"odata.error\":{\"code\":\"-1, Microsoft.SharePoint.Client.InvalidClientQueryException\",\"message\":{\"lang\":\"en-US\",\"value\":\"The expression \\\"web/GetFolderByServerRelativeUrl('/sites/<SiteName>/Shared Documents/General/ExportLargeData/<Foldername>')/ListItemAllFields/breakroleinheritance(copyroleassignments=false,clearSubscopes=true\\\" is not valid.\"}}}

    Any clue on this v-hashadapu , Gabry , Poojara_D12 

    Regards,
    Sidhant