Forum Discussion
Get all combined subDirectories with Folder.Contents() from multiple servers (mainDirectories),
No, you need to bring out the big guns. AD extracts, Purview, Scanner API. Low code/no code tools tend to be overwhelmed when confronted with massive amounts of data.
possible value relevance: Most companies are SMEs (future value/cashflow). Most SMEs are not fully digitalized.
Many SMEs are MS-clients, still running file-servers, many run RDBs, fewer RDFs or kafka, i.e. data is stored in files.
The wish is to have a stackable select-ETL (until full digitalization, if at all) to select/access file-data via PBI (PQ/PP).
AIM: Enable any user to browse & combi-select multiple subdirectories to query file-content with separate ETLs.
Regards
You may want to continue exploring other tools. While Power Query is very powerful and flexible it is not a "proper" ETL tool. Its sole aim is to prepare data so that it can be used by Vertipaq. There was never a design goal to load anything elsewhere.
Thank you for the VertiPaq hint, indeed helpful/new.
You may want to continue exploring other tools?
It means power platform apps/page/automate/va, eventhub/dataverse...?
Coming py/R/SQL/git, MS strongly adviced no/low-Code power platform.No, you need to bring out the big guns. AD extracts, Purview, Scanner API. Low code/no code tools tend to be overwhelmed when confronted with massive amounts of data.
Thanks again. Issue closed Please note: Process is exactly designed to avoid massive data.
The select process serves the purpose to select only those few files (data) of current interest.
There is just no singel point of truth SPOT, since files flaot on file server.This fact requires iterative (therefore performant) folder search (folder browsing).
Hypothesis: Many SMEs should face a similar situation, (i.e. money on the street?)
Note on the concrete task to be accomplished that appears to be a standard BI use case
Task: Load data from distributed XLS-files (of same kind) from changing (renamed/moved) folders
via PBI (PQ) into a flattened MS-Cube to consolidate file-data for comparative pivot-analysis.Task: Load data from distributed XLS-files (of same kind) from changing (renamed/moved) folders via PBI (PQ) into a flattened MS-Cube to consolidate file-data for comparative pivot-analysis.This is from when? 2013 ?