Forum Discussion

snowrider1799's avatar
snowrider1799
Helper I
6 years ago

Multiple calls to the same dataflow

Hi,   I have multiple datasets in a dataflow calling the same web API: e.g. https://www.abc.com/page=1, from page 1 to page 19 (and growing). Each page consists of about 100k rows of data.   Eac...
Dataflow
lbendlin's avatar
lbendlin
Super User
6 years ago

Does it have to be Power BI?  Could you pull that web contents with curl or Powershell , place it into a single CSV file ( I can show you a simple Powershell script for that) and then feed that one file to your dataflow?

snowrider1799's avatar
snowrider1799
Helper I
6 years ago

Hi lbendlin ,

 

I have 19 pages, and total about 2mil data, it will continue to grow in times.

 

For dataflow #1, i am supposed to combine 19 pages with 5 columns (for example).

 

Can this be done via the powershell? If yes, i will be interested to know. 

 

 

  • lbendlin's avatar
    lbendlin
    Super User
    6 years ago

    Here is a simple PowerShell script that concatenates any number of CSV files, stripping out the header row for all files except the first one.

     

    $Path = "C:\Users\xxx\Documents"
$Target = "your output file.csv"
$g = get-childItem "$Path\*.csv" 
$First = $true
foreach($f in $g) {
    if($First) {
        Get-Content "$Path\$f" | Add-Content $Target 
        $First = $false
    } else {
       Get-Content "$Path\$f" | Select -Skip 1 | Add-Content $Target 
    }
}