Improve time of load from web

Why not download the list?

 

 

Hi Omid_Motamedise ,

 

If the data doesn't change often, you can implement incremental loading to get only new or updated data. But this does not seem to be the case here.

In addition to Tom's opinion, you can load multiple pages in parallel. This can significantly reduce the total loading time.

Here's the modified codes:

let
    // Define a function to load a single page
    LoadPage = (page as number) as table =>
        let
            URL = "https://companiesmarketcap.com/aud/page/" & Text.From(page) & "/",
            Source = Web.BrowserContents(URL),
            ExtractedTable = Html.Table(Source, {{"Column1", ".d-none"}, {"Column2", ".d-none + *"}, {"Column3", ".rh-sm"}, {"Column4", ".rh-sm + *"}, {"Column5", ".company-name"}, {"Column6", ".company-code"}, {"Column7", ".rank"}, {"Column8", ".td-right:nth-child(5)"}, {"Column9", ".p-0 + *"}, {"Column10", ".currency-symbol-left"}}, [RowSelector=".d-none"]),
            PromotedHeaders = Table.PromoteHeaders(ExtractedTable, [PromoteAllScalars=true]),
            // Inspect the column names
            ColumnNames = Table.ColumnNames(PromotedHeaders),
            // Adjust column names if necessary
            RenamedColumns = Table.RenameColumns(PromotedHeaders, List.Zip({ColumnNames, {"#", "Name", "1d", "Price (30 days)", "Column5", "Column6", "Column7", "Column8", "Column9", "Column10"}})),
            ChangedType = Table.TransformColumnTypes(RenamedColumns,{{"#", type text}, {"Name", type text}, {"1d", type text}, {"Price (30 days)", type text}, {"Column5", type text}, {"Column6", type text}, {"Column7", Int64.Type}, {"Column8", Currency.Type}, {"Column9", type text}, {"Column10", type text}}),
            RemovedBottomRows = Table.RemoveLastN(ChangedType, 1)
        in
            RemovedBottomRows,

    // Generate a list of page numbers
    PageNumbers = {1..10}, // Adjust the range as needed

    // Load all pages in parallel
    LoadAllPages = List.Transform(PageNumbers, each LoadPage(_)),

    // Combine all tables into one
    CombinedTable = Table.Combine(LoadAllPages)
in
    CombinedTable

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.