Scrape Fantasy Baseball Site

[mmickle23]

Hello-

I would like to scrape the data from a table on my fantasy baseball site.  I think the list is being rendered using javascript.  Not 100% sure though.  In the past I have seen URLs that would change and I could key off of that.... 

So for example I need to scrape the page that says "All".  However, when I click on "All" the URL doesn't change.  So when have "1" selected the URL is the same as when I have "All" selected.  (See below images)

In the past I could use a changing URL that ended with "/1" or "/All"  << I guess things are more complicated now 

https://atl-01.statsplus.net/ohbl/draft/#current

When "1" is selected:

When "All" is selected:

My assumption is that I need to mimic the "All" click and then retrieve the data in the table: draftmaintable-div

Does anyone know how I can recreate this click action and then scrape this specific table?  I know this can be done using Python and Beautiful Soup and a lifetime ago I used VBA ie automation to do it.  However, I figured there is a much easier way to do this that I am unaware of..  Looking for some help if anyone has any tips?

Here is some of the source code that I think contains the JS that does what I need to mimic:

 $('a[data-toggle="tab"]').on('shown.bs.tab', function (e) {
    var target = $(e.target).attr('href');
    currentTab = target;
    getTabDynamicContent(target);
  });


                    // Then re-load from server - this is most of the get_mylist_and_options function
                    var data = { 'info': 'mylist', 'lid': 100 };
                    do_draft_ajax(data, 'json', function(msg) {
                      /*
                      * JSON with array of player objects
                      * [ { pid: x, pos: '1B', first: 'First', last: 'Last'}, ... ]
                      *
                      * var mySettings = { 'loaded': false, 'listpick': false, 'autopick': false, 'list': [] };
                      *  list is just pid's once loaded
                      */

                      // Set the local copy of all the settings for highlighting Save button
                      $('#sortable-list').empty();

                      mySettings.list = [];

                      // Populate the draft list as if each player had been clicked
                      msg.list.forEach( function(p) {
                        add_player_to_list(p.pid, p.first, p.last, p.pos);
                        mySettings.list.push(p.pid);
                      });

                      listLoaded = true;

                      update_list_size();

                      // Reset the save / undo button state to disabled
                      $('#save-button').addClass('btn-default disabled').removeClass('btn-warning');
                      $('#undo-button').addClass('disabled');
                    });
                  }
              },
              cancel: function(){ }
          }
      });  




// If no active tab, set to the #current tab
  var h = location.hash;
  if (h == '' || h == '#_=_') {
    h = '#current';
    $('a[href="' + h + '"]').tab('show');
    if(history.replaceState) {
      history.replaceState(null, null, '#current');
    }
  } else {
    currentTab = h;
    getTabDynamicContent(h);    
  }

  jconfirm.defaults = { theme: alertTheme };
  });
</script>

[mmickle23]

Hey, Thank you so much for the assistance.  I will look into the API.  This gets me 99% of the way there already.  You're awesome!

[v-saisrao-msft]

Hi @mmickle23 

Thanks for reaching out to Microsoft forum community.

Could you please confirm if your query have been resolved? If they have, kindly mark the helpful response and accept it as the solution. This will assist other community members in resolving similar issues more efficiently.

Thank you.

[jgeddes · Accepted solution]

You might be able to get what you want with this code...

let
    url = "https://atl-01.statsplus.net/ohbl/draftajax/",
    formData = "{""info"":""round"",""lid"":""100"", ""round"":""All""}",
    formDataJson = Text.ToBinary(Uri.BuildQueryString(Json.Document(formData))),
    response = Web.Contents(url, [Content=formDataJson, Headers=[#"Content-Type"="application/x-www-form-urlencoded"]]),
    htmlTable = Html.Table(response, {{"Round", "TD:nth-child(1)"}, {"Pick", "TD:nth-child(2)"}, {"Overall", "TD:nth-child(3)"}, {"Team", "TD:nth-child(4)"}, {"Selection", "TD:nth-child(5)"}, {"Column6", ".active .player-name-modal"}, {"Pick Time", "TD:nth-child(6)"}, {"Column8", "SUP"}}, [RowSelector="TD:nth-child(1)"])

in
    htmlTable

Hope this points you in the right direction.

[mmickle23]

Hey jgeddes-

This is exactly what I was looking to do.  I very much appreciate you providing the query to complete the task.  I was curious of one more thing.  In the background I can see in the website source code that it retreives an id (unique identifier) for each player.  Is there anyway for me to just look at the raw JSON?

I see this step looks to pull in a CSV.  I am somewhat familiar with JSON and could probably parse it. Just curious as this is the one part of data that I'm missing that I don't see rendering on the webpage but I know is there....

In the Page Source Code:

Also, any assistance on the general idea you are using to get the data would be great.  It's great that you were able to help, but would like to try to understand what you did at a fundamental level.  So I could potentially use it at some point in the future.

Again Thank you already for what you have provided!! This is wonderful!

[jgeddes]

I found the api url and form data needed by inspecting the site you provided.

From there I constructed the call with the form data and then formatted the response into a html table.

If you are looking for raw json data my suggestion would be to get the api documentation for the site and adjust the call accordingly.