Forum Discussion

mmickle23's avatar
mmickle23
Regular Visitor
1 year ago
Solved

Scrape Fantasy Baseball Site

Hello-

 

I would like to scrape the data from a table on my fantasy baseball site.  I think the list is being rendered using javascript.  Not 100% sure though.  In the past I have seen URLs that would change and I could key off of that.... 

 

So for example I need to scrape the page that says "All".  However, when I click on "All" the URL doesn't change.  So when have "1" selected the URL is the same as when I have "All" selected.  (See below images)

 

In the past I could use a changing URL that ended with "/1" or "/All"  << I guess things are more complicated now 

 

https://atl-01.statsplus.net/ohbl/draft/#current

 

When "1" is selected:

 

 

When "All" is selected:

 

 

My assumption is that I need to mimic the "All" click and then retrieve the data in the table: draftmaintable-div

 

 

 

Does anyone know how I can recreate this click action and then scrape this specific table?  I know this can be done using Python and Beautiful Soup and a lifetime ago I used VBA ie automation to do it.  However, I figured there is a much easier way to do this that I am unaware of..  Looking for some help if anyone has any tips?

 

 

Here is some of the source code that I think contains the JS that does what I need to mimic:

 

 

 $('a[data-toggle="tab"]').on('shown.bs.tab', function (e) {
    var target = $(e.target).attr('href');
    currentTab = target;
    getTabDynamicContent(target);
  });


                    // Then re-load from server - this is most of the get_mylist_and_options function
                    var data = { 'info': 'mylist', 'lid': 100 };
                    do_draft_ajax(data, 'json', function(msg) {
                      /*
                      * JSON with array of player objects
                      * [ { pid: x, pos: '1B', first: 'First', last: 'Last'}, ... ]
                      *
                      * var mySettings = { 'loaded': false, 'listpick': false, 'autopick': false, 'list': [] };
                      *  list is just pid's once loaded
                      */

                      // Set the local copy of all the settings for highlighting Save button
                      $('#sortable-list').empty();

                      mySettings.list = [];

                      // Populate the draft list as if each player had been clicked
                      msg.list.forEach( function(p) {
                        add_player_to_list(p.pid, p.first, p.last, p.pos);
                        mySettings.list.push(p.pid);
                      });

                      listLoaded = true;

                      update_list_size();

                      // Reset the save / undo button state to disabled
                      $('#save-button').addClass('btn-default disabled').removeClass('btn-warning');
                      $('#undo-button').addClass('disabled');
                    });
                  }
              },
              cancel: function(){ }
          }
      });  




// If no active tab, set to the #current tab
  var h = location.hash;
  if (h == '' || h == '#_=_') {
    h = '#current';
    $('a[href="' + h + '"]').tab('show');
    if(history.replaceState) {
      history.replaceState(null, null, '#current');
    }
  } else {
    currentTab = h;
    getTabDynamicContent(h);    
  }

  jconfirm.defaults = { theme: alertTheme };
  });
</script>

 

 

 

  • You might be able to get what you want with this code...

    let
        url = "https://atl-01.statsplus.net/ohbl/draftajax/",
        formData = "{""info"":""round"",""lid"":""100"", ""round"":""All""}",
        formDataJson = Text.ToBinary(Uri.BuildQueryString(Json.Document(formData))),
        response = Web.Contents(url, [Content=formDataJson, Headers=[#"Content-Type"="application/x-www-form-urlencoded"]]),
        htmlTable = Html.Table(response, {{"Round", "TD:nth-child(1)"}, {"Pick", "TD:nth-child(2)"}, {"Overall", "TD:nth-child(3)"}, {"Team", "TD:nth-child(4)"}, {"Selection", "TD:nth-child(5)"}, {"Column6", ".active .player-name-modal"}, {"Pick Time", "TD:nth-child(6)"}, {"Column8", "SUP"}}, [RowSelector="TD:nth-child(1)"])
    
    in
        htmlTable

    Hope this points you in the right direction.

5 Replies

  • You might be able to get what you want with this code...

    let
        url = "https://atl-01.statsplus.net/ohbl/draftajax/",
        formData = "{""info"":""round"",""lid"":""100"", ""round"":""All""}",
        formDataJson = Text.ToBinary(Uri.BuildQueryString(Json.Document(formData))),
        response = Web.Contents(url, [Content=formDataJson, Headers=[#"Content-Type"="application/x-www-form-urlencoded"]]),
        htmlTable = Html.Table(response, {{"Round", "TD:nth-child(1)"}, {"Pick", "TD:nth-child(2)"}, {"Overall", "TD:nth-child(3)"}, {"Team", "TD:nth-child(4)"}, {"Selection", "TD:nth-child(5)"}, {"Column6", ".active .player-name-modal"}, {"Pick Time", "TD:nth-child(6)"}, {"Column8", "SUP"}}, [RowSelector="TD:nth-child(1)"])
    
    in
        htmlTable

    Hope this points you in the right direction.

    • mmickle23's avatar
      mmickle23
      Regular Visitor

      Hey jgeddes-

       

      This is exactly what I was looking to do.  I very much appreciate you providing the query to complete the task.  I was curious of one more thing.  In the background I can see in the website source code that it retreives an id (unique identifier) for each player.  Is there anyway for me to just look at the raw JSON?

       

      I see this step looks to pull in a CSV.  I am somewhat familiar with JSON and could probably parse it. Just curious as this is the one part of data that I'm missing that I don't see rendering on the webpage but I know is there....

       

       

      In the Page Source Code:

       

       

      Also, any assistance on the general idea you are using to get the data would be great.  It's great that you were able to help, but would like to try to understand what you did at a fundamental level.  So I could potentially use it at some point in the future.

       

      Again Thank you already for what you have provided!! This is wonderful!

       

      • jgeddes's avatar
        jgeddes
        Super User

        I found the api url and form data needed by inspecting the site you provided.

        From there I constructed the call with the form data and then formatted the response into a html table.

        If you are looking for raw json data my suggestion would be to get the api documentation for the site and adjust the call accordingly. 

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Community Support

    Hi mmickle23 

    Thanks for reaching out to Microsoft forum community.

     

    Could you please confirm if your query have been resolved? If they have, kindly mark the helpful response and accept it as the solution. This will assist other community members in resolving similar issues more efficiently.

    Thank you.

  • mmickle23's avatar
    mmickle23
    Regular Visitor

    Hey, Thank you so much for the assistance.  I will look into the API.  This gets me 99% of the way there already.  You're awesome!