<jupyter_start><jupyter_text>Predicting NBA MVP - Webscraping I want to use the machine learning to predict the MVP for the NBA.First we'll scrape the mvp, player, and team stats from the website basketball-reference.com. Each year has its own webpage, so we'll need to automate the downloading of each year's information, parse the information from each page, then concatenate the pages into one csv file. That file gets saved for later use.  Import all the things<jupyter_code>import pandas as pd

import time
import requests
import os
#import shutil

from bs4 import BeautifulSoup
from selenium import webdriver
#from selenium.webdriver.common.keys import Keys
from selenium.webdriver.edge.service import Service<jupyter_output><empty_output><jupyter_text>Get all the statistics  Get the MVP data from 1991-2021<jupyter_code># list of years
years = list(range(1991, 2021))
# url for the MVP stats follows this form - the year goes between the brackets
url_start = "https://www.basketball-reference.com/awards/awards_{}.html"
# loop through all years and save the html page locally to parse later
# you may need to create an mvp folder first!!

for year in years:
    url = url_start.format(year)
    
    data = requests.get(url)
    
    with open("mvp/{}.html".format(year), "w+", errors = 'replace') as f:
        f.write(data.text)

# test first html file (1991) to see if it parsed correctly
with open("mvp/1991.html") as f:
    page = f.read()
    
soup = BeautifulSoup(page, 'html.parser')
soup.find('tr', class_="over_header").decompose()
mvp_table = soup.find_all(id="mvp")[0]
mvp_1991 = pd.read_html(str(mvp_table))[0]
mvp_1991.head(1)
mvp_1991["Year"] = 1991
mvp_1991.head(1)
# everything parses correctly, so create a loop to process every html file in the mvp folder

dfs = []
for year in years:
    with open("mvp/{}.html".format(year)) as f:
        page = f.read()
    
    soup = BeautifulSoup(page, 'html.parser')
    soup.find('tr', class_="over_header").decompose()
    mvp_table = soup.find_all(id="mvp")[0]
    mvp_df = pd.read_html(str(mvp_table))[0]
    mvp_df["Year"] = year
    dfs.append(mvp_df)
# merge the years together

mvps = pd.concat(dfs)
mvps.tail()
# save to csv file
mvps.to_csv("mvps.csv")<jupyter_output><empty_output><jupyter_text>Get the player stats for 1991-2021<jupyter_code># all the urls follow this form - the years go inside the brackets
player_stats_url = "https://www.basketball-reference.com/leagues/NBA_{}_per_game.html"
# loop through all years and save the html page locally to parse later
# you may need to create a player folder first!!

for year in years:
    url = player_stats_url.format(year)
    
    data = requests.get(url)
    
    with open("player/{}.html".format(year), "w+", errors = 'replace') as f:
        f.write(data.text)
# set a variable for the Edge webdriver location
# the sites with the player information only gives part of the information
# so we need to automate the javascript to get the rest

service = Service(verbose = True)
driver = webdriver.Edge(service = service)
#loop through all the html files and parse them

for year in years:
    url = player_stats_url.format(year)
    
    driver.get(url)
    driver.execute_script("window.scrollTo(1,10000)")
    time.sleep(2)
    
    with open("player/{}.html".format(year), "w+", errors = 'replace') as f:
        f.write(driver.page_source)
# combine all the tables into a list of dataframes
# dfs[0] would be the stats table for 1991

dfs = []
for year in years:
    with open("player/{}.html".format(year)) as f:
        page = f.read()
    
    soup = BeautifulSoup(page, 'html.parser')
    soup.find('tr', class_="thead").decompose()
    player_table = soup.find_all(id="per_game_stats")[0]
    player_df = pd.read_html(str(player_table))[0]
    player_df["Year"] = year
    dfs.append(player_df)
# creates a dataframe from the list of dataframes
# concatenating them all together at axis = 0

players = pd.concat(dfs)
players.head()
# save file to csv
players.to_csv("players.csv")<jupyter_output><empty_output><jupyter_text>Get the team stats for 1991-2021 Conventional wisdom says that mvps will come from winning teams more than losing ones. I'll pull the team stats into the data to see if this is only a myth or if there's really something to it.<jupyter_code># the team stats follows this form - the years go inside the bracket

team_stats_url = "https://www.basketball-reference.com/leagues/NBA_{}_standings.html"
# loop through and create html files for each year

for year in years:
    url = team_stats_url.format(year)
    
    data = requests.get(url)
    
    with open("team/{}.html".format(year), "w+", errors = 'replace') as f:
        f.write(data.text)
# combine all the tables into a list of dataframes
# dfs[0] would be the stats table for 1991

dfs = []
for year in years:
    with open("team/{}.html".format(year)) as f:
        page = f.read()
    
    soup = BeautifulSoup(page, 'html.parser')
    soup.find('tr', class_="thead").decompose()
    e_table = soup.find_all(id="divs_standings_E")[0]
    e_df = pd.read_html(str(e_table))[0]
    e_df["Year"] = year
    e_df["Team"] = e_df["Eastern Conference"]
    del e_df["Eastern Conference"]
    dfs.append(e_df)
    
    w_table = soup.find_all(id="divs_standings_W")[0]
    w_df = pd.read_html(str(w_table))[0]
    w_df["Year"] = year
    w_df["Team"] = w_df["Western Conference"]
    del w_df["Western Conference"]
    dfs.append(w_df)
# creates a dataframe from the list of dataframes
# concatenating them all together at axis = 0

teams = pd.concat(dfs)
teams.tail()
# save the file to csv
teams.to_csv("teams.csv")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup