<jupyter_start><jupyter_text>Data Analysis<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Size of the dataset is quite small, and neural networks tend to overfit on small datasets. While a small MLP would perhaps fare ok, I prefer using XGBoost due to the tabular nature of the data.  Idea 1: Create an extra column from the car's manufacturer We are going to assume that the word of the `car name` column is the manufacturer. This will give us another datapoint. While the physical characteristics like weight and horsepower should be much more indicative of the value of the target variable, one could assume that each manufacturer has their own proprietary technology that might reduce the mpg or something along those lines. Under the mantra of **_more data cannot hurt_**, we will perform an with and without these added feature.<jupyter_code>df_mfct = df.copy()
df_mfct['mfct'] = df_mfct['car name'].transform(lambda x: x.split(' ')[0])
df_mfct
Counter(df_mfct.mfct), f'NUMBER OF MANUFACTURERS: {len(set(Counter(df_mfct.mfct)))}'<jupyter_output><empty_output><jupyter_text>Our assumption seems to hold pretty well! Although there is some minor data noise: `volkswagen` and `vw` likely refer to the same manufacturer. Same goes with `mercedes-benz` and `mercedes` or `maxda` and `mazda` (cute typo!) or `toyota` and `toyouta`. Let's quickly solve this issue.<jupyter_code>replacement_dict = {
    'vw': 'volkswagen',
    'vokswagen': 'volkswagen',
    'mercedes': 'mercedes-benz',
    'maxda': 'mazda',
    'toyouta': 'toyota',
    'chevroelt': 'chevrolet',
    'chevy': 'chevrolet',
    'capri': 'ford'
}
df_mfct['mfct'] = df_mfct['mfct'].transform(lambda x: replacement_dict[x] if x in replacement_dict else x)
Counter(df_mfct.mfct)<jupyter_output><empty_output><jupyter_text>I will quickly drop exactly 1 row for which I cannot find any extra information out in the wild. It is for the greater good and won't hurt us too much.<jupyter_code>df_mfct = df_mfct[df_mfct['mfct'] != 'hi']<jupyter_output><empty_output><jupyter_text>Idea 2: Use the geographical position of the producer as another data point I will admit, this might inch towards overthinking. But different manufacturers design cars for different consumer realities e.g. USA based producers might not have fuel efficiency i.e. mpg in mind when building a car since gasoline is quite cheap in the country. Let us introduce a new categorical feature for where the manufacturer is located. On the other hand, big manufacturers tend to ship worlwide + the dataset seems to be compiled on USA-based cars from Kaggle's dataset metadata. This feature might be thoroughly useless, but investigating is worth it. We will look the data on the internet for this task. Nota bene: The origin feature might be doing this already. It is categorically encoded, with a domain of [1, 2, 3]. These might be the USA, Europe, Asia i.e. continents of origin, but the metadata says nothign about this. We will proceed with both, although some duplication might be possible.<jupyter_code>geo_dict = {
    'chevrolet': 'USA',
    'buick': 'USA',
    'plymouth': 'USA',
    'amc': 'USA',
    'ford': 'USA',
    'pontiac': 'USA',
    'dodge': 'USA',
    'toyota': 'Japan',
    'datsun': 'Japan',
    'volkswagen': 'Germany',
    'peugeot': 'France',
    'audi': 'Germany',
    'saab': 'Sweden',
    'bmw': 'Germany',
    'mercury': 'USA',
    'opel': 'Germany',
    'fiat': 'Italy',
    'oldsmobile': 'USA',
    'chrysler': 'USA',
    'mazda': 'Japan',
    'volvo': 'Sweden',
    'renault': 'France',
    'honda': 'Japan',
    'subaru': 'Japan',
    'mercedes-benz': 'Germany',
    'cadillac': 'USA',
    'triumph': 'UK',
    'nissan': 'Japan'
}<jupyter_output><empty_output><jupyter_text>Pretty nice! Most companies are clustered around `USA`, `Japan` and `Germany`, which raises my hopes<jupyter_code>df_mfct_geo = df_mfct.copy()
df_mfct_geo['geo'] = df_mfct_geo['mfct'].apply(lambda x: geo_dict[x].lower())
df_mfct_geo<jupyter_output><empty_output><jupyter_text>Idea 3: Change model year XX to XXXX representation It seems the model year implicitly assumes the 20th century, and thus values are 19XX. Again, this is paranoid me, but I'd prefer to use the full representation, in case someone tries to use this model in the 21st century, thus making predictions more robust. Also, I find the use of whitespace in column names deeply offensive.<jupyter_code>df_mfct_geo_year = df_mfct_geo.copy()
df_mfct_geo_year['model_year'] = df_mfct_geo_year['model year'].apply(lambda x: int(f"19{x}"))
del df_mfct_geo_year['model year']
df_mfct_geo_year<jupyter_output><empty_output><jupyter_text>Preparing the data  Dealing with horsepower missing values 6 rows have a missing rows value. The rows are marked with '?', turning the whole column into a string column. We replace '?' with NaN, turn the column into floats, and use a linear interpolation provided by pandas. **Update: interpolating increases MAE, just drop the rows**<jupyter_code>df_mfct_geo_year_hp = df_mfct_geo_year.copy()
# df_mfct_geo_year_hp['horsepower'] = df_mfct_geo_year_hp['horsepower'].apply(lambda x: int(x) if x != '?' else np.nan)
# df_mfct_geo_year_hp['horsepower'] = df_mfct_geo_year_hp['horsepower'].interpolate()<jupyter_output><empty_output><jupyter_text>Hot encoding the `geo` feature and the `mfct` feature<jupyter_code>mfct_ohe = pd.get_dummies(df_mfct_geo_year['mfct'], dummy_na=True)
mfct_ohe = mfct_ohe.rename(columns={np.nan: 'mfct_nan'})
mfct_ohe
geo_ohe = pd.get_dummies(df_mfct_geo_year['geo'], dummy_na=True)
geo_ohe = geo_ohe.rename(columns={np.nan: 'geo_nan'})
geo_ohe<jupyter_output><empty_output><jupyter_text>Putting it all together Let's append the OHE columns together and drop the ones that they are replacing<jupyter_code>df_concat = pd.concat([df_mfct_geo_year_hp, mfct_ohe, geo_ohe], axis=1)
df_concat
df_final = df_concat.drop(columns=['car name', 'mfct', 'geo'])
# for col in ['cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model_year']:
#     scaler = StandardScaler()
#     df_final[col] = scaler.fit_transform(df_final[col].values.reshape(-1, 1))
df_final
y, X = df_final[['mpg']].to_numpy().reshape(-1, ), df_final.drop(columns=['mpg']).to_numpy()
y.shape, X.shape<jupyter_output><empty_output><jupyter_text>Model evaluation We will use cross validation and the MEA to evaluate the model. We will test the effect of the hypotheses mentioned above regarding features. We should be careful about setting all random states in cross_validation and models to fixed values in order to get reproducible results.  First hypothesis - enhanced columns<jupyter_code>model = XGBRegressor(seed=42, random_state=42, colsample_bytree=1)
cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=42)
scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)
scores = np.absolute(scores)
print('Mean MAE %.3f STD MAE %.3f' % (scores.mean(), scores.std()) )<jupyter_output>/Users/bratu/Desktop/dsp/venv/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/Users/bratu/Desktop/dsp/venv/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/Users/bratu/Desktop/dsp/venv/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/Users/bratu/Desktop/dsp/venv/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a futur[...]<jupyter_text>Second hypothesis - plain columns By plain columns we understand columns that do not carry any information about manufacturer or geolocation.<jupyter_code>list(geo_ohe.columns)
df_final_second_h = df_final.drop(columns=list(geo_ohe.columns) + list(mfct_ohe.columns))
df_final_second_h
y_sec_hyp, X_sec_hyp = df_final_second_h[['mpg']].to_numpy().reshape(-1, ), df_final_second_h.drop(columns=['mpg']).to_numpy()
y_sec_hyp.shape, X_sec_hyp.shape
model_sec_hyp = XGBRegressor(seed=42, random_state=42, colsample_bytree=1)
cv_sec_hyp = RepeatedKFold(n_splits=10, n_repeats=3, random_state=42)
scores_sec_hyp = cross_val_score(model_sec_hyp, X_sec_hyp, y_sec_hyp, scoring='neg_mean_absolute_error', cv=cv_sec_hyp, n_jobs=-1)
scores_sec_hyp = np.absolute(scores_sec_hyp)
print('Mean MAE %.3f STD MAE %.3f' % (scores_sec_hyp.mean(), scores_sec_hyp.std()) )<jupyter_output>Mean MAE 2.049 STD MAE 0.298<jupyter_text>Third hypothesis - plain columns minus origins  Quick test to determine wether the origin column is relevant. It might encode all the geoencode information I was speculating above.<jupyter_code>df_final_third_h = df_final.drop(columns=list(geo_ohe.columns) + list(mfct_ohe.columns) + ['origin'])
df_final_third_h
y_third_hyp, X_third_hyp = df_final_third_h[['mpg']].to_numpy().reshape(-1, ), df_final_third_h.drop(columns=['mpg']).to_numpy()
y_third_hyp.shape, X_third_hyp.shape
model_third_hyp = XGBRegressor(seed=42, random_state=42, colsample_bytree=1)
cv_third_hyp = RepeatedKFold(n_splits=10, n_repeats=3, random_state=42)
scores_third_hyp = cross_val_score(model_third_hyp, X_sec_hyp, y_sec_hyp, scoring='neg_mean_absolute_error', cv=cv_third_hyp, n_jobs=-1)
scores_third_hyp = np.absolute(scores_third_hyp)
print('Mean MAE %.3f STD MAE %.3f' % (scores_third_hyp.mean(), scores_third_hyp.std()) )<jupyter_output>Mean MAE 2.049 STD MAE 0.298<jupyter_text>Fourth hypothesis: Only the `manufacturer` enhanced column matters<jupyter_code>df_final_fourth_h = df_final.drop(columns=list(geo_ohe.columns))
df_final_fourth_h
y_fourth_h, X_fourth_h = df_final_fourth_h[['mpg']].to_numpy().reshape(-1, ), df_final_fourth_h.drop(columns=['mpg']).to_numpy()
y_fourth_h.shape, X_fourth_h.shape
model_fourth_hyp = XGBRegressor(seed=42, random_state=42, colsample_bytree=1)
cv_fourth_hyp = RepeatedKFold(n_splits=10, n_repeats=3, random_state=42)
scores_fourth_hyp = cross_val_score(model_fourth_hyp, X_fourth_h, y_fourth_h, scoring='neg_mean_absolute_error', cv=cv_fourth_hyp, n_jobs=-1)
scores_fourth_hyp = np.absolute(scores_fourth_hyp)
print('Mean MAE %.3f STD MAE %.3f' % (scores_fourth_hyp.mean(), scores_fourth_hyp.std()) )<jupyter_output>Mean MAE 2.013 STD MAE 0.314<jupyter_text>Fifth hypothesis: Only the `geo` enhanced column matters<jupyter_code>df_final_fifth_h = df_final.drop(columns=list(list(mfct_ohe.columns)))
df_final_fifth_h
y_fifth_h, X_fifth_h = df_final_fifth_h[['mpg']].to_numpy().reshape(-1, ), df_final_fifth_h.drop(columns=['mpg']).to_numpy()
y_fifth_h.shape, X_fifth_h.shape
model_fifth_hyp = XGBRegressor(seed=42, random_state=42, colsample_bytree=1)
cv_fifth_hyp = RepeatedKFold(n_splits=10, n_repeats=3, random_state=42)
scores_fifth_hyp = cross_val_score(model_fifth_hyp, X_fifth_h, y_fifth_h, scoring='neg_mean_absolute_error', cv=cv_fifth_hyp, n_jobs=-1)
scores_fifth_hyp = np.absolute(scores_fifth_hyp)
print('Mean MAE %.3f STD MAE %.3f' % (scores_fifth_hyp.mean(), scores_fifth_hyp.std()) )<jupyter_output>Mean MAE 2.021 STD MAE 0.303<jupyter_text>Conclusion We conclude that our enhanced columns indeed improve the performance of the model (hypotheses 1 + 4). Adding only the manufacturer feature fares a better (4) than adding both, although scores are quite close, and 1 has lower variation. These things considered, we will move with the engineered manufacturer feature into training, where we will leverage Bayesian search for hyperparameter optimization.<jupyter_code># with open('best.csv', 'w+') as fp:
#     df_final_fourth_h.to_csv(fp)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup