<jupyter_start><jupyter_text>INFO 3350/6350 Lecture 11: Classification I To do* HW 5 due **Friday**, 11:59pm    * Max 3 slip days for this one    * Note that slip day limits for each assignment are hard stops. Slip days *are* your flexibility for extceptional circumstances. No additional flexibility is possible.* This week    * Friday sections *do* meet    * Read HDA, ch. 6 before next Monday    * No discussion readings or response posts* Next week (classification, cont.)    * Monday: Read HDA, ch. 8 and "The curse(s) of dimensionality"    * Wednesday: Read articles by Underwood and by Norvig        * Response to Underwood or Norvig due by 4:00pm by next Tuesday, 3/8, if assigned to you by NetID.    * Friday: Section as usual Repeat: Supervised and unsupervised learning* "Clustering" is a type of *unsupervised* learning* "Classification" is a type of *supervised* learning* **Both seek to assign a finite set of labels to input objects on the basis of features associated with those objects*** In *supervised* learning, you know the labels in advance    * You supply a set of (correctly) labeled objects to the algorithm    * The algorithm then "learns" which features are associated with which labels, so as to minimize classification errors* In *unsupervised* learning, you don't know the labels (though you might know how many labels there should be)    * You supply a feature matrix and a definition of what a "good" clustering (label set) should be    * The algorithm then assigns labels to the input objects so as to best satisfy the supplied definition of "good" (typically, minimizing variance within clusters and maximizing difference between clusters) Repeat: Why prefer one or the other?* **Unsupervised** methods are often used early in a project, when you're looking for unknown stucture in your data    * Unsupervised methods are your only option if you don't know what the appropriate set of labels might be for your data set    * Also appropriate if you don't have (many) labeled instances    * Also appropriate when you know and care a lot about the features themselves rather than the classes        * Clustering produces a set of labels that are most compatible with the features (as well as the distance metric and learning algorithm, of course) that you've supplied.    * Unsupervised methods are typically "cheap" to set up, but costly to evaluate* **Supervised** methods require you to know in advance the full set of appropriate labels for your data    * Supervised methods often have high initial costs, but are easier to evaluate (because you already have a set of correctly labeled instances that you can use for validation).    * Appropriate when you care more about the output labels than the features.        * Weighs features such that you produce the set of labels closest to the training data.        * No guarantee that the features on which the supervised method relies will be the ones that you believe are abstracty important.        That said, it's possible to examine (directly) or to evaluate (indirectly) the importance of specific features in a trained supervised model. This can be interesting if you want to understand the features on which the model relies most heavily. Machine learning glossariesIf you're finding yourself confused by some of the volcabulary of machine learning, here are two useful glossaries that may help:* [Google's machine learning glossary](https://developers.google.com/machine-learning/glossary). Very extensive, leans toward neural methods.* [A shorter glossary](https://ml-cheatsheet.readthedocs.io/en/latest/glossary.html) for quicker reference.  Algorithms *k* Nearest Neighbors* Assign to new objects the most common label among *k* nearest neighbors in the training set* Very low computational cost, easy to understand and interpret, and can fit complex decision spaces with ease* Not usually the best performance for real-world cases An artificial exampleLet's try this with 3 synthetic blobs and then switch to 4 ...**NB.** We use the `make_blobs` function to create synthetic data for purposes of illustration. This has no role in actual clustering on classification problems. Don't try to use `make_blobs` as part of your approach to learning from real data.<jupyter_code># kMeans and kNN on articial data
# Adapted from sklearn examples

import numpy as np
import matplotlib.pyplot as plt

from sklearn.cluster import KMeans
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_blobs

plt.figure(figsize=(12, 12))

n_samples = 1500
n_blobs=4
X, y = make_blobs(
    n_samples=n_samples, 
    centers=n_blobs
)

# Input data
plt.subplot(231)
plt.scatter(X[:, 0], X[:, 1], c=y, alpha=0.5)
plt.title("Input data")


# Clustering
y_pred = KMeans(n_clusters=3).fit_predict(X)

plt.subplot(232)
plt.scatter(X[:, 0], X[:, 1], c=y_pred, alpha=0.5)
plt.title("kMeans")


# Classification
y_pred = KNeighborsClassifier(n_neighbors=5).fit(X, y).predict(X)
print(f"kNN score (uniform density): {KNeighborsClassifier(n_neighbors=5).fit(X, y).score(X,y):.3f}")

plt.subplot(233)
plt.scatter(X[:, 0], X[:, 1], c=y_pred, alpha=0.5)
plt.title("kNN")

# Different variance
X_varied, y_varied = make_blobs(
    n_samples=n_samples, 
    centers=n_blobs,
    cluster_std=np.random.choice( # randomize variance of each cluster
        [1.0, 5.0, 2.0], 
        size=n_blobs, 
        replace=True
    )
)

plt.subplot(234)
plt.scatter(X_varied[:, 0], X_varied[:, 1], c=y_varied, alpha=0.5)
plt.title("Unequal density")

y_pred = KMeans(n_clusters=3).fit_predict(X_varied)

plt.subplot(235)
plt.scatter(X_varied[:, 0], X_varied[:, 1], c=y_pred, alpha=0.5)
plt.title("kMeans")

y_pred = KNeighborsClassifier(n_neighbors=5).fit(X_varied, y_varied).predict(X_varied)
print(f"kNN score (varied density): {KNeighborsClassifier(n_neighbors=5).fit(X_varied, y_varied).score(X_varied, y_varied):.3f}")

plt.subplot(236)
plt.scatter(X_varied[:, 0], X_varied[:, 1], c=y_pred, alpha=0.5)
plt.title("kNN")


plt.show()<jupyter_output>kNN score (uniform density): 0.993
kNN score (varied density): 0.853
StarcoderdataJupyterStructuredCleanDedup