<jupyter_start><jupyter_text>Plants Clustering AnalysisA look at plants in different parts of North America. Data Source- [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Plants)- USDA, NRCS. 2008. The PLANTS Database ([Web Link](http://plants.usda.gov/), 31 December 2008). National Plant Data Center, Baton Rouge, LA 70874-4490 USA.<jupyter_code>import csv

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.cluster import KMeans, AgglomerativeClustering
from sklearn.metrics import silhouette_score
import scipy.cluster.hierarchy as sch
from sklearn.manifold import TSNE
# URL to access plant data
plants_data_url = "https://archive.ics.uci.edu/ml/machine-learning-databases/plants/plants.data"

# Import the plant names only (first col in csv) into a DataFrame
plants = pd.read_csv(plants_data_url,
                     encoding='latin1',
                     header=None,
                     usecols=[0])

plants.columns = ["latin_name"]

plants.head()
plants.shape
# Extract abbreviations to use as column names
cols = []

with open('./Data/abbrv.txt', 'r') as f:
    reader = csv.reader(f)
    for row in reader:
        cols.append(row[0])

print(cols)
# Add empty columns for each State/Province to plants df
for col in cols:
    plants[col] = np.nan

plants = plants.set_index('latin_name')
plants.head()
# Extract State/Province for each plant
with open('./Data/plants.data', 'r', encoding='latin1') as f:
    reader = csv.reader(f)
    for row in reader:
        for state in row[1:]:
            plants.loc[row[0], state] = 1
plants.head()
# Drop non-North American columns (dengl, fraspm, gl)
plants.drop(['dengl', 'fraspm', 'gl'], axis=1, inplace=True)

# Drop rows with all missing values (plants not in NA)
plants.dropna(how='all', inplace=True)

plants.shape
# Fill NaNs with zeroes
plants = plants.fillna(0)
plants.head()
# Split data into training and test sets
X_train, X_test = train_test_split(plants,
                                   test_size=0.3,
                                   random_state=42)

print(X_train.shape, X_test.shape)
# Baseline clustering analysis
kmeans = KMeans(n_clusters=7, random_state=42)

kmeans.fit(X_train)
print(kmeans.inertia_)
# Graph number of clusters vs. inertia for "elbow method"
sum_sq_dist = []

for k in range(1, 16):
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X_train)
    sum_sq_dist.append(kmeans.inertia_)
    
plt.plot(range(1, 16),
         sum_sq_dist,
         'bx-')
plt.xlabel('k')
plt.ylabel('Inertia - Sum Sq. Distances')
plt.title('Elbow Method for Optimal K')
plt.show()
sum_sq_dist
# Silhouette score method
sil_scores = []

for k in range(2, 8):
    kmeans = KMeans(n_clusters=k, random_state=42)
    preds = kmeans.fit_predict(X_train)
    score = silhouette_score(X_train, preds, metric='euclidean')
    sil_scores.append(score)

print(sil_scores)

plt.plot(range(2, 8),
         sil_scores,
         'bx-')
plt.xlabel('k')
plt.ylabel('Silhouette Score')
plt.title('Silhouette Score for Optimal K')
plt.show()<jupyter_output>[0.5206021170769546, 0.46944774688158764, 0.3501494151525396, 0.3417105426640366, 0.33350867069438594, 0.13727482081733883]<jupyter_text>The elbow method seems to support 2-4 overall clusters of plants, with the most dramatic change to inertia from 1 to 2 clusters. The silhouette scores also peaks at 2 clusters, and flattens at 4.<jupyter_code># Hierarchical clustering
hc = AgglomerativeClustering(affinity='euclidean', linkage='complete')
hc.fit(X_train)

print(hc.n_clusters_)
# Draw dendrogram - complete
plt.figure(figsize=(20, 10))
dendrogram = sch.dendrogram(sch.linkage(X_train, method='complete'))
plt.savefig('./Figures/Dendrogram-complete.jpg')
plt.show()<jupyter_output><empty_output><jupyter_text>There seem to be 5 relatively distinct 'sub-clusters' in the green cluster, but it's harder to tell given the display of the red one.<jupyter_code># TSNE Visualization
tsne = TSNE(n_components=2, verbose=1, perplexity=30)
tsne_results = tsne.fit_transform(X_train)
plt.figure(figsize=(20, 10))
sns.scatterplot(x=tsne_results[:, 0],
                y=tsne_results[:, 1],
                alpha=0.3)

plt.title('TSNE Results')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup