Created main.py
Created main.py

Running "python main.py" will ping the e-booksdirectory.com database for all math textbooks, pull the (first layer of their) download links, search for all PDF urls, pull all PDF files to a local copy, then run optical character recognition on each textbook to generate a text file. This text file can be viewed as a vector in high dimensional Euclidean space, wherein each standard basis vector corresponds to a unique human word, and wherein each vector coordinate is the relative frequency of the associated word.

Texts may then be regarded as admitting a distance between one another (in, say, the p-norm on sequence spaces, \ell_p). Here, we choose p=1 and consider the 1-norm between texts. We use k-means clustering to find 7 clusters of texts which, hopefully, correspond to varying difficulties of math texts.
b-g-goodell/book-clustering