Custom metrics for SEO robots.txt (#178)
Custom metrics for SEO robots.txt (#178)

* Initial SEO additions for almanac.js

added rel nofollow, ugc, sponsored and imageLink counts to the seo-anchor-elements data
added title-elements parameter that returns an array of the text in title tags
added content property that ruturns visible word count and count of each heading tag level
added image property that reports on alt, loading, with and height attribute use
added seo-data-nosnippet property that counts the use of the rel seo-data-nosnippet attribute

* Added h1_text

I added the h1-text variable  to retrieve the text from the first h1 tag in the rendered code.

* Updated h1-text using new instructions from Tony

* some renaming to try and be consistent
updates to support SQL query changes from using raw body to almanac.js

* Added structured-data that includes counts of json-ld scripts and how many error

* Updating used by info based on the 2019 scripts. There may be a few no longer used.
Moved property comments to be consistent
places a gap between properties to make them more readable

* expanded heading data based on metrics requirements
first draft of expanded structured-data info based on requirements
amp property with data based on requirements

* try catch on properties and whole thing
using $WPT_REQUESTS to get http headers
seoText function to convert node into visible text including img alt text
structured-data property enhanced to support requirements
removed 10.5 as new structured-data property includes all it had (and works better)
added externalSameDomain and externalDifferentDomain counts to seo-anchor-elements
title and heading properties refactored (smaller but more useful output)
removed seo-titles as covered by headings
added canonical property
added robots property that calculates final robots settings.

* fixed old issue with parseNodes discovered from almanac failing last year.
added try catch around whole thing
checking www. on sd context names
removed attempt to remove // CDATA in json-ld scripts as it broke things

* improved errorLogging
renaming properties to the underscore format
access to raw html and headers via $WPT_BODIES
properties updated to split response data into raw/rendered/header versions
http_status_code, noscript, buttons, dirs, inputs, scripts, videos, audios, raw_html properties added for the Markup chapter
meta_description, hreflang properties added for the SEO chapter
target_blank/noopener/noreferrer info added
amp info consolidated into one property
h1-text removed

* fix to comment counter

* fix to comments checking and sd property name

* changed anchor_links to anchors as it analysis more than just the ones that become links
added protocol and target name data
added a markup property that lists deprecated elements

* dofollow and follow links :-)

* changed errors to logs
rel_alternate_mobile
compatMode
classes property that lista all class names
left blank values as blank values (works fine as object properties
some refactoring
merged inputs based properties
anchors logic completely restructured.
including primary title, meta description and heading text
structured data checks id usage
iframes property to capture new loading attribute
amp supports the emoji way of specifying amp
robots reports on the google meta tag
svgs added that identifies common ways svgs are added

* fix for issue with raw html not finding head tags
fixed test for change in title

* Added robots_txt custom metrics file

* this time I saved the file before commiting

* added more try/catch error logging
made regexes case insensitive
no longer gathering all class and id values
anchors now returns both rendered and raw results
fixing things like names
added size data to the raw_html property

* Improved comments

* extra info in the images property
cropping title string in cast it's super long

* Split code into two new custom metrics files
added detecting of SD sameAs
testing

* reverted almanac.js back to the master (hopefully)
renamed wtp_metrics.js to wtp_bodies.js

* branching out robots

* Update custom_metrics/robots_txt.js

remove commented out line

Co-authored-by: Rick Viscomi <ad0d620829b39d553e837da62a9ee1d462d6a941@users.noreply.github.com>

* nullish coalescing to zero

Co-authored-by: Antoine Eripret <8d3338e780a29dced33a0f958c62487443f0441e@gmail.com>
Co-authored-by: Rick Viscomi <ad0d620829b39d553e837da62a9ee1d462d6a941@users.noreply.github.com>
HTTPArchive/httparchive,HTTPArchive/httparchive