test sentinels and only append the token if not (#884)
test sentinels and only append the token if not (#884)

the previous approach meant that extra tokens were
generated at the end which makes it unsuitable for
most metrics, as the extra token is not present
in the ref
dpressel/baseline,dpressel/baseline,dpressel/baseline,dpressel/baseline