As agreed in an 1834 compact between New York and New Jersey that set the state border at the bay 's midpoint , the original islands remain New York territory despite their location on the New Jersey side of the state line .