Testing an engine (UCI) against an older version of itself I am making modifications to a chess engine.

I want to get quick feedback on whether the modified version seems stronger than an unmodified version or not, by fighting the older version multiple times.

Does anyone know about a good tool to do this? (Currently I have some homemade code to do this using python-chess ... but a tried and tested tool would be preferred)
Cutechess is a very good solution for this. You can use either the CLI or the GUI to run matches of engines (both UCI and XBoard protocol), and at least the CLI even supports SPRT as a termination criterion of the match, see the help.

Cutechess is also heavily used in the development of Stockfish when running matches between different versions of Stockfish on fishtest.