CI: build only on pull request or push to master (#1005)
CI: build only on pull request or push to master (#1005)

I think this is all we need. Because I'm pushing to branches on the main
repo, on my branches I see almost 70 checks. 33 for `push` and 33 for
`pull_request`. This will remove the duplicate `push` checks.
rmagick/rmagick,rmagick/rmagick,rmagick/rmagick