Can I reply to an illegal move with checkmate? I was playing with my brother, and we both had equal number of pieces.
He carelessly moved his king in the direction of my queen. I was about to checkmate my brother, but he told that my mating move was invalid.  If his king comes in my direction and gets caught, I must say that I don't keep there.  

I was confused. Can anyone please help me?
Yes. Putting your own king into danger is considered an illegal move. Depending on the context, illegal moves can be treated in different ways:


In most casual games, players just agree to roll back to the position before the illegal move was made.
In tournament games, the player who didn't make the illegal move gets 2 extra minutes on the clock. I guess this does not apply, as most likely you aren't playing with a chess clock. Anyway, the second illegal move is punished with a loss


Since you're playing a casual game with your brother, just go for the one you agree on!