What if you messed up your notation a lot? While playing in an OTB game where both players take notation, it's not uncommon for people to make a mistake here and there. Is there any rule regarding having a lot of mistakes? Furthermore, what if you deliberately make mistakes such as writing in a random move every turn? Is there any penalty for doing this? Perhaps there's a difference between USCF and FIDE rules.
Is there any rule regarding having a lot of mistakes? Furthermore,
what if you deliberately make mistakes such as writing in a random
move every turn?

There is no obvious rule against this. The obvious place to start for FIDE is article 8.1.1:

Article 8: The recording of the moves
8.1.1 In the course of play each player is required to record his own moves and those of his opponent in the correct manner, move after move, as clearly and legibly as possible, in the algebraic notation (Appendix C), on the ‘scoresheet’ prescribed for the competition.

You must write the moves down but there is no obvious penalty for mistakes, deliberate or otherwise. Which begs the question - Why bother? Why bother writing the moves down at all let alone why try and not make mistakes?
Well, that is answered obliquely in Appendix C:

Appendix C. Algebraic notation
FIDE recognises for its own tournaments and matches only one system of notation, the Algebraic System, and recommends the use of this uniform chess notation also for chess literature and periodicals. Scoresheets using a notation system other than algebraic may not be used as evidence in cases where normally the scoresheet of a player is used for that purpose. An arbiter who observes that a player is using a notation system other than the algebraic should warn the player of this requirement.

Makes sense. If your scoresheet is needed to settle a dispute or make a claim and it isn't suitable (either because you used the wrong notation or it can't be read) then your claim is likely to fail.
Two examples:

9.2.1 The game is drawn, upon a correct claim by a player having the move, when the same position for at least the third time (not necessarily by a repetition of moves):
9.2.1.1   is about to appear, if he first writes his move, which cannot be changed, on his scoresheet and declares to the arbiter his intention to make this move

and

9.3 The game is drawn, upon a correct claim by a player having the move, if:
9.3.1 he writes his move, which cannot be changed, on his scoresheet and declares to the arbiter his intention to make this move which will result in the last 50 moves by each player having been made without the movement of any pawn and without any capture

If you don't have a legible, correct scoresheet then you can't demonstrate a 3 fold repetition or 50 moves without a capture or pawn move for a 50 move draw claim.