What is the difference between non-determinism and randomness? I recently heard this -
"A non-deterministic machine is not the same as a probabilistic machine. In crude terms, a non-deterministic machine is a probabilistic machine in which probabilities for transitions are not known".

I feel as if I get the point but I really don't. Could someone explain this to me (in the context of machines or in general)?

Edit 1:
Just to clarify, the quote was in context of finite automaton, but the question is meaningful for Turing machines too as others have answered. 

Also, I hear people say - "... then I choose object x from the set non-deterministically". I used to think they mean - "randomly". Hence the confusion.
It's important to understand that computer scientists use the term "nondeterministic" differently from how it's typically used in other sciences.  A nondeterministic TM is actually deterministic in the physics sense--that is to say, an NTM always produces the same answer on a given input: it either always accepts, or always rejects.  A probabilistic TM will accept or reject an input with a certain probability, so on one run it might accept and on another it might reject.

In more detail:  At each step in the computation performed by an NTM, instead of having a single transition rule, there are multiple rules that can be invoked.  To determine if the NTM accepts or rejects, you look at all possible branches of the computation.  (So if there are, say, exactly 2 transitions to choose from at each step, and each computation branch has a total of N steps, then there will be $2^N$ total brances to consider.)  For a standard NTM, an input is accepted if any of the computation branches accepts.

This last part of the definition can be modified to get other, related types of Turing machines.  If you are interested in problems that have a unique solution, you can have the TM accept if exactly one branch accepts.  If you are interested in majority behavior, you can define the TM to accept if more than half of the branches accept.  And if you randomly (according to some probability distribution) choose one of the possible branches, and accept or reject based on what that branch does, then you've got a probabilistic TM.