Buyer security on BrickLink I'm interested in buying LEGO bricks from BrickLink; the inventory is impressive, prices are good, and it is recommended by many LEGO builders.

Since you buy from unknown sellers abroad, I want to minimize the risk of scams or getting disappointed in some other way. What steps can I take to minimize this risk, especially concerning the payment method? Generally speaking, how big of a problem is non-serious BrickLink sellers?

I don't see much discussion of this at all on BrickLink's How to buy page.
As a buyer on BrickLink you place your order to the seller, who then must acknowledge your request and quote you a full price (including any taxes, shipping costs, or other fees; most buyers will lists these additional cost details on their individual seller page) if you don't like the price they quote, you can walk away.  If you agree then you need to give them the money.

Most of the payments on BrickLink are processed through PayPal, which for all of its many faults, consistently sides with buyers over sellers in cases of disputes, so you can not only dispute any problems using BrickLink's resolution tools, but also with PayPal to get your money back.

BrickLink also has a feedback and reputation system you can use to check a sellers previous sales; so you can see if the buyer has any negative feedback or problem orders, but also it can show you how much volume they are processing.  Someone making a lot of sales is likely to take the process very seriously.  BrickLink tends to take non responsive sellers NRS or non paying buyers NPB very seriously, if you receive 3 of these complaints your buying or selling privileges will be permanently revoked, so someone with a large positive BrickLink reputation, likely isn't having these problems.  BrickLink has a whole section on Problem Resolution you could read through if you are concerned in how the different processes work.

Also, if you are concerned about buying from another country, BrickLink allows you to filter sellers based on country.  Buying from only your country will likely limit what is available, but it can make the process simpler in terms of shipping, customs, and different currencies.