Are black filaments more brittle? I've been 3D printing for a while and I've noticed that, when printing small parts, my colored plastics (PLA, PLA+ and ABS) have better layer adhesion than black ones.

Did you notice this?

What could be the cause?
Not inherently.
There are two things at work that might cause one color to test weaker than others even as its properties otherwise are functionally identical:

A bad print among good ones.
A bad roll among good ones.

Let's take a look at both, then do a little excursus into plastics and color.
A bad print
There are probably thousands of reasons a print might fail, but bad layer bonding and squish-ability under torsion strongly hint to under extrusion. Now, under extrusion itself can be caused by a plethora of reasons: a clogged nozzle is equally as possible as too thinner diameter as is just a bad temperature. The last one is, in my opinion, the most likely culprit: filaments may look the same and feel the same and bond the same, but in different colors, they sometimes demand different print settings.
As an example, I print most of my Kaisertech PLAs at 200°C, as that offers a quite good result for all of them. Yet when I started I had a white China PLA and a crystal clear PLA from the same manufacturer, both came from the same warehouse in the same shipment. The clear one is quite more brittle on the roll, but their starting-to-print temperature differs by 5°C - the white started to extrude at 180°C decently and printed ok at 195°C-200°C, while the clear needed only 175°C to start to be extrudeable and was really printable at 190°C. Yet recently I tried the same roll again to achieve fully clear prints, and with 210°C and lots of overextrusion, I managed to go almost solid-clear. Because of such experience, I suggest tweaking the settings.
A Bad Filament
There are several reasons why one roll might resulting in bad prints, but the most prominent are that the roll has gone bad over bad storage. It might be stored too hot or too humid, making it brittle or bubble in the hotend. Aging under UV plays a role (it degrades PLA). And dimensional accuracy plays a role because it affects the whole roll of filament. This is why tests should always be performed with equally treated and measured samples to achieve comparability.
Excursus: Plastics and color
What gives a plastic its color? Pigments added to it. Now, pigments can be of varied kinds. Usually, they are embedded in the plastic (=not bonded to the carrier plastic), and the plastic polymer is often either inherently transparent(ish) or white. Let's take some examples to look at...

Yellow. Yellow can be made from a lot of stuff but many yellow pigments react to UV light by decay more than other colors, leading to yellow to fade quickly in comparison to other colors. It has varied chemical compositions, often they can become quite complex.
Black. Black pigment is typically the most simplistic coloration to achieve: pure powdered carbon is one of our most potent black pigments, and also one of the cheapest, making black plastic one of the most common plastics. In contrast to other colors, carbon can't fade. But the plastic around it decomposes and turns white, fading the color this way.

Now, most colorings are - in physical terms - sizeable. Some few to a couple dozen atoms, making them range in the Angström (~Atom diameter) to few nanometer area overall. However, even something as complex as $C_{22}H_{20}O_{13}$ (Carmine) is relatively small compared to the $(C_3H_4O_2)_n$ of PolyLacticAcid, aka PLA. Poly tells us that n is at least 100, because shorter chains are oligomers, not polymers. In comparison, our red carmine pigment is more dense, much more compact in fact. As a result, a 100-chain of PLA is not just in the Angstöm area but in the dozen nanometer to micrometer range - a magnitude of at least 2 larger. Unless we have a huge excess of pigment or a pigment that reacts with the plastic under heat, then the impact of it on the strength should be neglectible to the other fillers often used.