There are a few ways to think about the "closeness" of a color to another color. A simple method that might work for you is to consider the RGB color space as a 3-dimensional euclidean space and just calculate distances using Pythagorean Theorem:
Dim colorDist As Double = Sqrt(Power(r1-r2,2)+Power(g1-g2,2)+Power(b1-b2,2))
Then you can select a maximum permissible distance.
There are more sophisticated ways that involve conversion to HSB color space or treating RGB color space as a vector space and taking inner products. If the Pythagorean Theorem method doesn't suit your needs, we can explore the more sophisticated solutions.
Note that in the example picture you posted, the non-yellow pixels are all black. If this is representative of the real images you have to use, then it will be easier to just detect black pixels and turn them white and non-black pixels to black.