I think that comes mostly with experience and experimentation.
There are a few basics that can be found almost anywhere. The color in the scene that matches the color filter being used will be the brightest part in the scene (a blue sky will be bright if you use a blue filter). The color that is on the opposite side of the color wheel will then be the darkest tones in the scene. The colors gradually in between will be gradually darker or lighter tones of grey.
Caucasian skin tones often look best when using a green filter. Landscapes often look best when using a yellow filter.
I've found that the rest is a matter of trying different filters being used at full strength and then observing what I like and don't like. Once I've settled on a particular color for use as the filter, I then experiment with changing the strength of the filter.
Everything else I do is just a matter of fine tuning the details.