When you bond, 2 consumer references are added to the stash. Then if you
set_keys that adds a consumer to the controller. If the stash is also the controller, then this means a 3rd consumer for the stash.
If you then change the controller, the controller doesn't inherit the reference, and if you chill and purge the keys, the keys are removed but the 3rd consumer isn't removed from the stash. As a result there's a consumer reference on the stash that can't be removed. Changing the controller back to the stash and setting the keys again only adds a new consumer reference.
This is the case when you use the "+ Validator" button on the UI and choose a different controller, because the controller is set at the last call of the batch:
bond, set_keys, validate, set_controller.
An update to this:
The problem emanates from the fact that both the stash and the controller can call
session.purge_keys, because it removes a consumer reference from the account that calls it. However, unless it is called by the same account that called
session.set_keys in the first place, the reference will be removed from the wrong account (if it has any) and the other account will be stuck with a reference that can't be removed.
So, step 4 to reproduce above should say "Purge keys from the controller". But this "ambiguity" allows for a few other scenarios under which an account gets stuck with a reference.
The easiest fix I see to prevent this from happening is to limit
session.purge_keys to the stash. That way we ensure both extrinsics are always called by the same account. Fixing already affected accounts will be more difficult I imagine.
However, unless it is called by the same account that called session.set_keys in the first place, the reference will be removed from the wrong account (if it has any) and the other account will be stuck with a reference that can't be removed.
Seems like a bug in ref counting and needs fixing. Added to project so that it won't get lost. Thanks for the report!