I'm pretty sure it's an average of all votes. ie: a vote of anything less than 4 for someone who is rated higher than 4 will result in a decrease. By the same token a vote of 4 for someone with a current rating of 2 will result in an increase. That's how I see it anyway.