Although this is a rare case, I can't take back my vote from comments, like I can do from answers or questions. Is it only my problem, or everyone's? If everyone's, then this is a 'feature request'.
As a matter of fact, the same rule about removing a vote does apply to questions and answers as well. From the priveleges pages:
You can undo your vote by clicking the same button you used to vote, so click the up arrow to undo an upvote or the down arrow to undo a downvote. To change a vote from up to down, click the down arrow, and vice versa. Votes can only be changed for a short period of time, so be careful with those clicks!
The exception to this rule is if the question is edited between the time when your vote was cast and the time you choose to remove your vote. If that has happened, then you are able to remove your vote. Since comments cannot be edited after a period of time, it makes sense that you can never change your vote.
It's not just you: Comments may be upvoted, but they cannot be un-upvoted (except during a brief period after the vote itself, ostensibly to allow one to take back a mis-click). The inability to un-upvote a comment is entirely by design (see: https://meta.stackexchange.com/a/129032/132874)