We don't check a user's credentials at the door.
Your claim that "we are brothers" is a lovely ideal, but it's optimistic at best: There's over twenty-five thousand users on Islam Stack Exchange, and quite frankly we have absolutely no idea how many of them are practicing Muslims, much less how qualified any of them are at answering questions.
Every single one of these people can ask and answer questions, and here that means questions about Islam, about the deen that you hold so dearly. Some of them could be imparting valuable knowledge, some could be spreading misinformation out of ignorance, or even others telling outright lies out of malice. Such is the nature of the Internet.
You are correct when you say Islam Stack Exchange is not like the other Stack Exchange sites: We have little in common with most of the Stack Exchange network which focuses on more technically-oriented topics. The closest site to ours is probably Christianity Stack Exchange which, quite frankly, has far stricter rules, stronger enforcement, and does a lot more downvoting than we do.
Our voting system is a major reason that people trust Stack Exchange to answer their questions on any number of topics: The people who ask probably don't have the knowledge and skills to recognize honest and useful answers from misinformation and lies. I find this especially the case here, many of the questions we receive show little to no understanding of Islamic fundamentals or the study of fiqh.
Voting is how we, the community, curate the information that is posted. It's a form of peer review, since the people who are best qualified to recognize valuable and useful information are the people who have expertise in the topic, in this case the topic of Islam. Users who earn a lot of reputation do so because the community has recognized the value of their posts; those users in turn are granted increased curating powers to help ensure that useful answers are promoted over less useful ones.
Downvotes are supposed to be a criticism of the post, not the user: They are not necessarily telling you that your post is wrong, so much as it's telling you that your post is not useful. What makes a post "useful" varies widely depending on the question and what information they're seeking, not only for the questionasker themselves but for anyone who might be searching for answers to that same question in the future.
Instead of taking downvotes personally, look at it as an opportunity to improve. The more useful your posts are — the more useful the community sees them — the more it will help you spread the deen.