There is already a process in place that automatically detects and flags a variety of low-quality posts, placing them into the Low Quality review queue. I am not familiar with the specifics — like many of the automatic processes the exact details are likely hidden to prevent abuse — but it tends to catch the most egregious cases, particularly very short posts with terrible English. Not sure if it takes blatant formatting abuse into account as well, but I would not be surprised.
As for more substantial posts that just happen to have terrible formatting and/or grammar, I doubt that an automated process would really be viable. Especially for a site like this, where a significant segment of our audience do not speak English as a first language, automatically detecting the difference between "no effort spent" and "very good answer with unfortunately terrible English" is non-trivial.
I, for one, don't think that we want to discourage answers from the second camp; there are a lot of valuable Islamic resources out there that are not readily available in English, and in order to get that information to an English-speaking us, just accepting and dealing with the language barrier is probably in our best interests in the long run.
As for education, that's essentially what voting is for. Users will theoretically learn that they're doing something right when they receive many upvotes, and that they're doing something wrong when they receive many downvotes. If the poor quality of the post is so bad that it's just not worth fixing to extract what (if any) core of "good answer" is in there, just downvote it. If the core of "good answer" is so useful that you feel it's actually worth the effort of untangling the poor writing, go ahead and upvote it.
Voting, being fundamentally based on how useful an answer is, sends a message to the user that this post is or is not any good, but it also sends a message to the rest of the community about which posts are worth reading (even if the formatting and/or grammar is atrocious) and, by extension, which posts have enough value that they're actually worth fixing. If you choose to be the one to fix it, or to leave constructive criticism as a comment, all the better.
If you feel that a post is just so bad that unravelling it is an unnecessary burden on anyone's time, that's what the Very Low Quality flag is for. I rather like Tim Post's description of the flag, so I will quote it here:
The post is an indecipherable mess and can't be salvaged in the amount of time we're prepared to ask our users to spend editing a single post. This could be due to the post being written so horribly it can't be understood, containing horribly formatted code, or both. Put simply, it would take way more than a reasonable amount of time to straighten this mess out, put that responsibility back on the person who wrote it while keeping it from distracting others.
If a user is clearly and consistently showing no effort to make their own posts useful, especially when it detracts from the quality of the site itself, feel free to flag them for moderator attention. Consistent low-quality is something that we do look into and, when necessary, take action against (and having a history of valid very-low-quality flags is very valuable when making a decision). But, for the most part, downvoting most offenses should suffice, as it not only gives the poster a proverbial slap on the wrist, but also helps remove the post from view and allows high-rep users to delete it if they see fit.
In the end the best judge of usefulness is still the community itself, even when that community is still developing.