I know what you're thinking, I really do. "Blech. ANOTHER Mary-Sue guide. These people all say the same thing! They should just get over themselves!" Or, maybe you're new to the essay section and haven't read all the Mary-Sue essays that pop up here and there. Either way, perhaps I am full of myself to think that I have something different to say, and a different way to phrase it, than all 2386419384935 other writers who've contributed their 2 cents to the Mary-Sue discussion. I'll let you be the judge of that. My role is simply to say what I have to say.
The Group - The Clique
At the beginning of Orson Scott Card's Speaker for the Dead, a scientist named Pipo tells his apprentice Novinha, "Every person is defined by the communities she belongs to." Not only is this true on Card's fictional planet of Lusitania, but it is also true is all of our fiction. Just substitute "character" for "person."
Every character is defined by the communities he belongs to. Let's assume that you've read The Hunger Games, and I'll use Katniss Everdeen to demonstrate my point. What communities does Katniss belong to? She lives in District 12. She's a tribute. That's a good start.
But so far, we only have a vague painting of Katniss, based on the communities she's part of. Readers don't fall in love with vague paintings, and we don't fall in love with generalizations. At this point, pretty much anyone could be the main character of THG. But anyone is not the main character. Katniss is. And if you ask me, I'll say that her character is one of the reasons I kept reading and liking the series, even after the monotony of the third book.
The next thing we must ask ourselves is, "How does my character differfrom the rest of her group?" A group is a stereotype, and differences are a character. So, let's revisit Katniss with this new question.
Katniss lives in District 12 - well, she also lives in the Seam. That's unusual. Her father was a miner, like most men in 12, but her father is dead, unlike most teenagers in 12. She sneaks over the fence to go hunting to support her family. Most 12ers don't do that.
Katniss is a tribute - well, she's also a volunteer tribute. The first volunteer tribute from 12. She's a tribute who tried to defy the games instead of just playing along.
Of course, it should be pointed out that this is a very easy example - the whole premise of the trilogy is built upon Katniss's differences and how they caused a revolution. But it does prove a point.
As much as you can, narrow down the groups of people that your character belongs to. Think about the "establishing shot" for a movie. It starts with a general shot of, say, Los Angeles. Then it zooms in on the neighborhood where the movie takes place. Then it focuses on the house of the main character. Finally, you're in the bathroom brushing his teeth with him. This is exactly the kind of narrowing of groups that I'm talking about.
Not that this should show up in your writing, necessarily. This is for you, for your outline or character sketch or whatever you prefer to call it. Fiction is an art of contrasts, and what groups your character belongs to (and the groups she doesn't) and how she differs from those groups can help establish your character beyond a vague stick figure. And, you may even take it a step further. How does she react to her differences? How does she react to her similarities? What does she think of the people who are in her groups? What does she think of the people who don't? Anything you decide about your character will breed questions to help you decide a dozen more things. Such is the beauty of fiction :)
This is a more commonly discussed problem when it comes to the topic of Mary-Sues. Simply put, all characters must have something that they want. As mentioned in my essay How to Become a Great Writer in 10 Minutes, this is a very general character formula:
(name) is a (adjective) (age) (noun) who wants (this)
Example: Jenson Andrews is an genius 18 year old student who wants to prove a mathematical equation before his arch nemesis/fellow genius can.
Note how Jenson's desire basically just wrote my plot for me. Of course, I don't know how he's going to go about doing that, or what complications are going to arise, but I have a conflict.
My favorite thing to tell my beta-ees is that your character must be ACTIVELY STRIVING after something throughout the novel, and he must want it badly enough (for whatever reason, which you must make clear) that he is willing to persevere and even do stupid things to accomplish it. Just like there is a difference between the writer who says "I want to get published" and the writer who sends his stories to agents all over, there is a difference between a character who dreams about his goals and a character who jumps off his bed and drives off to go find them. If your character is not willing to risk everything for his desire, he needs a new desire - or you need a new character.
(I also will point out the word "everything" in that sentence. How your character will define that word, "everything," says a lot about him, and it informs the plot more than you might think.)
It doesn't need to be noble, and it doesn't need to be to save the world; it just needs to be something that your character cares enough about to do whatever it takes. This concept is very closely related to what writers like to call "tension" - and we can open that can of worms some other time.
Of course, the most common explanation for Mary-Sues, except usually it goes by another name: flaws. I'll spare you much explanation on this one, because I'm sure you've heard it before, but let's put it this way: your character is not a god. So let's act like it.
I think writing opens opportunities for a lot of self-examination. Not to be a killjoy, but I'm sure there's at least one thing about yourself you don't like - besides some physical feature, because we all are embarrassed about the zits on our faces. There are some undesirable traits in every person, just like there are desirable ones. Let's get personal for a minute. One of my poor qualities is that I'm kind of flaky. I'm also not a huge fan of trusting people. But (and for the record, it's not bragging to focus on/share your good qualities every now and then) I'm pretty good at making my friends laugh, and I'm pretty decent at teaching people. I think noticing the various sides of yourself is the first step to unpacking the layers of your main character.
People say that Mary-Sues are characters without flaws. This is true. He's the tall, dark, and handsome demon battler who never seems to feel pain and is a great kisser - I may or may not be thinking of Jace Wayland from the Mortal Instruments, but I don't think he's all Mary-Sue. Your main character is just as human as the rest of us, and she struggles with self-esteem just like we do. Maybe she has trust issues just like me. Maybe she's super full of herself. Who knows? The important thing is that you know - and your readers, after they've spent some time with your story.
But you know what's equally bad? Characters who are all flaws. This would be what I'll call the Shallow-Villain-Syndrome, or SVS. Folks, nobody is evil for the sake of being evil. We've moved beyond The Lorax where the bad guy is some evil maniac who lies to everybody about air quality and makes millions off of gullible citizens and then laughs about it "because I'm just so evil mwuahahaha." Bad guys are the main characters of their own stories, and they can't be any less developed than your good guy. Bad guys have reasons for the things they do, and they have motivations (ACTIVELY STRIVING, remember?), and they have good qualities. Usually, those good qualities are used for evil purposes - and that begs the question, WHY? Why the heck would I, for example, use my teaching abilities to teach the beauty of communism to sixth graders? The simple answer is that I must think communism is good.
Here's a secret: most bad guys do the things they do because they think they're good things. Even Hitler thought he was doing everyone a favor by eradicating the Jews. Good deeds gone awry are generally the cause of most evil. This is another discussion, of course, but the important thing to remember is that the bad guy always has a reason for what he does, and it's always noble in his eyes.
Your protagonist will not be all good and your antagonist will not be all bad. A mix of traits is the key, and a mix of emotions at various points in the story - a character who is happy all the time is no more fun to read about than a character who's angry all the time. I have mood swings, and I have good qualities and bad qualities, and so do you, and so does your character. Take some time to think about this one.
Sometimes people mistake a cliched situation as an indicator of a Mary-Sue. This is not always the case (although people do think these things for a reason). Cliches do usually indicate lazy plotting, which includes lazy character development. I have an easy cliche-avoidance trick, and the more you practice it, the less lazy you're inclined to be with your writing in general.
When I'm developing a character, or a plot, a host of ideas inevitably come to my mind. So I take and idea, turn it around for half a moment, and then I say, "Okay, great, but why else could that be?"
My theory is that we generally tend to think in cliches and stereotypes, so taking the first idea that comes to mind is usually a rip-off of something that's been done a thousand times before, or it's an obvious solution. Asking, "What else forces you to use your creative mind in a way you may not use it very often. So, let me ask you a question. What else could cause that? What else could go wrong? Why else is she like that? What else could happen?
I will bring this up, though I won't spend a lot of time on it. People run away from a weird name screaming "Mary-Sue!" faster than you can finish saying it. Of course an unusual name is not in itself an indicator of a Mary-Sue - that would be completely ridiculous. However, if a writer, especially a new writer, gives her character a name like Honeydew Lovely Aesthetica, it tends to reveal an assumption that the character will be considered special and unique based on her name. It's very much akin to the "defiant jaw" syndrome (i.e. "His tan skin was stretched tautly over his sharp, defiant jaw"). Very little about your character that she cannot control should have anything to do with the way she is characterized. A bizarre name is usually an excuse not to go through the process explained in section 1, with the spiel about what groups your character belongs to.
So, I've said everything that my brain told me to say about the Mary-Sue. I hope this has been helpful to you, and I hope I've explained things in a new way, different than all the other character guides that are already out there. The next step is on you: go out and find your character and apply anything you've learned!