Everybody loves a surprise. Our movies, TV shows, books and other media are constantly pulling rabbits out of hats and revealing that the truth is something other than what we believed. But some twists are better than others — and here's the crucial difference.
These days, we hear a lot about the importance of preserving the sanctity of the Big Surprise — spoilers are destroying pop culture, the argument goes, because they are ruining the experience of discovering a story the way the storyteller wants to tell it. (Even though the average movie trailer gives away more than a dozen spoiler articles.)
But the flipside is also true — storytellers rely too much on endless rabbit holes. There are too many pointless mysteries, with no good answers, and too many people who see the "endless mystification" model of storytelling as the best thing to aspire to. And the truth is, a lot of twists in today's pop culture feel cheap and pointless.
So what's the difference between a cruddy twist and a good twist? It's pretty simple:
A meh twist pulls the rug out from under the audience. But a good twist pulls the rug out from under the characters.
The first kind of twist is just pulling one over on the viewers, or readers. The second is creating a situation where the characters themselves believe that something is the case, and then find out they were wrong.
The latter kind of twist is always going to have more emotional impact, because it actually affects the characters in a visceral way. We care about people who make mistakes, and we root for people who screw up for the best possible reasons. We also feel a surprise a lot more when it's just as much a surprise to the characters as it is to us.
And a really great twist, when a character realizes he or she was wrong about everything, provides a great turning point. Think about the best surprises in fiction (Ned Stark's fate in Game of Thrones, Blake realizing he's sacrificed everything to get to an empty room in Blake's 7, "Pressure Point") and the thing they have in common is the characters having a dramatic realization.
Characters are harder to misdirect than audiences
But setting up a situation where the characters are drastically wrong about what's going on is much harder than just misleading an audience.
Unless the story is seriously capsizing, the audience only knows what the storyteller lets them know. The writer, director or editor controls what the audience sees. You can easily use clever editing or random tricks to keep the audience from finding out a crucial piece of info until you're ready to reveal it. (Joss Whedon has leaned on the "leave out the most important part of a conversation, only to show it in a flashback later" device more than once.) It's trivial to keep the audience in the dark about all sorts of things, just by not showing them.
(Update: Another example of this is the sort of thing Lost used to do all the time, tricking the audience into thinking it's the past when it's the future, or someone is off the island when they're actually on the island, etc. etc.)
But it's a different matter entirely to keep characters charging in the wrong direction. If you don't do it right, then people start to wonder if your heroes are brain damaged. Are they on drugs? Why don't they ask the obvious questions or look behind them? When they ask a direct question and don't get a straight answer, why don't they keep probing?