Predictability, very much like stereotypes, has a bad reputation amongst readers and writers alike. To most people, predictable is the same thing as boring. They want, or say they want, unpredictable stories that end in ways that they did not expect whatsoever.
"I did not see that coming."
That is one of the phrases that most people connect with a story that is well-written and surprising.
It is something of a misdirection.
Stories are not truly unpredictable. The closest sorts of stories that can be called unpredictable are things like Looney Toons, Monty Python and Alice In Wonderland. However, even in this case, the apparent randomness is expected by the viewers and readers. Same with such humorous works as Discworld, with its mountain of puns.
The unpredictability of these stories is, itself, predictable. It is well within your expectations in dealing with such things.
True unpredictability is very easy to achieve. If you decide, out of the blue, to introduce an entirely new character in the last pages of the story in order to solve all the problems, you've just done something unpredictable.
Mystery novel enthusiasts the world over can tell you how popular that sort of thing is.
Anything you do in a story must be either fitting within the framework of what has come before, or establishing the framework for what will come afterwards. Everything in fiction is defined, and the further in the story you get, the more defined that story becomes.
Unpredictability renders such definitions null and void and ruin the consistency of the story.
However, you still want that appearance of unpredictability. You want to be surprising.
You do not want the reader to see "it" coming.
Surprising, however, is not the same unpredictable. You can work with the definition of the world in order to achieve surprise.
When you reach your surprise, or your twist, you want the reader to immediately agree that the sequence makes sense. You want them to have evidence from earlier parts of the story that points to exactly what you just decided to show them.
This is what we call foreshadowing.
You leave clues strung throughout a story that imply a specific end or result. The closer an event is to occurring, the more frequent and obvious you want your foreshadowing to be, but you want to start foreshadowing in small, occasional ways for any major event you have decided on as soon as you have decided it.
At the best, you want this foreshadowing to be so minor that nobody notices it until they've reached the end and all those clues click into place. You want the foreshadowing of minor events in a story combine to foreshadow the greater events.
You want to have enough clues in the storyline that someone who reads your book or story can possibly figure out what is coming, but in a way that few people will piece it together. The same clue, for instance, can foreshadow many different things coming in the longer story.
Also note that only the reader requires this subconscious level of predictability. The characters in your story do not need to be able to predict the ending of the story. In fact, very often, it is better if they can't predict the ending.
Some of the evidence that readers will be using to predict your story, subconsciously or unconsciously, include knowledge of cliches, tropes and stereotypes. As such, you should be well educated about such things. It is best to be aware of these rather than let these guide your story entirely, but a lack of awareness of these trends is likely to ruin your surprises.
Some people like to make use of red herrings, clues that lead the reader off in entirely the wrong direction. For my own style, I do not prefer to deliberately create truly extraneous and misleading clues. Instead, I prefer to have clues that can be read in more than one way. To me, it is best if the reader creates their own red herrings rather than for me to give them a primrose path leading off in the wrong direction.
