Despite being an old thread, Im going to jump into this because I love me some world-building. Its where my stories usually start, its what I enjoy. I got started writing about the time i got started in DnD, and so the GM mentality mentioned earlier definitely slants my view. But because of that I have had to think a lot about the limits of world-building, whats its good for and where it gets in the way. Basically it about telling myself when to stop building the world and start bloody writing something :-p
Anyway, heres where ive gotten on it:
To clarify, Im talking from the Writer's perspective here, about what the writer needs to know, more than about what they actually show the reader these things. How to convey the history to the reader is a whole other conversation, and is very situational.
Anyway-
Backstory is crucial. It it what defines your characters, determines how they feel, how they react, how they experience the world. Without a solid backstory you have no idea where your characters are coming from, and so will have no idea where they should be going. Backstory is the Path they have walked. Without putting real time into developing the backstory, your character will feel hollow and your reader will fill in the gaps with whatever is familiar (usually cliches).
World-building is about how well lit that Path is. It can vary much more than backstory, depending on the tone of the tale. But the key is that you always need to know more than your readers. If you are telling a story that has a lot of confusion and suspense and an "anything can happen next" feel, a whole lot of world-building isnt really necessary, because its all situational. But thats because your character's perspective has shrunk down to immediate concerns, they are riding the moment, and long term or big picture ideas are not important. Your path is dark, foggy, and has things lurching out at the reader from no-where, and only vague shapes looming in the distance. And that can be a good story, whole genre's are based on it.
But the more your readers need to know, the more you need to have worked out ahead of time. And the more your story is going to vary from what is familiar, the more you need to present. If its a cop story on the mean streets, you can pretty much pick a level and theme of crime and let the Genre fill the gaps with the familiarities of the legal system, paperwork, IA investigations, street sources, etc. Take the same story and put it in a feudal society, or on a spaceship, or in some historical period, and suddenly you have to direct the reader instead of expecting them to be funneled naturally to what you expect. This is best done, IMO, by shining a spotlight on either the similarities with the familiar, or its differences. If its crazy aliens and a backwards legal system, maybe you make a point of the standard Cop grips about long hours, paper-work, and crappy supplies due to budget cuts; at the same time you highlight the fact that their pens are worms that poop ink and or that the only sentences are public beating, exile, or death.
On the other hand, its easy (for me at least) to fall in the trap where you have a huge amount detail in the world that it bogs down the story or you feel the need to include an giant exposition dump in the middle of some scene. This is an easy way loose readers. You cant tell them a bunch of information like that until you have given them a reason to give a damn. But by the same token, the more invested they are in the story, the more of the details they will soak up, the more they will want it, and the more hours they will dump into picking apart your work for fragments to theorize with on some internet forum. The same can be said for backstory though, you dont have a character stand up and recite their entire a life story very often, but if that life story has been a big gaping hole with a spotlight on it, then the reader will want to know, will want that piece of the puzzle.
Another thing that you gain from the time spent world-building is a sense of a bigger world. Thats not always what you may be after; you can tell a story that is carried entirely by the characters, a literary Our Town where the world is a vague backdrop at best, and the action on stage. Older literature trended this way much. <i>Pride and Prejudice</i> had a war going on, but the specifics in no way impacted the tale, and long exposition about the political pressures of the day would have been pointless. But the critical claim of <i>Pride and Prejudice plus Blood, Guts, and Zombies</i> shows that modern audiences enjoy a bit more of a complete picture; especially when something supernatural is afoot, because by definition the familiar rules have already been broken. When that happens the writer needs to step in and replace them with new rules, or the readers loose their way. So toss in other bits in passing. Mention other events and characters, other stories. Toss in a mention of the Jade court, just to make people wonder. It can be in passing, but it gives the sense of wider vista's to explore, and pulls the reader in. Part of the success of Harry Potter was in how much other stuff had gone on and was continuing to go on off screen, and it doesnt take much. The bustle of key locations, all the various historical names mentioned in the classes, the newspaper gossip. Its what turns a stage into a world. If your world has only one story to be told, it can feel small and boring. Your story should be just one of the many stories that world could tell.
It doesnt have to be a crystal clear world before page one, just have all the rules and framework outlined in your head. The rules are important to keeping a reader interested, invested, and satisfied. The difference between a successful twist or red hearing or foreshadow, and one that seems random and out of place is the framework that the world-building gives you. If your story takes a sudden left turn, but they find themselves in a place that make sense they are surprised but still with it. They dont have to see it coming, but you need to have provided enough framework so that it make sense in retrospect. At the end of the Con movie you want them to see all the clever clues that they missed along the way, not just inform them that some stuff happened off screen to make everything OK. And with anything that can be called scifi/fantasy that is even more crucial, because you have already set the precedent of normal assumptions and rules being broken (thats what make it fantasy). So as a writer I like to have as much of the world-building done as I can ahead of time, or at least some big pillars that I can count on - Isles of Stability. I can change my world as I write a book (its gets harder on book 2 and three, as things are anchored in the audiences mind)and I can add to it, I just need to maintain consistency in front of the reader. I can always say so-and-so was lying that time, or was just misinformed himself, but use that too much and they will stop believing the characters as sources of information, which then limits the writers options to give information.
One of my biggest problems is often doing too much world building and not enough character dev up front. You dont need to have names and backstories of every player that may come on the board, but have the framework. Again, this is especially important in Scifi/Fantasy because you are redefining what is Possible and what is Impossible; Make sure you actually have it defined. But its not a DnD game, all the player are in your control, you dont have to have stuff ready in stand-by because nobody is going to wander off. But if there is going to be a Magic Knight, know there are others out there, make it three if that plays into a mythology link you enjoy, but you probably dont need them named and statted until they actually show up. But if they figure heavily into the backstory, by all means have their details down. Know that there are 5 schools of magic, and that they do different things, and know their limitations. But you probably dont need to spell out the organizational structure of all of them or their leader's backstories unless it actually impacts your story. But if you mention it in a news article, as part of the backdrop, you might mention him by name instead of a faceless title, just so there are extra details floating around. Those details become hooks later on, and you will often be surprised to find you have laid useful groundwork without even realizing it.
Wow, that was quite a rant, where did my morning go?