Who - or what - decides the length of a science fiction novel? The answer has less to do with artistic impulse - and more to do with Safeway - than you may think.
Author Charlie Stross tried to answer the question on his blog:
Here's how one of my editors (who's been in the business for close to 40 years) explained it to me ...
Until the early 1990s, mass market SF/F paperbacks in the US were primarily sold via grocery store racks, supplied by local distributors (400+ of them). The standard wire rack held books face-out, either against a wall or on a rotating stand. And that's where the short form factor novel became established. Thinner books meant you could shove more of them into a rack that was, say, three inches deep. Go over half an inch thick, and you could no longer fit six paperbacks in a 3" rack. And there was only so much rack space to go around.
During the inflationary 1970s and early 1980s, prices of just about everything soared. The publishers needed to increase their cover prices to compensate. But the grocery wholesalers who sold the books insisted "the product's gotta weigh more if you want to charge more". They weren't in the book business, after all, so just as buffalo tomatoes got bigger, so did paperbacks. (Even though this meant there was less room to go round in the wire racks.) You can only get so much milage by using thicker paper and a bigger typeface; so they began looking for longer novels.
Of course, then supermarkets stopped carrying SF novels, and the bookstores wanted smaller books - Stross gives more information on his blog, but that doesn't mean that his commenters necessarily agree with him, offering up different theories:
I think part of the reason is that the nature of SF changed. Science Fiction was much more speculative during the 60's. The average author came up with an idea, the more outrageous the better, clothed with the minimal amount of scenery and characterization to explore it, then wound the story up. By the time the 90's came around the elements of speculation were severely degraded leaving the field with bloated fantastic adventures using the tropes of SF.
[There is a problem with your] notion that books are or were distributed (as paperbacks) by "grocery distributors." I have no idea where you (or your editor) got this, but it isn't true and never was. Most pbs were distributed to the mass markets (grocery chains, drugstore chains, etc.) by each area's *single* magazine distributor, who has had a monopoly on distribution since 1958 when American News left that business. In many, if not most areas, that distributor is ARA, a vast mob-owned conglomerate which also controls food services as well. Now, book stores use more and other distributors, as well as dealing directly with publishers. But you were talking mass market. (Wal-Mart is probably unique; it makes end runs around all distributors by being its own distributor and selling at a 30% discount on cover price...but handles only "best sellers" for the most part.)
[S]cience fiction writers in earlier eras had to work much harder and produce much more. You may think your productivity impressive, but compare yourself to Robert Silverberg, who regularly churned out, what? 6 or 7 novels per year? During the late 1950s and throughout the 1960s, Silverberg's productivity could only be described as "superhuman." And given the low advances in those days, that kind of output was necessary if a writer was to make a decent living as a science fiction writer. Otherwise, the writer typically had to jump ship from novels into screenwriting. Ray Bradbury and Richard Matheson and Charles Beaumont and Harlan Ellison and quite few other talented science fiction discovered this during the 1950s and 1960s. Writing your guts out got you a couple of thousand per novel, if you were lucky, and as inflation skyrocketed in the late 60s and throughout the 70s, it became harder and harder to live on the proceeds of two or three novels per year. If memory serves, Samuel R. Delaney got 1200 dollars for Babel-17. You couldn't live on that for a year even in 1965. In fact, you couldn't live even on 3 times that. Tragic examples like Phillip K. Dick, who was pretty living on soup and spaghetti by the late 70s with the paltry three thousand he got per novel, offered a cautionary tale to other writers. So the monsters of productivity tended to leave the field for the greener pastures of TV and Hollywood, leaving relatively less productive writers. Someone who writes fewer books per year will probably tend to write longer books, so that's another explanation.
In fact, the only thing that everyone seems to definitively agree on is that SF books used to be a lot shorter, back in the day, and then ballooned in size during the '80s and '90s before beginning to shrink again recently... perhaps because authors are spending time online discussing the changing lengths of novels.
Why are SF and fantasy novels the length they are? [Charlie's Diary]