...presented in roughly chronological order with, no doubt, some timeline errors since I'm not James Mal.
OD&D -- The Categorical Era
The idea in this era was that every kind of PC fit into a broad category, and the classes were just ways of defining these broad categories.
In the beginning: D&D Is A Kind of Wargame
Therefore, everyone on the battlefield can be divided into:
Fighting Men, who generally fight things with weapons, and
Magic-Users, who generally fight things with magic.
(Note that this isn't necessarily automatically limiting. Theoretically, all PCs ever made could fit into one or the other of these categories. There's only a problem once you say that magic users can't use weapons or armor. Then suddenly all kinds of characters known in literature [or not yet known, but imaginable] don't fit the scheme.)
(Also, this is why they aren't called "wizards" and "warriors". Just because you can't use spells doesn't make you a warrior, and just because you do, it doesn't mean you're a wizard. This naming convention would become outdated the second that the next class came around...)
Wait, D&D Is A Game Of Dungeon Exploration
So it might be useful to have a class that's useful in a dungeon, despite not being much in a fight. Thus: The Thief.
Oh, Wait, I Read A Book That Makes Me Want To Put Fighting Priests In But They Don't Fit The "Everybody Uses Magic Or Uses Weapons But Not Both" Post-Wargame Schema
Oh, Also, I Like The Idea of People Being Able To Be Elves or Dwarves But I Think That Should Make A Mechanical Difference In The Game And There Is, As Yet, No Way To Differentiate Categories Of PCs Except by Class
Thus: the Race-As-Classes. The Elf class, The Dwarf class, etc.
AD&D--The Archetypal Era
Hey, Wouldn't It Be Cool To Invent Some "Specialist" Classes, Where The PCs Could Trade Being Able To Do One Thing For Being Able To Do Something Else?
Thus, the sub-classes: paladin, druid, ranger, illlusionist, assassin, and many of the classes presented in Dragon Magazine. Plus, arguably, multi-classed characters.
This was a big deal.
What then happened was a shift in the thinking about class--the idea became that class represented, essentially, not the broad category into which the PC could be sorted for certain purposes, but rather a sort of "genre" of PC. That is, the class wasn't there so much there to place limits on the PCs mechanical function in the game, but to define what kind of character the PC was.
In the original scheme, a knight was a kind of fighter. In the new scheme, a knight was like a paladin, but--fuck--less religious. Could he just be a fighter? Well, but why should the paladin get its own special subclass with attendant powers yet the knight doesn't? Damn, we'll need a new class...
After this happened, having, say, a "druid" as a subclass seemed less like a fun bonus extra for people who wanted to play a specialist cleric and more like a sort of promise that one day every sort of fantasy archetype would have its own subclass with special powers.
By This Time The Game We Invented Was Making Us Lots Of Money So We Bought Some Weed
Thus: Monk and Bard.
Oh, Wait A Second, If The Idea Is To Have A Class Representing Every Fantasy Archetype, There's A Whole Bunch We Haven't Covered Yet
Barbarian, Cavalier, Thief/Acrobat, etc.
The '80s D&D cartoon is actually a leading indicator here--each character on the show was a different class--implying that character class and personality type were essentially one. I suppose it's even possible that the cavalier (paladin without religion) and the acrobat (thief without stealing) were invented for the show--anybody know for sure?
Some Other, Non-D&D, Games Being Made Around The Same Era
Hey Wait, This Whole Categorizing-PCs Thing Seems To Be Kind Of Unwieldy, Let's Just Go With Another Thing Entirely
Thus: skill-based systems.
Ok, But Class Is Kind Of Fun, Like I Know Being Defined By Your Job Is A Drag, But There's Something Kind of Interesting About How These Archetypal Professions Help Describe The Game World...
Thus: profession-based systems like Warhammer.
Later D&D--The Customizable Era
By this point, the people running D&D have to balance two considerations--one one hand, they feel class is an important and defining feature of the game that sets it off against all other games (How do you know it's D&D if you can't say "I'm a 3rd level cleric"?) but, on the other hand they want to give the players the same freedom that PCs have in newer systems.
Ok, We Have Classes But Hey It's Not Like There's A Character You Can Build In Any Other Game That You Couldn't Also Do In D&D
Thus rules that "soften" class divisions: skills and skill points, trading armor protection for spell-failure chance or armor-class bonus, etc.
Hey, Wait, I Bet We Can Make Money Off This...
Thus: prestige classes.
Careful examination of published prestige classes (which, I admit, sounds like the title of the chapter right after "Waterboarding" in the Geneva Conventions on Torture) shows that the people writing them not only decided to (or were forced to) mine every imaginable fantasy archetype for new classes, they also mined every imaginable fantasy character.
Many prestige classes are so detailed that they could basically describe your character for you, soup-to-nuts. "The prophet of Erathaol is a seer and visionary, a medium of the heavenly will, pronouncing judgement on corruption and evil in the world, speaking words of comfort to the oppressed and downtrodden, and announcing the work of the archons in the world."
It's a trade-off--you get power, you give us your personality (and $32.95 for the book you found the prestige class in).
I never played D&D with prestige classes, but I imagine it's possible it could actually produce an interesting dynamic, with the PCs changing over time as their role became more defined. Those of you who did--did the PCs tend to become more like players wanted them to be as they got prestige levels, or did their roles suddenly take big left turns when they got a prestige level?