Whether or not he wants it that way, the "comments" section of James M.'s (fantastic, scholarly, useful) Grognardia blog has become a sort of forum for everyone who wants to talk about what they do or don't like about D&D (guilty), no matter how tangentially related to the topic it might be. So, predictably, when he posted this relatively innocent bit about experience points it ignited a comment-storm about xp.
Now, I don't know about your players, but here's the deal on mine:
Experience points don't motivate my players at all.
(This is my Rule About X.P. #1)
Don't get me wrong here: my players like xp, my players will take xp, my players will get angry if you try to take xp away from them once they have it, but they never do anything because they think it'll get them x.p..
Here's how things usually go:
The players show up. They are excited to finally get to play D&D.
They get their PCs out and then try to figure out where the action is--"action" can in this case be defined as "whatever seems interesting to do, find, or kill in a given adventure".
They find some action. They have fun and snacks.
This Action either: Reveals More Places Where There Might Be Action or Sets Off An Unfortunate Chain of Events. Or both.
The PCs try to find more Action, or extricate themselves from the Events, or both. They have more fun and snacks.
Then the players go home.
Then a few days later they get an e-mail where I detail all the x.p. they got and how, and they go "Oh yeah, xp! Sweet."
If they get enough then maybe they level up--this changes the nature of the game slightly and so keeps things interesting.
Anyway, the point of rule #1 is that the widely-held belief that xp is necessarily an important motivating engine in a game and therefore knowing what players get xp for in a game allows you to gauge the philosophy, meaning, or morality of a given game is a fallacy. At least with the people I play with.
There are many corollaries of this rule--one being that not "rewarding" something with x.p. doesn't mean it's not an important part of the game. So, while it's true that in OD&D the players don't get xp for role-playing (in the sense of "acting"), neither do they get xp for eating pizza or quoting Monty Python movies--but just try to stop them.
If you look at the AD&D DMG, despite lack of rules rewarding good role-playing, there are rules which punish bad role-playing. The DMG also has recommendations for punishing other kinds of behavior at the table which make the game less fun. The idea is: Gary assumed the players would be role-playing and not acting like dicks, so he felt no need to positively incentivize them to talk like a funny dwarf or not attack other players for no reason. It was only when RPGs got so popular that dickheads and teenagers started playing them that more complex incentive schemes started getting developed.
(Incidentally, in the DMG, Gary also seemed to assume--correctly in my limited experience--that storylines would naturally develop out of games and so storytelling didn't need to be incentivized either.)
I beg you to keep Rule #1--the only important rule--in mind while we move on to the other Rules About X.P..
(#2) The X.P. Value of A Given Treasure is Presumed To Be Proportional To The Amount of Problem-Solving Required To Get That Treasure
Bilbo didn't get x.p. for trusting Gandalf, walking around with dwarves, surviving a goblin attack, and then beating Gollum at a riddle game. But he got x.p. for the ring, which abstractly represents all the effort of doing all of that stuff and more.
The idea is to reward problem-solving without having to calculate every little thing--so instead of rewarding the individual steps in the solving of the problem, the game gives a lump award in the form of treasure.
It's not meant to be "a reward for stealing" (though it can be that) it's a reward for all the adventuring that lead up to the stealing.
D&D is, as everyone knows, very big on abstractly resolving stuff involving a lot of variables rather than calculating all those variables separately. And yes, such abstract systems are not infallible--but rule #2 is the intention behind xp-for-treasure, and if you want to fuck with a rule, it's good to know why the rule was there in the first place.
(#3) The X.P. Reward For Killing Monsters Can Be Thought of As A Bonus Reward on TOP Of The Abstract Treasure Problem-Solving Bonus
In other words, you get a treasure reward that accounts for all the blood, sweat, tears, and problem-solving involved in getting that treasure, but if some of that problem-solving took the form of monster-killing, you get a bonus.
Why exactly you get this bonus is a matter of debate, but I think one of the most important reasons is: unlike traps and other problems you solve in D&D, monsters are entirely defined in the rules. That is, you don't get a set xp for solving riddles or defeating traps because Gary had no way of quantifying how hard your DMs traps or riddles were, but he knew how hard your monsters were because he statted them himself. So the monster bonus represents a degree of granularity and detail tacked on to an otherwise abstract system.
In 3e, they tried to break it down further by giving traps challenge ratings just like monsters, but that still left every other challenge a DM could think of unquantified, and stil left puzzle-traps unaccounted for. A published game can, obviously, never prescribe an appropriate general reward for puzzles because: 1-The difficulty of puzzles can't be abstractly generalized and 2-The moment you get more specific about a puzzle in the published rules, the players know the answer to the puzzle and it's not a puzzle.
(#4) X.P. is, Therefore, An Abstract Measure of The Amount of Problem-Solving A Given Player Has Done With A Given Character
It's not How Much Fighting Your Fighter Has Done or How Much Stealing Your Thief Has Done, or How Long Your Wizard Has Spent In Front Of Spellbooks, it's a reward for the player. And the reward essentially is: solve enough problems with this kind of character and eventually you earn the right to try to solve different kinds of problems with a slightly different (i.e. levelled-up) character.
Which leads to:
(#5) The Purpose of X.P. In The D&D Is To Create An Automatic Method By Which The Nature of the Problems Faced In The Game Will Change Over Time So The Game Won't Get Stale