I'm not arguing for an expanded playoff, but....
1973 is the counter-argument to this. Here were the top teams heading into the bowl season:
Associated Press
_1. Alabama (11-0-0)
_2. Oklahoma (10-0-1)
_3. Notre Dame (10-0-0)
_4. Ohio State (9-0-1)
_5. Michigan (10-0-1)
_6. Penn State (11-0-0)
_7. Southern Cal (9-1-1)
_8. Texas (8-2-0)
_9. UCLA (9-2-0)
10. Arizona State (10-1-0)
11. Texas Tech (10-1-0)
12. Nebraska (8-2-1)
Clearly, the top 6 teams in 1973 deserved to get into a hypothetical playoff, with Southern Cal making a strong case as the Pac-8 representative, and with mid-majors Arizona State (then in the WAC) and Texas Tech (SWC) also having an argument for inclusion. Miami of Ohio finished the regular season at 10-0-0 and #15 in the pre-bowl poll, and of course the MAC Defenders of the Faith would have been crowing for the little guy to get a shot.
2007 was a similar cluster, with no single team really standing out from the pack:
BCS Rankings
_1. Ohio State (11-1)
_2. Louisiana State (11-2)
_3. Virginia Tech (11-2)
_4. Oklahoma (11-2)
_5. Georgia (10-2)
_6. Missouri (10-2)
_7. Southern Cal (10-2)
_8. Kansas (11-1)
_9. West Virginia (10-2)
10. Hawaii (12-0)
Even in 2014, a pair of Power5 one-loss teams (Baylor, TCU) were left out of the playoffs. Not that I'm upset about that....
So while four teams is a really good number in most years, you will always have seasons where five or six or even more teams would be deserving. But life isn't fair, and I'd personally vote for under-inclusion rather than over-inclusion. Leave the playoffs at four teams and let #5 cry like a baby for the next year.