Ninja Gaiden and Marble Madness are certainly worthy canidates. Just try and beat Ninja Gaiden without save states on an emulator or other cheating means...that end boss is pretty much impossible even with infinite continues.
Fester's Quest on NES was pretty tough
Dragon's Lair on NES is bad, tough just to get past the first screen
Silver Surfer on NES
Teenage Mutant Ninja Turtles I (Not the awesome arcade game)
Probably the toughest game I've ever played has been "No Escape" on SNES. It was based off a movie with the same game, and it's ridiculous. I can get through a level, maybe two of it and thats it. I've found no strategies online, it's just evil.
Heres a Youtube video of it. This dude got farther than I did, but it's just mind numbingly evil and not fun: [ame="http://www.youtube.com/watch?v=yAKo4WaFD2U"]No Escape Review - YouTube[/ame]