What are some openings that normally a player with less or no understanding of opening theory would suffer agonizingly? Or perhaps, openings that are extremely confusing to play against?
To me, the Latvian Gambit/French Defense is one of the most confusing openings.
Depends if you have the white pieces or the black pieces;
white: 1. Nf3 - Versatile, deceptive, unassuming. Typically, blacks first move dictates the style and nature of the game. However, with 1. Nf3, white has countless systems at his disposal depending on how black responds.
black: against e4: 1... Nf6 - Alekhine's defense. From move number one, black is directly attacking white's center. Many novice players will over-extend themselves and start pushing their pieces hastily. Alekhine's defense is a tricky, formidable response to 1. e4, as Bobby Fischer showed us in 1972 against Boris Spassky.
against d4: 1...f5 - Dutch defense. The Dutch defense, whose famous practitioners include Aleksander Alekhine, Mikhail Botvinnik, and Hikaru Nakamura, is a dangerous response to 1. d4. Black is taking dead aim at white's kingside and, specifically, the e4 square.
I quite like the polish 1.b4 opening. The oddness catches some newer players off guard and it's not unusual for players to leave their rook wide open after going for your exposed b4 pawn.
It's actually quite a fun opening to use generally. You're sacrificing the middle for early development on the flank, I think its weakness is it gives away your intentions early on in the game.
Then there's always the fried liver attack, I think someone mentioned it above; It's straight on the attack and if you've never played against it the pressure is agonizing.
http://en.wikipedia.org/wiki/Sokolsky_Opening http://en.wikipedia.org/wiki/Two_Knights_Defense,_Fried_Liver_Attack
Openings that involve opening up the f-file can turn dangerous very quickly. The King's Gambit may cause the inexperienced player some regret after a check on h4 or after ...g4 drives away a knight on f3, perhaps leaving the d-pawn unprotected. For Black, the Latvian Gambit (or Greco Counter Gambit; in no way to be confused with the French Defense) can similarly lead to dangerous checks on h5, or to forking checks following Qe2 - b5, attacking the King as well as squares on b7 (pawn) and f5 (Bishop?).
From my point of view, hypermodern openings are good to catch the unaware players as most such players tend to overextend themselves and in the process, opening whole lot of weaknesses in their position for you to strike.
Some examples of such openings that I found useful in my practice for the purpose you are asking:
Openings for White: Reti, Larsen
Openings for Black: Pirc, King's Indian, Queen's Indian
If one has no understanding of opening theory then performance will suffer dramatically no matter what opening is played. There are very sharp openings out there such as Two Knights Defense where one has to know exact sequence of moves for both sides:
[StartPly "19"]
[FEN ""]
1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6
here white can play 4. Ng5, 4. Nc3, 4. d3 and 4. d4
Also Ruy Lopez, Schliemann Defense is very sharp for black. Black has to know it with the tips of the fingers or lose very quickly
[StartPly "19"]
[FEN ""]
1. e4 e5 2. Nf3 Nc6 3. Bb5 f5
Ultra-sharp Polugaevsky variation is yet another example of how important the tempo is in the opening:
[StartPly "19"]
[FEN ""]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nd4 Nf6 5. Nc3 a6 6. Bg5 b5
Closed opening are known to be slow but there are also very sharp lines like Botvinnik Variation. It is known (on GM level!) when seeing one move further in this line was enough to secure a win.
[StartPly "19"]
[FEN ""]
1. d4 d5 2. c4 c6 3. Nf3 Nf6 4. Nc3 e6 5. Bg5 dxc4 6. e4 b5 7. e5 h6 8. Bh4 g5 9. Ng5 hxg5 10. Bg5 Nbd7
The Latvian Gambit (or King's Gambit in reverse) may be confusing, but it's also somewhat unsound on principle since White already has the initiative and you're giving up the pawn from a position of weakness. The French Defense is a lot sounder and can be very strong in the hands of someone who knows what they are doing. As someone else has mentioned, hypermodern openings can be confusing if you are not familiar with chess theory concerning controlling the center, but they are still not as strong as classical openings immediately contesting center control. If you know what you are doing, you can reach favorable positions through transposition with hypermodern openings.
