What is a chess problem?
The wikipedia definition is pretty good:
A chess problem, also called a chess composition, is a puzzle set by
the composer using chess pieces on a chess board, which presents the
solver with a particular task.
In constrast, problemists use the term “tactical puzzle" to describe the kinds of realistic positional chess challenges that you can find in chess.com, lichess.com etc. Some top chess coaches argue that endgame studies & directmates are also excellent to improve over the board skills. Endgame studies in particular are a cross-over, relevant to the game but also can be regarded as chess artistry of the highest order.
However, I want to explain why chess problems can be fascinating in their own terms. From this perspective, any relevance to over-the-board play is only a happy side-effect. This reply complements other excellent answers here, which give more of a taxonomy of chess problems.
Swimming races and high-diving are both Olympic sports conducted in a swimming pool. Similarly. over the board play, problem composition and problem solving are all FIDE sports conducted over a chess board (and in all three cases, FM, IM & GM titles are awarded).
The Queen's Wombat
Probably the best known chess problem today, is alas the one shown in episode 6 of The Queen's Gambit" Netflix TV series. I loved the original book by Walter R. Tevis, and thought the TV series succeeding in creating something even better. In almost every dimension, the series developers hit the ball out of the park: casting, acting, sets, music, direction, lighting, and of course the wonderful chess itself. Hurray! Most importantly, they got the chess games right.
However there was a problem scene, and they got it wrong, in my opinion. Problemists didn't expect to be centre-stage in this series, or even mentioned: obviously our hobby is a minor interest. But the chess advisors (Pandolfini & Kasparov) are swimmers not divers. They don't understand what problems are all about. Beth quickly solves a bland mate in 3. Beth’s reaction is in character: she is a "Spike" in Magic the Gathering lingo: she just wants to win. But it’s jarring that the character Wexler (a "Johnny" in MtG parlance) says "this is my favourite". While this is reasonable for the apparently stupendous problem described in the book, the one in the TV show couldn't be anyone’s favourite. In the Netflix scene, Wexler is also being talked down to, talked over, has unsexy haircut etc.
If chess problems are irrelevant, then so are novels and TV series.
So what are problemists trying to do?
The wikipedia definition goes on to say:
A chess problem fundamentally differs from over-the-board play in that
the latter involves a struggle between black and white, whereas the
former involves a competition between the composer and the solver.
The word "competition" does not really capture the dynamic, which is in many ways co-operative, and more like the relationship between writer and reader.
As a composer, I am throwing a ball to the solver. If I manage to throw the ball fast, or high, it may be harder for the solver to catch, and it's more satisfying for both of us when they do. However solving difficulty is not the primary objective.
As a composer, I try to crystalize a vague idea into a coherent form, and then interest a solver into looking at it. A composer's relationship with the solver is synergistic and most of us play both roles.
Composers struggle with the recalcitrant chess pieces, which wilfully refuse to be herded. Why do we persist? Why don’t just make easier problems, rather than pushing these poor pieces to their limits? It’s not that all composers are competitive-minded (though some are) the point is that a problem must be essentially original. If I make a problem and it turns out to have been done before by someone else, then they have priority - all my effort is wasted!
Being anticipated by another composer is the worst feeling in the chess problem world. If my problem is merely found to not work properly (e.g. there’s a second solution that I didn’t spot) then that’s bad but usually it’s fixable in some way, sometimes it even becomes even better than the original version in other ways. But with anticipation there is nothing that can be done. To have built on another composer's ideas is great: that's often how things move forwards - but there must be original content in some way.
Composers are like songbirds in a wood listening to the other birds and chirping out new variants. And we can still hear their songs of deceased composers even 100 years ago; in a curious way they are still alive to us creatively.
This absolute requirement for originality is is what drives us all outwards to search for new fertile design space, into new kinds of stipulation (co-operative play, play to lose, even beyond the regular bounds of chess rules into fairy chess). And it pushes us to try to create the most elaborate and difficult houses of cards, which require extreme ingenuity, strategy and patience.
Perhaps the best way to be distinctive is through artistry. It is said that: "Style is the feather in the arrow, not the feather in the cap." Composers try to show grace like a high-diver: to demonstrate what mastery we have, if we are occasionally able to plunge from high into the water with hardly a ripple.