I have file Rybka.exe and when I use it for the first moves, it always goes Nf3 or Nc3 only (even at strength 1000 and depth 1). I assume that is because it doesn't have opening book assigned (?) to it. How does one connect opening books in .ctg (or other?) formats to the chess engine? I do not want to get some GUI, but rather understand how it is done.
I don't have Rybka 4 but I do have the free version (Rybka 2.3.2). Rybka 2.3.2 doesn't support engine book. To verify this, open the engine in command, then type "uci". Look at how I did it.
Look the UCI options, we have one for multi-pv and others. But there is nothing for an opening book. If the engine doesn't support an opening book, you have to rely on a GUI to do it, for example, Fritz.
On the other hand, if you do the same thing to Stockfish, you'll see a Book options where you can specify path to your book be used by the engine.
It sounds like you're using the standard Chessbase interface when you mentioned a .ctg file.
The Chessbase interface is used for many engines, so this title may work: How To Make Fritz Chess Software Play a Specific Chess Opening
Try this: http://www.chesscentral.com/Chess_Opening_Book_a/137.htm