As I try to understand how one would publish a large chess book, it feels that chess gives some pretty custom requirements for an editor or publisher. I am pretty sure that chess authors use Chessbase or similar software for writing their books and then pass them off to editors who proofread and do all the final layout and typesetting for printed version. Since Chessbase does not support proper document management (export to RTF is as far as it goes), and layout software like InDesign don't support adding chess diagrams, do software publishers like Gambit, Everyman chess, Quality Chess, etc have their own chess aware custom publishing tools? Or do editors just use regular publishing software and then insert diagrams by hand to optimize for page space as they see fit, with plenty of room for manual errors (i.e. inserting diagrams that don't match the moves)?
This is less a question about chess, than a publishing question. The answer is:
No. The publishing industry largely uses publishing software (open source, and proprietary) to publish books, including chess books. You've named one - InDesign.
Software such as Adobe's InDesign or Open Source software such as SILE or LaTex can lay out a book given the correct font. Fonts can be used with Typesetting software to represent positions and pieces.
For example a common book font is Chess Mérida:
This font also includes black backgrounds and pieces on backgrounds so you can make images such as:
The font converts that from the text:
!""""""""# $tMvWlVmT% $OoOoOoOo% $ + + + +% $+ + + + % $ + + + +% $+ + + + % $pPpPpPpP% $RnBqKbNr% /(((((((()
(Download the font here and cut that text into a Word Document and change the text to the Chess font and see for yourself)
You can find many chess fonts to use with your publishing software here.
I also had this question, and as a graphic designer, I was stunned to find no real solution was created for InDesign, and LaTeX was the only viable option to typeset a book or magazine.
So I took it upon myself to write a script to create scalable chess diagrams from FEN notation. I also created a font family for figurine chess notation. I know it's late for this. But no one else has created it, and I figure someone will find my solution helpful.
For more information and to download the free script and font, see my website.
All publishers have their own software and standards for publishing.
Are you self publishing and printing your own book? If not then you must ask whoever will print or publish the book what their requirements are.
When I was a small publisher looking at this I just used a chess font and arranged the diagrams myself. But that was several decades ago. Now there is fancier software to help book layout including chess specific ones.
I'm not a major publisher, but I have re-published some old books on chess problems electronically. I ended up using InDesign, but before that I used FrameMaker, Tex/LaTeX, ... and I could have done a decentish job with Word or WordPerfect once I had figured out what could and could not be done well.
and layout software like InDesign don't support adding chess diagrams, do software publishers like Gambit, Everyman chess, Quality Chess, etc have their own chess aware custom publishing tools?
It is almost irrelevant what InDesign and other tools support. Hot type setters did not have anything but their hands and their type, and they still managed to do some very good books. InDesign did (and probably still does) have so much fundamental support for type setting -- as does TeX (and so LaTeX) -- that you can make do without intrinsic hand-holding. You will need some external tools, such as character set converters, and diagram editors.
In my case, I had a fixed page format, and a design page size and font to work with. No scaling, no resizing, no user-specified fonts, font sizes or page sizes (i.e. no e-books like Kindle books). Diagrams were based on Partae Hastings or Linares diagram with some custom edits to avoid the original and excessively wide border type, allowing diagram edges to snug up against column edges with no white space. Both families (if I recall) had special diagram sets, including rotated pieces.
Then, each diagram (in text form using the Partae character mapping) was imported and assigned the font. Paragraph formatting was adjusted to set lines solid, and not allow any adjustments or hyphenation. The first line typically needed to be adjusted for vertical spacing, but first line was always diagram border ("cuuuuuuuuC"), and so very easy to search-and-replace paragraph format, and the last line could usually need less space after, too.
The result works well with module-based book design, as it is rather static. (I've not done anything in this line for perhaps 15 years, but you can find whatever I did at http://www.anders.thulin.name/old/oldstuff/ )
A similar approach would work for chess moves, but you would have to decide how you wanted the moves displayed before hand (but that is typically a house style, and doesn't change much). Changing your mind half-way through a book would not be a trivial thing, though. (I mean, entering a game, and then deciding short algrebraic in two columns, changing to full algebraic, then trying figurine in-line, and ending up with English descriptive would not work without considerable support: in InDesign it would probably be technically possible to write a plugin for it, ... but very few would need it, as that kind of indecisiveness belongs elsewhere than in a publishing shop. IMNSHO.)
For a very well done chess book by a major publisher look at C. H. O'd. Alexander's Fischer v. Spassky Reykjavik 1972 (Penguin Books, 1972), designed by Derek Birdsall.