Do major chess publishers use custom software for typesetting chess books? As I try to understand how one would publish a large chess book, it feels that chess gives some pretty custom requirements for an editor or publisher. I am pretty sure that chess authors use Chessbase or similar software for writing their books and then pass them off to editors who proofread and do all the final layout and typesetting for printed version. Since Chessbase does not support proper document management (export to RTF is as far as it goes), and layout software like InDesign don't support adding chess diagrams, do software publishers like Gambit, Everyman chess, Quality Chess, etc have their own chess aware custom publishing tools? Or do editors just use regular publishing software and then insert diagrams by hand to optimize for page space as they see fit, with plenty of room for manual errors (i.e. inserting diagrams that don't match the moves)?
This is less a question about chess, than a publishing question.  The answer is:

No.  The publishing industry largely uses publishing software (open source, and proprietary) to publish books, including chess books.  You've named one - InDesign.

Software such as Adobe's InDesign or Open Source software such as SILE or LaTex can lay out a book given the correct font.  Fonts can be used with Typesetting software to represent positions and pieces.
For example a common book font is Chess Mérida:

This font also includes black backgrounds and pieces on backgrounds so you can make images such as:

The font converts that from the text:
!&quot;&quot;&quot;&quot;&quot;&quot;&quot;&quot;#  
$tMvWlVmT%  
$OoOoOoOo%  
$ + + + +%  
$+ + + + %  
$ + + + +%  
$+ + + + %  
$pPpPpPpP%  
$RnBqKbNr%  
/(((((((()  

(Download the font here and cut that text into a Word Document and change the text to the Chess font and see for yourself)
You can find many chess fonts to use with your publishing software here.