Note that there’s still quite a bit of difference between wargame rules and other board games, at least the lighter board games. It’s been a while since I’ve played wargames, but at the time I believe the GMT Games were the gold standard for rules, putting out consistently clear rulebooks.
Based on the rulebooks I’ve seen, the big tension in rulebooks for complex games is the trade-off between getting up to speed to play your first game versus being able to look up rules and answer questions during a game. For example, Simmons Games (e.g., Napoleon’s Triumph) are known for their tight, concise rules: if you have a rules question, you can quickly find an unambiguous answer. That doesn’t mean you understand what the heck you’re doing when you sit down to play your first couple games.
Other games organize the rules in roughly chronological order based on the sequence of play so that players can get the sense how a procedure or turn might go. But doesn’t mean players can always find the answer to their question. For example, if the game has system(s) that are relevant to multiple phases of the turn, those rules might be scattered across different parts of the rulebook.
I still have my copy of Carrier, which is notorious for being hard to learn. The game has several subsystems, one or two of which are kinda complicated (the hidden information system with intel level, location level, and searched level comes to mind). The game used programmed scenarios (i.e., read some rules, play a stripped down version of the game to learn those rules) to teach each system before – hopefully – being able to put it all together. I think a lot of people either got lost on the more complicated systems, because they couldn’t wrap their heads around the system without the context of the overall game; or they could go through the simple programmed scenarios but the trees made it hard for them to see the forest.
One strategy that a number of games have adopted (I know I have one non-wargame that does this. Fury of Dracula? I know Root does this.) is to have two books: one is a learn-to-play book, and the other is a reference rulebook to look up rules. You’re probably familiar with how to write a reference rulebook. My advice here is that, if players have a question, all of the information needed to answer that question should be in one place, and players should not need to remember or assemble pieces of information in separate parts of the rulebook. An index so players can locate rules quickly and a glossary of technical terms are also useful.
Part of the learn-to-play book is teaching players how to follow the rules, like how to resolve an attack or what actions players are or are not allowed to do at a given time. (For this, the standard tactic seems to be illustrating rules or procedures using examples.)
But knowing the rules is not the same as understanding how to play, and while old-school wargames were satisfied with defining the former and leaving the players to figure out the latter, more people these days seem to expect the company to lower the barrier on the latter. So the organization and maybe some of the contents of the learn-to-play book should help new players onramp to their first game. For example, some GMT Games give an extended example of play of multiple turns to discuss how the whole turn fits together, maybe why a player did a certain thing. On one of the player aids in Scythe suggests goals for players to aim for on their first several turns so they can get a basic economy going. Other games give players general strategic advice or point out aspects of the game they should be watch out for. That said, remember that for many players, part of the fun of a game is learning how to play well, so don’t overdo it. The goal should be avoiding players being lost and unsure what to do then quitting in frustration.
The organization of the learn-to-play book should reflect what is it about your game makes it hard to learn. The Carrier rulebook was probably organized in a right way, because a huge stumbling block is understanding the individual systems. Where that rulebook had problems is probably not telling players what something was for (what does location level represent?) or how the systems fit together. When people teach games face to face, a common approach is to go top-down: give an overview of the game and player’s objectives, then fill in details about sequence of play or systems that players can connect to the overall structure you’ve already established.
So when you’ve taught new players Divine Right, what is the hardest thing for them to learn? Gotta run. Hope that helps.