Split tagging into a validation part including prioritization

in tag.{h,c} and {mdoc,man}_validate.c
and into a formatting part including command line argument checking
in term_tag.{h,c}, html.c, and {mdoc|man}_{term|html}.c.

Immediate functional benefits include:
* Improved prioritization of automatic tags for .Em and .Sy.
* Avoiding bogus automatic tags when .Em, .Fn, or .Sy are explicitly tagged.
* Explicit tagging of .Er and .Fl now works in HTML output.
* Automatic tagging of .IP and .TP now works in HTML output.
But mainly, this patch provides clean earth to build further improvements on.

Technical changes:
* Main program: Write a tag file for ASCII and UTF-8 output only.
* All formatters: There is no more need to delay writing the tags.
* mdoc(7)+man(7) formatters: No more need for elaborate syntax tree inspection.
* HTML formatter: If available, use the "string" attribute as the tag.
* HTML formatter: New function to write permalinks, to reduce code duplication.

Style cleanup in the vicinity while here:
* mdoc(7) terminal formatter: To set up bold font for children,
defer to termp_bold_pre() rather than calling term_fontpush() manually.
* mdoc(7) terminal formatter: Garbage collect some duplicate functions.
* mdoc(7) HTML formatter: Unify <code> handling, delete redundant functions.
* Where possible, use switch statements rather than if cascades.
* Get rid of some more Yoda notation.

The necessity for such changes was first discussed with kn@, but i didn't
bother him with a request to review the resulting -673/+782 line patch.

18 files changed, 176 insertions, 19 deletions

diff --git a/regress/man/IP/Makefile b/regress/man/IP/Makefile
index 70094e5f..8d607c62 100644
--- a/regress/man/IP/Makefile
+++ b/regress/man/IP/Makefile
@@ -1,8 +1,9 @@
-# $OpenBSD: Makefile,v 1.10 2020/02/27 01:25:58 schwarze Exp $
+# $OpenBSD: Makefile,v 1.11 2020/03/13 00:31:05 schwarze Exp $
 
-REGRESS_TARGETS = bullet empty literal longhead manyargs spacing vert width
+REGRESS_TARGETS = bullet empty literal longhead manyargs spacing tag vert width
+TAG_TARGETS	= empty tag
 UTF8_TARGETS	= bullet
 LINT_TARGETS	= empty
-HTML_TARGETS	= bullet literal
+HTML_TARGETS	= bullet empty literal tag
 
 .include <bsd.regress.mk>
diff --git a/regress/man/IP/empty.in b/regress/man/IP/empty.in
index 21c42983..bb814870 100644
--- a/regress/man/IP/empty.in
+++ b/regress/man/IP/empty.in
@@ -1,8 +1,9 @@
-.\" $OpenBSD: empty.in,v 1.2 2017/07/04 14:53:23 schwarze Exp $
+.\" $OpenBSD: empty.in,v 1.3 2020/03/13 00:31:05 schwarze Exp $
 .TH IP-EMPTY 1 "July 17, 2012"
 .SH NAME
 IP-empty \- empty indented paragraphs
 .SH DESCRIPTION
+BEGINTEST
 regular
 text
 .IP
@@ -25,3 +26,4 @@ text
 .RE
 regular
 text
+ENDTEST
diff --git a/regress/man/IP/empty.out_ascii b/regress/man/IP/empty.out_ascii
index 24ab17fa..b17190ab 100644
--- a/regress/man/IP/empty.out_ascii
+++ b/regress/man/IP/empty.out_ascii
@@ -6,7 +6,7 @@ NNAAMMEE
        IP-empty - empty indented paragraphs
 
 DDEESSCCRRIIPPTTIIOONN
-       regular text
+       BEGINTEST regular text
 
               indented text
 
@@ -19,7 +19,7 @@ DDEESSCCRRIIPPTTIIOONN
        Empty IP is deleted, RS does not cause additional spacing:
 
               tag    indented text
-       regular text
+       regular text ENDTEST
 
 
 
diff --git a/regress/man/IP/empty.out_html b/regress/man/IP/empty.out_html
new file mode 100644
index 00000000..f2498dba
--- /dev/null
+++ b/regress/man/IP/empty.out_html
@@ -0,0 +1,18 @@
+<dl class="Bl-tag">
+  <dt></dt>
+  <dd>indented text</dd>
+</dl>
+<p class="Pp">Empty IP is deleted:</p>
+<dl class="Bl-tag">
+  <dt id="tag1"><a class="permalink" href="#tag1">tag1</a></dt>
+  <dd></dd>
+  <dt id="tag2"><a class="permalink" href="#tag2">tag2</a></dt>
+  <dd>indented text</dd>
+</dl>
+<p class="Pp">Empty IP is deleted, RS does not cause additional spacing:</p>
+<div class="Bd-indent">
+<dl class="Bl-tag">
+  <dt id="tag"><a class="permalink" href="#tag">tag</a></dt>
+  <dd>indented text</dd>
+</dl>
+</div>
diff --git a/regress/man/IP/empty.out_lint b/regress/man/IP/empty.out_lint
index 6c666c88..3514b94f 100644
--- a/regress/man/IP/empty.out_lint
+++ b/regress/man/IP/empty.out_lint
@@ -1,2 +1,2 @@
-mandoc: empty.in:13:2: WARNING: skipping paragraph macro: IP empty
-mandoc: empty.in:20:2: WARNING: skipping paragraph macro: IP empty
+mandoc: empty.in:14:2: WARNING: skipping paragraph macro: IP empty
+mandoc: empty.in:21:2: WARNING: skipping paragraph macro: IP empty
diff --git a/regress/man/IP/empty.out_tag b/regress/man/IP/empty.out_tag
new file mode 100644
index 00000000..f35f1120
--- /dev/null
+++ b/regress/man/IP/empty.out_tag
@@ -0,0 +1,3 @@
+tag1 15
+tag2 17
+tag 21
diff --git a/regress/man/IP/literal.out_html b/regress/man/IP/literal.out_html
index b61fc843..3b9cc429 100644
--- a/regress/man/IP/literal.out_html
+++ b/regress/man/IP/literal.out_html
@@ -1,5 +1,5 @@
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag"><a class="permalink" href="#tag">tag</a></dt>
   <dd>indented regular text</dd>
 </dl>
 <p class="Pp">new regular paragraph</p>
@@ -8,7 +8,7 @@ literal
 text
 </pre>
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag_2"><a class="permalink" href="#tag_2">tag</a></dt>
   <dd>
     <pre>
 indented
@@ -32,7 +32,7 @@ literal
 text
 </pre>
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag_3"><a class="permalink" href="#tag_3">tag</a></dt>
   <dd>
     <pre>
 indented
@@ -48,7 +48,7 @@ text
   out of indented paragraph</a></h2>
 <p class="Pp">regular text</p>
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag_4"><a class="permalink" href="#tag_4">tag</a></dt>
   <dd>indented regular text
     <pre>
 indented
diff --git a/regress/man/IP/tag.in b/regress/man/IP/tag.in
new file mode 100644
index 00000000..038fa969
--- /dev/null
+++ b/regress/man/IP/tag.in
@@ -0,0 +1,18 @@
+.\" $OpenBSD: tag.in,v 1.1 2020/03/13 00:31:05 schwarze Exp $
+.TH IP-TAG 1 "March 10, 2020"
+.SH NAME
+IP-tag \- automatic tagging of indented blocks
+.SH DESCRIPTION
+BEGINTEST
+initial
+text
+.IP " strong" 10n
+text
+.IP "-strong"
+text
+.IP "\&\fI \-weak\fP"
+text
+.IP " strong"
+text
+.PP
+ENDTEST
diff --git a/regress/man/IP/tag.out_ascii b/regress/man/IP/tag.out_ascii
new file mode 100644
index 00000000..afd4f587
--- /dev/null
+++ b/regress/man/IP/tag.out_ascii
@@ -0,0 +1,23 @@
+IP-TAG(1)                   General Commands Manual                  IP-TAG(1)
+
+
+
+NNAAMMEE
+       IP-tag - automatic tagging of indented blocks
+
+DDEESSCCRRIIPPTTIIOONN
+       BEGINTEST initial text
+
+        strong   text
+
+       -strong   text
+
+        _-_w_e_a_k    text
+
+        strong   text
+
+       ENDTEST
+
+
+
+OpenBSD                         March 10, 2020                       IP-TAG(1)
diff --git a/regress/man/IP/tag.out_html b/regress/man/IP/tag.out_html
new file mode 100644
index 00000000..4d25b12b
--- /dev/null
+++ b/regress/man/IP/tag.out_html
@@ -0,0 +1,10 @@
+<dl class="Bl-tag">
+  <dt> strong</dt>
+  <dd>text</dd>
+  <dt id="strong"><a class="permalink" href="#strong">-strong</a></dt>
+  <dd>text</dd>
+  <dt id="weak"><a class="permalink" href="#weak"><i> -weak</i></a></dt>
+  <dd>text</dd>
+  <dt> strong</dt>
+  <dd>text</dd>
+</dl>
diff --git a/regress/man/IP/tag.out_tag b/regress/man/IP/tag.out_tag
new file mode 100644
index 00000000..5be038db
--- /dev/null
+++ b/regress/man/IP/tag.out_tag
@@ -0,0 +1,2 @@
+strong 13
+weak 15
diff --git a/regress/man/TP/Makefile b/regress/man/TP/Makefile
index ca2475b0..9dd2fde7 100644
--- a/regress/man/TP/Makefile
+++ b/regress/man/TP/Makefile
@@ -1,9 +1,10 @@
-# $OpenBSD: Makefile,v 1.16 2020/02/27 01:25:58 schwarze Exp $
+# $OpenBSD: Makefile,v 1.17 2020/03/13 00:31:05 schwarze Exp $
 
 REGRESS_TARGETS	 = badarg broken double eof fill indent literal longhead
-REGRESS_TARGETS	+= macrotag manyargs sameline spacing vert width
+REGRESS_TARGETS	+= macrotag manyargs sameline spacing tag vert width
+TAG_TARGETS	 = tag
 LINT_TARGETS	 = broken double eof
-HTML_TARGETS	 = literal vert
+HTML_TARGETS	 = literal tag vert
 
 # groff-1.22.3 defects:
 # - If .TP precedes .RE, the latter does not properly reset indentation.
diff --git a/regress/man/TP/literal.out_html b/regress/man/TP/literal.out_html
index 7f5b1c1f..8b818ed8 100644
--- a/regress/man/TP/literal.out_html
+++ b/regress/man/TP/literal.out_html
@@ -1,5 +1,5 @@
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag"><a class="permalink" href="#tag">tag</a></dt>
   <dd>regular indented text</dd>
 </dl>
 <p class="Pp">regular paragraph</p>
@@ -8,7 +8,7 @@ literal
 text
 </pre>
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag_2"><a class="permalink" href="#tag_2">tag</a></dt>
   <dd>
     <pre>
 indented
diff --git a/regress/man/TP/tag.in b/regress/man/TP/tag.in
new file mode 100644
index 00000000..34d1e151
--- /dev/null
+++ b/regress/man/TP/tag.in
@@ -0,0 +1,31 @@
+.\" $OpenBSD: tag.in,v 1.1 2020/03/13 00:31:05 schwarze Exp $
+.TH IP-TAG 1 "March 10, 2020"
+.SH NAME
+IP-tag \- automatic tagging of indented blocks
+.SH DESCRIPTION
+BEGINTEST
+initial
+text
+.TP 10n
+.I " plain"
+text
+.TP
+plain
+text
+.TP
+.I "plain "
+text
+.TP
+\& strong
+text
+.TP
+.B -strong
+text
+.TP
+\&\fI \-weak\fP
+text
+.TP
+.B "strong "
+text
+.PP
+ENDTEST
diff --git a/regress/man/TP/tag.out_ascii b/regress/man/TP/tag.out_ascii
new file mode 100644
index 00000000..4da2efa0
--- /dev/null
+++ b/regress/man/TP/tag.out_ascii
@@ -0,0 +1,29 @@
+IP-TAG(1)                   General Commands Manual                  IP-TAG(1)
+
+
+
+NNAAMMEE
+       IP-tag - automatic tagging of indented blocks
+
+DDEESSCCRRIIPPTTIIOONN
+       BEGINTEST initial text
+
+        _p_l_a_i_n    text
+
+       plain     text
+
+       _p_l_a_i_n     text
+
+        strong   text
+
+       --ssttrroonngg   text
+
+        _-_w_e_a_k    text
+
+       ssttrroonngg    text
+
+       ENDTEST
+
+
+
+OpenBSD                         March 10, 2020                       IP-TAG(1)
diff --git a/regress/man/TP/tag.out_html b/regress/man/TP/tag.out_html
new file mode 100644
index 00000000..3fbbe41d
--- /dev/null
+++ b/regress/man/TP/tag.out_html
@@ -0,0 +1,16 @@
+<dl class="Bl-tag">
+  <dt><i> plain</i></dt>
+  <dd>text</dd>
+  <dt id="plain"><a class="permalink" href="#plain">plain</a></dt>
+  <dd>text</dd>
+  <dt><i>plain </i></dt>
+  <dd>text</dd>
+  <dt> strong</dt>
+  <dd>text</dd>
+  <dt id="strong"><a class="permalink" href="#strong"><b>-strong</b></a></dt>
+  <dd>text</dd>
+  <dt id="weak"><a class="permalink" href="#weak"><i> -weak</i></a></dt>
+  <dd>text</dd>
+  <dt><b>strong </b></dt>
+  <dd>text</dd>
+</dl>
diff --git a/regress/man/TP/tag.out_tag b/regress/man/TP/tag.out_tag
new file mode 100644
index 00000000..88f6cff8
--- /dev/null
+++ b/regress/man/TP/tag.out_tag
@@ -0,0 +1,3 @@
+plain 13
+strong 19
+weak 21
diff --git a/regress/man/TP/vert.out_html b/regress/man/TP/vert.out_html
index 7301819f..1124ed33 100644
--- a/regress/man/TP/vert.out_html
+++ b/regress/man/TP/vert.out_html
@@ -2,8 +2,8 @@
 <section class="Sh">
 <h1 class="Sh" id="DESCRIPTION"><a class="permalink" href="#DESCRIPTION">DESCRIPTION</a></h1>
 <dl class="Bl-tag">
-  <dt>tag</dt>
+  <dt id="tag"><a class="permalink" href="#tag">tag</a></dt>
   <dd>text</dd>
-  <dt>tag</dt>
+  <dt id="tag_2"><a class="permalink" href="#tag_2">tag</a></dt>
   <dd>text</dd>
 </dl>