regress/regress.pl.1

   1 .\"     $Id: regress.pl.1,v 1.2 2017/03/05 19:57:39 schwarze Exp $
   2 .\"
   3 .\" Copyright (c) 2017 Ingo Schwarze <schwarze@openbsd.org>
   4 .\"
   5 .\" Permission to use, copy, modify, and distribute this software for any
   6 .\" purpose with or without fee is hereby granted, provided that the above
   7 .\" copyright notice and this permission notice appear in all copies.
   8 .\"
   9 .\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  10 .\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  11 .\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
  12 .\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  13 .\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  14 .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
  15 .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  16 .\"
  17 .Dd $Mdocdate: March 5 2017 $
  18 .Dt REGRESS.PL 1
  19 .Os
  20 .Sh NAME
  21 .Nm regress.pl
  22 .Nd portable steering script for mandoc regression tests
  23 .Sh SYNOPSIS
  24 .Nm ./regress.pl
  25 .Oo
  26 .Ar directory Ns Op Pf : Ar test
  27 .Op Ar modifier ...
  28 .Oc
  29 .Sh DESCRIPTION
  30 The
  31 .Nm
  32 steering script allows running the
  33 .Xr mandoc 1
  34 regression suite on arbitrary operating systems,
  35 even though the suite was designed for OpenBSD only.
  36 .Pp
  37 When run without an argument,
  38 .Nm
  39 runs the complete regression suite.
  40 .Pp
  41 When run with one argument, that argument can be:
  42 .Bl -enum
  43 .It
  44 A single dot to run the complete suite.
  45 .It
  46 One of the top level directories, for example
  47 .Pa mdoc ,
  48 to run the test suite for a complete language or feature group.
  49 .It
  50 A subdirectory, for example
  51 .Pa man/IP ,
  52 to run the tests for a specific macro or an individual feature.
  53 .It
  54 A subdirectory with a test name appended with a colon, for example
  55 .Pa char/unicode : Ns Pa named ,
  56 to run the tests for one particular input file.
  57 .El
  58 .Pp
  59 Any additional arguments modify the way the tests are run.
  60 The default is
  61 .Cm all .
  62 The following modifiers are available:
  63 .Bl -tag -width markdown
  64 .It Cm all
  65 Run all kinds of subtests.
  66 This implies all other modifiers except
  67 .Cm verbose
  68 and
  69 .Cm clean .
  70 .It Cm ascii
  71 Run subtests for
  72 .Fl T Cm ascii
  73 output mode.
  74 .It Cm clean
  75 Remove all output files created by running the tests.
  76 .It Cm html
  77 Run subtests for
  78 .Fl T Cm html
  79 output mode.
  80 .It Cm lint
  81 Run subtests for
  82 .Fl T Cm lint
  83 warning and error output.
  84 .It Cm man
  85 Run subtests for
  86 .Fl T Cm man
  87 output mode.
  88 .It Cm markdown
  89 Run subtests for
  90 .Fl T Cm markdown
  91 output mode.
  92 .It Cm utf8
  93 Run subtests for
  94 .Fl T Cm utf8
  95 output mode.
  96 .It Cm verbose
  97 Display approximate indications of what is being done.
  98 .El
  99 .Pp
 100 The amount of summary lines shown can be modified by giving an
 101 argument consisting of a single digit:
 102 .Bl -tag -width verbose
 103 .It Cm 3
 104 Show all summary lines for all directories entered.
 105 Even without
 106 .Cm verbose ,
 107 this generates more than hundred lines of output when running the
 108 complete regression suite.
 109 .It Cm 2
 110 This is the default.
 111 It shows the summary lines for the
 112 .Ar directory
 113 given on the command line and its immediate children.
 114 Except for
 115 .Pa mdoc ,
 116 the output usually fits on one screen.
 117 .It Cm 1
 118 Only show a single summary line for the whole run.
 119 .It Cm 0
 120 Do not show any summary lines.
 121 No output means success.
 122 Success or failure can also be seen from the exit status.
 123 .El
 124 .Pp
 125 All failed tests are always reported, even when the
 126 .Cm 0
 127 modifier is given.
 128 .Sh EXIT STATUS
 129 .Ex -std
 130 .Sh EXAMPLES
 131 The recommended invocation for casual users:
 132 .Pp
 133 .Dl ./regress.pl
 134 .Pp
 135 Maximum output:
 136 .Pp
 137 .Dl ./regress.pl \&. verbose
 138 .Pp
 139 Complete check, but keep the tree clean:
 140 .Pp
 141 .Dl ./regress.pl \&. all clean
 142 .Pp
 143 Test all of
 144 .Pa mdoc ,
 145 but don't print the usual 65 lines of output:
 146 .Pp
 147 .Dl ./regress.pl mdoc 1
 148 .Pp
 149 Investigate a specific failure:
 150 .Pp
 151 .Dl ./regress.pl mdoc/Bd:broken man verbose
 152 .Sh HISTORY
 153 The
 154 .Nm
 155 script appeared in release 1.14.1 of the portable
 156 .Sy mandoc
 157 distribution.
 158 .Sh AUTHORS
 159 .An Ingo Schwarze Aq Mt schwarze@openbsd.org
 160 .Sh CAVEATS
 161 This script is not optimized for elegance.
 162 Regression suites for other software should not copy the design.
 163 .Pp
 164 The problem it solves is that the
 165 .Sy mandoc
 166 regression suite is tightly integrated into the regression
 167 testing system of the OpenBSD base system, which requires
 168 both OpenBSD
 169 .Xr make 1 ,
 170 working neither with POSIX make nor with GNU make, and which
 171 also requires the OpenBSD-specific Makefile fragments in
 172 .Pa /usr/share/mk .
 173 The workaround of parsing the Makefiles by hand and constructing
 174 the required command lines by hand is unavoidably messy; it's
 175 the classic no-no of parsing a language with an ad-hoc incomplete
 176 parser.
 177 But the problem of providing this regression suite for other
 178 operating systems stood unsolved for many years, and no cleaner
 179 solution was found that could be implemented with reasonable effort.
 180 So maybe this is better than nothing.
 181 .Pp
 182 The top-level Makefiles for running this regression suite on
 183 OpenBSD are not included in the portable distribution.
 184 They are too OpenBSD-specific to be useful elsewhere,
 185 and on OpenBSD itself, the suite ought be run natively from
 186 .Pa /usr/src/regress/usr.bin/mandoc
 187 and not from the portable distribution.
 188 .Pp
 189 The
 190 .Pa db
 191 subdirectory of the regression suite is not included.
 192 It uses a Makefile structure that differs vastly from the
 193 rest of the suite.