regress/regress.pl.1

   1 .\"     $Id: regress.pl.1,v 1.1 2017/02/08 03:02:13 schwarze Exp $
   2 .\"
   3 .\" Copyright (c) 2017 Ingo Schwarze <schwarze@openbsd.org>
   4 .\"
   5 .\" Permission to use, copy, modify, and distribute this software for any
   6 .\" purpose with or without fee is hereby granted, provided that the above
   7 .\" copyright notice and this permission notice appear in all copies.
   8 .\"
   9 .\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  10 .\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  11 .\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
  12 .\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  13 .\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  14 .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
  15 .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  16 .\"
  17 .Dd $Mdocdate: February 8 2017 $
  18 .Dt REGRESS.PL 1
  19 .Os
  20 .Sh NAME
  21 .Nm regress.pl
  22 .Nd portable steering script for mandoc regression tests
  23 .Sh SYNOPSIS
  24 .Nm ./regress.pl
  25 .Oo
  26 .Ar directory Ns Op Pf : Ar test
  27 .Op Ar modifier ...
  28 .Oc
  29 .Sh DESCRIPTION
  30 The
  31 .Nm
  32 steering script allows running the
  33 .Xr mandoc 1
  34 regression suite on arbitrary operating systems,
  35 even though the suite was designed for OpenBSD only.
  36 .Pp
  37 When run without an argument,
  38 .Nm
  39 runs the complete regression suite.
  40 .Pp
  41 When run with one argument, that argument can be:
  42 .Bl -enum
  43 .It
  44 A single dot to run the complete suite.
  45 .It
  46 One of the top level directories, for example
  47 .Pa mdoc ,
  48 to run the test suite for a complete language or feature group.
  49 .It
  50 A subdirectory, for example
  51 .Pa man/IP ,
  52 to run the tests for a specific macro or an individual feature.
  53 .It
  54 A subdirectory with a test name appended with a colon, for example
  55 .Pa char/unicode : Ns Pa named ,
  56 to run the tests for one particular input file.
  57 .El
  58 .Pp
  59 Any additional arguments modify the way the tests are run.
  60 The default is
  61 .Cm all .
  62 The following modifiers are available:
  63 .Bl -tag -width verbose
  64 .It Cm all
  65 Run all kinds of subtests.
  66 This implies all other modifiers except
  67 .Cm verbose
  68 and
  69 .Cm clean .
  70 .It Cm ascii
  71 Run subtests for
  72 .Fl T Cm ascii
  73 output mode.
  74 .It Cm clean
  75 Remove all output files created by running the tests.
  76 .It Cm html
  77 Run subtests for
  78 .Fl T Cm html
  79 output mode.
  80 .It Cm lint
  81 Run subtests for
  82 .Fl T Cm lint
  83 warning and error output.
  84 .It Cm man
  85 Run subtests for
  86 .Fl T Cm man
  87 output mode.
  88 .It Cm utf8
  89 Run subtests for
  90 .Fl T Cm utf8
  91 output mode.
  92 .It Cm verbose
  93 Display approximate indications of what is being done.
  94 .El
  95 .Pp
  96 The amount of summary lines shown can be modified by giving an
  97 argument consisting of a single digit:
  98 .Bl -tag -width verbose
  99 .It Cm 3
 100 Show all summary lines for all directories entered.
 101 Even without
 102 .Cm verbose ,
 103 this generates more than hundred lines of output when running the
 104 complete regression suite.
 105 .It Cm 2
 106 This is the default.
 107 It shows the summary lines for the
 108 .Ar directory
 109 given on the command line and its immediate children.
 110 Except for
 111 .Pa mdoc ,
 112 the output usually fits on one screen.
 113 .It Cm 1
 114 Only show a single summary line for the whole run.
 115 .It Cm 0
 116 Do not show any summary lines.
 117 No output means success.
 118 Success or failure can also be seen from the exit status.
 119 .El
 120 .Pp
 121 All failed tests are always reported, even when the
 122 .Cm 0
 123 modifier is given.
 124 .Sh EXIT STATUS
 125 .Ex -std
 126 .Sh EXAMPLES
 127 The recommended invocation for casual users:
 128 .Pp
 129 .Dl ./regress.pl
 130 .Pp
 131 Maximum output:
 132 .Pp
 133 .Dl ./regress.pl \&. verbose
 134 .Pp
 135 Complete check, but keep the tree clean:
 136 .Pp
 137 .Dl ./regress.pl \&. all clean
 138 .Pp
 139 Test all of
 140 .Pa mdoc ,
 141 but don't print the usual 65 lines of output:
 142 .Pp
 143 .Dl ./regress.pl mdoc 1
 144 .Pp
 145 Investigate a specific failure:
 146 .Pp
 147 .Dl ./regress.pl mdoc/Bd:broken man verbose
 148 .Sh HISTORY
 149 The
 150 .Nm
 151 script appeared in release 1.14.1 of the portable
 152 .Sy mandoc
 153 distribution.
 154 .Sh AUTHORS
 155 .An Ingo Schwarze Aq Mt schwarze@openbsd.org
 156 .Sh CAVEATS
 157 This script is not optimized for elegance.
 158 Regression suites for other software should not copy the design.
 159 .Pp
 160 The problem it solves is that the
 161 .Sy mandoc
 162 regression suite is tightly integrated into the regression
 163 testing system of the OpenBSD base system, which requires
 164 both OpenBSD
 165 .Xr make 1 ,
 166 working neither with POSIX make nor with GNU make, and which
 167 also requires the OpenBSD-specific Makefile fragments in
 168 .Pa /usr/share/mk .
 169 The workaround of parsing the Makefiles by hand and constructing
 170 the required command lines by hand is unavoidably messy; it's
 171 the classic no-no of parsing a language with an ad-hoc incomplete
 172 parser.
 173 But the problem of providing this regression suite for other
 174 operating systems stood unsolved for many years, and no cleaner
 175 solution was found that could be implemented with reasonable effort.
 176 So maybe this is better than nothing.
 177 .Pp
 178 The top-level Makefiles for running this regression suite on
 179 OpenBSD are not included in the portable distribution.
 180 They are too OpenBSD-specific to be useful elsewhere,
 181 and on OpenBSD itself, the suite ought be run natively from
 182 .Pa /usr/src/regress/usr.bin/mandoc
 183 and not from the portable distribution.
 184 .Pp
 185 The
 186 .Pa db
 187 subdirectory of the regression suite is not included.
 188 It uses a Makefile structure that differs vastly from the
 189 rest of the suite.