[blfs-dev] Test Suites
qrux.qed at gmail.com
Thu Feb 16 15:29:37 PST 2012
On Feb 16, 2012, at 2:57 PM, Ken Moffat wrote:
> On Thu, Feb 16, 2012 at 02:22:08PM -0800, Qrux wrote:
>> In principle, I think testsuites are awesome; I like the confidence they give that the system works as advertised.
> In practice, I find them full of false failures - often, google
> knows that the particular test always fails, occasionally there is
> even a fix to make it work or to just drop the failing test. More
> often, any fix doesn't apply, or doesn't solve the failure.
> Meanwhile, I've seen all sorts of weird and wonderful bugs in
> desktop programs over the years - but I've never seen one that
> was caught by the testsuite.
Are you referring specifically to X11-related test suites?
Because, despite personally loving tests in principle, I'd be willing to divide the book (in concept--not proposing any major reorgs, Bruce) into "stuff you can-and-should test" and "everything else". IMO, X and devices (e.g., scanners, non-keyboard USB devices, printers, audio) fall on the latter side of that line. Most other things can--and should be--tested. By "most other things", I think of stuff like openssl where a single test failure is a sign of a major problem that deserves much attention.
I think once you get into X, you're so deep into userland that it may not even matter to test stuff. And, this is not an attempt to disparage X or GUIs; I've run Motif, twm, fvwm, wmaker, and finally KDE desktops for many years, before I finally switched to Mac OS for UI--heck, I've even own the X11 manuals, and have written code from the blue book. :) But, testing GUI apps is hard, and may not be particularly relevant.
The one gray area would be libraries embedded in GUI apps, and a STRONG warning should be attached to any package that owns libraries that become dependencies for other software (i.e., they cannot be--or are not commonly--separated out from the GUI part of the app).
More information about the blfs-dev