scripting

Björn Lindberg d95-bli at nada.kth.se
Thu Jul 15 05:25:00 PDT 2004


Bennett Todd <bet at rahul.net> writes:

> 2004-07-14T17:08:52 Björn Lindberg:
> > This is sort of what I mean; bash was developed during and after the
> > POSIX standardization took place, and was intended to follow the
> > specification.
> 
> My recollection is that bash was developed as a /bin/sh replacement
> with some enhancements, well before POSIX started looking at /bin/sh
> --- or at least before their looking was publicly discussed.
> 
> The enhancements grew over time; for sure one source of ideas
> for enhancements was ksh, particularly when bash tracked POSIX
> standardization development. But the development was for sure
> independent, resulting in particular in a shell whose finger-feel in
> the cmdline editing and completion was wildly different, and whose
> scripting wasn't compatible in the funky extensions, only in the

I see, thanks for relating this. I think I need to give bash more
credit than I originally did. I just didn't think bash was "big
enough" to influence the POSIX process at the time.

Although I prefer ksh93 for scripting since it is much faster and has
more and better features (also is more POSIX-compliant in my
experience), bash is still a bit stronger in inteactive use in my
opinion. The readline integration is hard to beat in terms of
flexibility and support for Emacs editing commands.

> POSIX core, which seems to be not too dissimilar from classic
> /bin/sh, at least from the docs you pointed me to earlier.

In my experience the differences are quite large. Not only the new
syntax, such as (( ... )) & [[ ... ]], more sophisticated parameter
expansion, pattern matching, etc., but also many smaller things, such
as being able to write "export FOO=bar" directly, instead of having to
write "FOO=bar; export FOO".

> > Thus, it is my impression that bash did not influence the
> > standardization process much, but rather just following it.
> 
> I don't know whether it influenced it or not, but it definitely
> developed its POSIX compatibility contemporary with POSIX designing
> their spec.
> 
> > ksh -- in contrast -- and in particular the wide-spread ksh88
> > version, was developed before the POSIX effort, and influencing
> > the standards work to a much greater degree.
> 
> Yup. No question. And if it had been open source, perhaps the
> influence would have resulted in a compatible shell at the time,
> either from POSIX more precisely picking up the changes exactly, or
> ksh being adjusted to track POSIX at the time rather than sometime
> later.
> 
> > It seems to me that all modern shells today (meaning those based
> > on the Bourne shell of course; so tcsh, rc, es, etc. excluded)
> > derive in their functionality and syntax from ksh, which in turn
> > laid the basis for the POSIX standard.
> 
> They all derive their syntax and functionality from the classic
> Bourne Shell. A few features were picked up ksh by way of POSIX,
> although not unmodified; ksh88 wasn't POSIX compatible.

Of course, but I meant the new syntax in ksh, such as the conditional
expressions, arithmetic expressions and parameter expansion. It is the
same for all those shells, and seems to have originated with ksh.

> > I appreciate your input, since you were there when it happened, so
> > to speak. :-)
> 
> Thanks, I was there, although I gotta confess that was a long time
> ago, the memories are a little fuzzy. As am I for that matter:-).
> 
> > > Bash was available long before 2.0, it was one of the earlier major
> > > GNU programs --- understandably, it's nearly as fundamental as an
> > > emacs and a C compiler.
> > 
> > I see. But when would you estimate that it was usable, and when was it
> > starting to become POSIX-compliant? Looking at the release numbers and
> > the release dates, it seems that 2.0 was a milestone of sorts. Is that
> > correct?
> 
> bash was seriously useable in the late 1980s. Wish I could find some
> bash1 history somewhere. I _believe_ Brian Fox originally wrote it,
> Chet Ramey launched a CWRU fork that was later consolidated back
> as the lead thread of bash 1.x, that being sometime around 1990 or
> so maybe. I've got this vague recollection that the fancy readline
> editing/history/completion was born on that branch, but I may be
> wrong.

Ah, that exlains why there is a CWRU/ directory in the sources. Do you
know why the fork was made? When googling, I am able to find lots of
information on later versions of bash, but nothing on the versions
preceeding Chet Ramey's fork.

The earliest mention on Usenet I can find using Google, is the thread
starting with message <26563 at teknowledge-vaxc.ARPA>, called "Bourne
Again Shell?". It is from 7 February 1989. It is hard to tell the
state of affairs from reading it; for one thing it is mentioned that
someone is working on adding job control, and that "ksh features will
start arriving after stability is achieved", it is also mentioned that
is still has a lot of bugs. On the other hand, another message sounds
like it is quite ready at this time.

Ah, but here is an interesting message from Chet Ramey dated 11 June
1989, message <346 at cwjcc.CWRU.Edu>:

  [...]
  >Anyone care to take a crack at writing an AT&T-free ksh?  :-) Tom.

  It's being done, and in more than one place.

  Brian Fox, with a little help from the masses (again, I'm one :-),
  has just released version 0.99 of bash, the Bourne-Again SHell.
  This is the GNU project's Bourne+ Shell.  It has the job control,
  command-line editing, tilde expansion, etc.  that ksh provides in
  addition to bourne shell compatibility.  It's really a good shell,
  already an implementation of the current POSIX shell specification
  (part of the output of 1003.2), and I expect it to pick up more ksh
  features as time goes on.  The first I'm expecting is expression
  evaluation (the (( )) stuff).
  [...]

It seems to support what you said earlier. It seems that at this time
the development is proceeding quickly, and that bash is still quite
buggy, as evidenced by message <8906262356.AA18490 at aurel.caltech.edu>.

> 2.0 was a major new release, but by no means the first widely-used
> and useable one; there's still a bash1 branch being maintained and
> available in various *BSD ports and Linux distros.
> 
> > > Long before Linux came around, many of us enjoyed working on
> > > other Unix systems, and for at least some of us, the standard
> > > working pattern was to put all the GNU stuff into /usr/local/bin
> > > and put that at the front of our PATH, giving us the finger-feel
> > > of a GNU system wherever we were.
> > 
> > Ah, interesting. Why was that? Was the GNU tools percieved as
> > superior, or was it based on ideological reasons?
> 
> I can't speak for everyone who did this; it was a widespread habit,
> different folks probably had different motivations.
> 
> For me it was a mix, although I wouldn't call any of the reasons
> idiological. I'd long since started pursuing open source
> alternatives by strong preference for base tools; I got launched on
> that habit when I switched text editors for the last time, in 1983,
> to JOVE; prior to that my favourite text editor was PC-Write,
> breathtakingly fast and used the function keys very well, but a
> binary-only DOS-only app. Once I switched to jove and learned it I
> never had to learn another.
> 
> But with the basic Unix /bin and /usr/bin utils, there was a more
> pragmatic reason: the Unix utilities generally had serious input
> size limitations of various sorts. I blew up grep with a hairy
> regexp very early in my career, downloaded Henry Spencer's regexp
> lib and wrote my own grep which didn't blow up. The GNU tools were
> being written to be robust and free of arbitrary buffer size limits,
> and were getting nice enhancements. One I remember from early on was
> the find ... -print0 | xargs -0 ... pair that lets you safely
> process files even if they've got Funky Chars in their filenames.

I know what you mean. When I was using Solaris a lot, I noticed that
the base tools were lacking many of the convenient features of the GNU
tools, as well as often being less robust. My guess is that the
commercial Unices stopped improving on most of the base tools, instead
focusing on other aspects of their system. When GNU started to
recreate the tools, they had no reason not to make them as good as
possible from the start.

> > > But my real favourite shell is so different from both that it dodges
> > > the whole issue. rc rules!
> > 
> > I have been meaning to look into both rc and es, but never found time
> > for it. For what do you prefer rc, ie for scripting or interactive
> > use, and why? What are the most important features of rc compared to
> > ksh93 or bash?
> 
> I've lost track of es, I remember being fond of it, I should hunt it
> down again and add it to my repertoire.
> 
> As I recall, es is a nearly completely upward-compatible extension
> to rc, that adds a powerful mechanism for extending the shell.
> That's vague but that's all I can trawl up.
> 
> rc is a shell devised by Rob Pike for Plan 9, with several goals:
> 
>  - be an unsurprising, routine Unix shell, simple cmdline use works
>    as expected
> 
>  - solve /bin/sh's quoting complexity. rc has exactly one kind of
>    quote, the single-quote, which quotes everything until the
>    matchnig single-quote, and the only internal interpretation is
>    that a pair of adjacent single-quotes get collapsed to one. Other
>    than that, everything is literal:
> 
> 	; echo 'Hi there! I''ve got quotes in.'
> 	Hi there! I've got quotes in.
> 	; 
> 
>  - The big reason /bin/sh's quoting is such a nightmare is repeated
>    passes of various sorts of variable expansion and interpolation
>    applied one after another, at various points in time. This in
>    turn is almost entirely because /bin/sh doesn't have lists, only
>    simple strings, so they're constantly being whitespace-split and
>    reassembled to simulate lists. So rc has lists.

Yes, I really miss lists in shell programming, and the amount of hoops
you have to jump through to simulate them can be quite bothersome.

rc really sounds like an improvement to ksh/bash. The reason I haven't
made the effort to learn it yet is based on the fact that it is not
very useful to learn a new shell, since when writing scripts you tend
to need portability anyway. Maybe I will look into it one day.

> I _think_ rc introduced the notion of nonlinear pipelines, now
> available in many other shells; rc let you:
> 
> 	; diff -u <{echo foo} <{echo bar}
> 	--- /tmp/rc5205.0       2004-07-14 14:12:01.000000000 -0400
> 	+++ /tmp/rc5205.1       2004-07-14 14:12:01.000000000 -0400
> 	@@ -1 +1 @@
> 	-foo
> 	+bar
> 	; 

That is nice; I tried it with the oldest ksh I could find, and it
worked on that to:

d95-bli at my:~> /usr/xpg4/bin/sh
u at h:w> diff -u <(echo foo) <(echo bar)
--- /dev/fd/65  Thu Jul 15 14:11:09 2004
+++ /dev/fd/66  Thu Jul 15 14:11:09 2004
@@ -1 +1 @@
-foo
+bar
u at h:w> Version M-11/16/88i
u at h:w> uname -a           
SunOS my.nada.kth.se 5.9 Generic_112233-12 sun4u sparc SUNW,Ultra-4

This is a ksh88 installed as POSIX shell on Solaris.

> rc uses its lists to handle pipeline exit status correctly:
> 
> 	; sh -c 'false|true && echo that was true?'
> 	that was true?
> 	; rc -c 'false|true && echo that was true?'
> 	; 
> 
> The $status in rc after false|true is a list containing (1 0), and
> unless all the numbers in a list are zero rc treats it as false.
> 
> rc is a _lot_ smaller than bash; with a libreadline statically
> linked in (along with statically linked uClibc, completely static
> executables):
> 
> 	; ls -l /bin/rc /bin/bash
> 	-rwxr-xr-x    1 root     root       502228 May 24 03:29 /bin/bash
> 	-rwxr-xr-x    1 root     root       181332 May 24 10:51 /bin/rc
> 	; 
> 
> Oh, and my choice of prompts are straight from rc; its defaults,
> reflecting another clever idea of Rob Pike's, is that (in /bin/sh
> terms) PS1="; " and PS2="\t". So loop bodies type in indented when
> you enter them interactively, and the prompt doesn't interfere with
> the command's interpretation if you cut-n-paste a whole series of
> commands into another shell. I've extended this sometimes, when I
> want a more informative prompt (basically, everywhere except my base
> system) I use a prompt that looks like (on hostname "foo")
> 
> 	: foo; 
> 
> taking advantage of the ":" null command to gobble everything up to
> the semicolon.

Oh, that's clever! I'll have to adopt that for my own shell, at least
the PS2 as tab. Thanks for the tip.


Björn



More information about the lfs-chat mailing list