"$( )" command substitution vs. embedded ")"

This page doesn't serve any purpose but fun, it's just about a special point concerning the robustness of shell parsers.

2009-12-26 (see recent changes)

$( ) is comfortable about nesting, but collides with other, "unbalanced", yet valid, closing parentheses in some shells.
Also, some parsers stumble over quotes in here-documents embedded in $( ).

The much more famous issue, $( ) vs. `...`, is mentioned in the additional end notes.

What does the standard say?

The SUSv3 rationale to shell commands, C.2.9, about "case x in (x)" embedded in "$( )":

"An optional left parenthesis before pattern was added to allow numerous historical KornShell scripts to conform. At one time, using the leading parenthesis was required if the case statement was to be embedded within a "$()" command substitution; this is no longer the case with the POSIX shell. Nevertheless, many historical scripts use the left parenthesis, if only because it makes matching-parenthesis searching easier in vi and other editors."

However, omitting the leading parentheses is still problematic, even in shells which claim posix conformance. And there are some more cornercases for "$( )". The following table lists the results for some testcases. These tests exploit the case command and here-documents. They were taken from the above URL or Usenet and the thread <mailman.20673.1135786095.20277.bug-bash@gnu.org> in gnu.bash.bug (Dan Jacobson and Eric Blake). Jilles Tjoelker pointed out the problem with '# \'.

Examples, each to be embedded in $( ) , see full testcases

A.1	"case x in x)"
A.2	"case x in x)" with a comment
A.3	"case x in (x)"
A.4	"case x in (x)" with a comment
A.5	"case x in (x)" with \n and without ";;" before "esac"
B	) in qoutes
C	) after a comment sign
D	) embedded in a here-document
E	) as delimiter for an embedded here-document (most academic)
F	unbalanced quote in an embedded here-document
G	\ at end of line
H	empty

Additional Notes:

• the original Almquist shell and its early variants fail with A.3 and A.4 (workaround) because their grammar doesn't accept an opening parentheses in a case command yet.
  Later ash variants usually accept the opening paren, e.g., BSD/OS 3.0, NetBSD 4.0, FreeBSD 4.1/5.1, (d)ash 0.3.5-4, Slackware 8.1, Minix 3.1.3, Busybox from birth.

  In fact, the ash parser doesn't need this workaround. However, this is also an issue of script portability.

• traditional Bourne shells are not listed because they don't know "$( )" yet and thus had not implemented the opening paren in a case pattern, yet.

• the original ash suffers from a SEGV for empty command substitutions

• A few words about the issue "$( )" vs. `...`:

  Without doubt, the "$( )" syntax is the cleaner design because there are no escaping rules to consider.

  An often mentioned advantage of the "$( )" syntax: it allows clean nesting for this reason. However, if you really have to nest command substitution, then you should split into separate commands, to ease maintencance and to improve readability. A better argument: backquotes are easily confused with apostrophes -- not so much when actually using them (in terminals with fixed width fonts), but especially in other sources of documentation using arbitrary fonts. Another better argument: \ is special in `...` so you have to escape twice.

  However, the "$( )" syntax is not as traditionally portable as the `...` form.

• ...and in ksh88 (at least from release a to i) you have to be aware of a subtle quoting issue inside $( ).
  Single quotes in embedded here-documents are converted to double quotes:

  echo $(cat << EOF
       "double quotes"
       'single quotes'
  EOF
  ) 

  results in

      "double quotes" "single quotes" 

  This happens for the $( ) form but not for the `` form.
  This construct is likely to be used, if you feed a large block of text to a command and want to save the command output.