"$( )" command substitution vs. embedded ")"

This page doesn't serve any purpose but fun, it's just about a special point concerning the robustness of shell parsers.

2016-04-17 (see recent changes)

In some shells, $( ) collides with other, "unbalanced", yet valid, right parentheses.
And there are some more subtle issues.

The much more famous issue, $( ) vs. `...`, is mentioned below in the additional end notes.

What does the standard say?

The SUSv3 rationale and SUSv4 rationale to shell commands, C.2.9, about "case x in (x)" embedded in "$( )":

"An optional left parenthesis before pattern was added to allow numerous historical KornShell scripts to conform.
At one time, using the leading parenthesis was required if the case statement was to be embedded within a "$()"
command substitution; this is no longer the case with the POSIX shell. Nevertheless, many historical scripts use
the left parenthesis, if only because it makes matching-parenthesis searching easier in vi and other editors."

However, there are some cornercases for "$( )" even in shells which try to implement posix.

The following table lists the results for some testcases. These tests exploit the case command and here-documents.
They were taken from the above URL or Usenet and the thread <mailman.20673.1135786095.20277.bug-bash@gnu.org>
in gnu.bash.bug (Dan Jacobson and Eric Blake). Jilles Tjoelker pointed out the problem with '# \'.
The table below unfortunately suggested wrong behaviour as correct for test D.2 until Robert Elz kindly pointed out that it should fail.

Examples, each to be embedded in $( ) , see full testcases

A.1	"case x in x)"
A.2	"case x in x)" with a comment
A.3	"case x in (x)"
A.4	"case x in (x)" with a comment
A.5	"case x in (x)" with \n and without ";;" before "esac"
B	) in quotes
C	) after a comment sign
D.1	) embedded in a here-document
D.2	) embedded in a here-document, with \() inside
D.3	) embedded in a here-document, eof and closing ) on same line According to POSIX/SUSv1-v4 it should fail with unterminated here-doc (see 2.7.4).
D.4	eof and closing ) on same line, combined with another correct terminator
E	) as delimiter for an embedded here-document (academic)
F	unbalanced quote in an embedded here-document
G	\ at end of line
H	empty

Additional Notes:

• the original Almquist shell and its early variants fail with A.3/4/5 (workaround) because their grammar doesn't accept an opening parentheses in a case command yet. Later ash variants usually accept the opening paren, e.g., BSD/OS 3.0, NetBSD 4.0, FreeBSD 4.1/5.1, (d)ash 0.3.5-4, Slackware 8.1, Minix 3.1.3, Busybox from birth.

  The ash parser doesn't need this workaround. However, this is also an issue of script portability.

• traditional Bourne shells are not listed because they don't know "$( )" yet and thus had not implemented the opening paren in a case pattern, yet.

• the original ash suffers from a SEGV for empty command substitutions

• A few words about the famous $( ) vs. `...`

  pro $() form:

  Without doubt, the "$( )" syntax is the cleaner design because there are no escaping rules to consider. And as \ is special in `...`, you even have to escape itself twice. Why was command substitution implemented differently, at all, in the first time? One possible argument: You need a recursive parser for $() which might have been too expensive at the time of 6th edition.

  Another argument: backquotes are easily confused with apostrophes -- not so much when actually using them (in terminals with fixed width fonts), but especially in other sources of documentation using arbitrary fonts.

  weak pro:

  An often mentioned advantage of the "$( )" syntax: it allows clean nesting. However, that's a weak argument: If you really have to nest command substitution, then you might want to split into separate commands to improve readability and to ease maintenance.

  contra:

  The "$( )" form is not traditionally portable because traditional Bourne shells only implement the `...` form.
  But the more the traditional Bourne shell disappears, the weaker this arguments becomes.

  academic contra:

  Parsing problems mentioned on this page, see above.

  An issue in ksh88, see below.

• In ksh88, at least from release a to i, you have to be aware of a subtle quoting issue inside $( ).
  Single quotes in embedded here-documents are converted to double quotes. This means that

      echo $(cat << EOF
           "double quotes"
           'single quotes'
      EOF
      ) 

  results in

      "double quotes" "single quotes" 

  This happens for the $( ) form but not for the `` form.
  This construct is likely to be used if you feed a large block of text to a command and want to save the command output.