The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours

2010-04-06 (see recent changes)

Here you'll find

• More reading
  • Origin
  • Unix FAQ
  • Andries Brouwer's findings
• Selected issues
  • Blank after #! required?
  • Blank forbidden?
  • env utility
  • splitting arguments
  • setuid support
  • can you nest #!
  • comments
  • maximum length
  • POSIX.2/SUS
  • Possible errors
• A table with test results on numerous systems

More reading

• The Origin

  See an old mail from Dennis Ritchie introducing the new feature, quoted in 4.0BSD, /usr/src/sys/newsys/sys1.c.
  So this mechanism was invented between Version 7 and Version 8. It was then available in 4BSD but not activated per default until 4.2BSD.
  (pointed out by Gunnar Ritter in <3B5B0BA4.XY112IX2@bigfoot.de>, de.comp.os.unix.shell.)

  In 4.3BSD Net/2 the code was removed due to the license war and had to be reimplemented for the descendants (e.g., NetBSD, 386BSD, BSDI).

• The Unix FAQ

  The paragraph "3.16) Why do some scripts start with #! ... ?" (local copy),
  emphasizes the history concerning shells, not the kernel.

  That document is wrong about two details:

  • #! was not invented at Berkeley (but they implemented it first in widely distributed releases), see above.
  • The document explicitly states that only csh was modified on the BSDs.
    However, the Bourne shell was modified likewise on BSDs as well.
    See the first occurence in 3BSD usr/src/cmd/sh/service.c.

• There is also an article from Andries Brouwer, which you shouldn't miss. It emphasizes some other things which are not explained here and follows a more generic approach (and it differs concerning a very few details).

• Wikipedia covers this topic with Interpreter directive and Shebang.

Selected issues

• Blank after #! required?

  There is a rumor, that a very few and very special, earlier Unix versions (particularly 4.2BSD derivatives) require you to separate the "#!" from the following path with a blank.
  You may also read, that (allegedly) such a kernel parses "#! /" as a 32-bit (long) magic. But it turns out that it is virtually impossible to find a Unix which actually required this.

  4.2BSD in fact doesn't require it, although previous versions of the GNU autoconf tutorial wrongly claimed this ("10. Portable Shell Programming", corrected with release 2.64, 2009-07-26).
  But instead, see (again 4.0BSD /usr/src/sys/newsys/sys1.c from above and) the first regular occurence in 4.2BSD, /usr/src/sys/sys/kern_exec.c. The source accepted a blank, but never required it.
  All this pointed out by Gunnar Ritter in <3B5B0BA4.XY112IX2@bigfoot.de> (and thanks to the new Caldera license, the code can be cited here now.)

  Instead, the origin of this myth "of the required blank" might be a particular release of 4.1 BSD: There is a manpage in a "4.1.snap" snapshot of 4.1BSD on the CSRG CDs, /usr/man/man2/exec.2 (4/1/81), where a space/tab after the #! is mentioned as mandatory. However, this is not true: the source itself remained unchanged. (Hint to the existence of such a manpage from Bruce Barnett in <ae3m9l$rti$0@208.20.133.66>).

  It's not clear whether this is a bug in documentation or if Berkeley planned to modify the BSD source but eventually did not.

  DYNIX is mentioned in the autoconf documentation, too. It's unclear if this variant might have implemented it in a few releases
  (perhaps following the abovementioned manual page). However, later releases did not implement it according to Usenet discussions.

  I asked David MacKenzie, the author of the autoconf documentation, about the actual origin of the autoconf note.
  But unfortunately neither the reporting author nor the very system are recorded anymore.

  Even intensive search of usenet archives didn't reveal any further hints to me.

• Blank forbidden?

  I found no evidence yet, that there's an implementation which forbids a blank after #!

• The env utility

  env(1) is often used with the #! mechanism to start an interpreter, which then only needs to be somewhere in your PATH, e.g. "#!/usr/bin/env perl".

  However, the location of env(1) may vary. Free-, Net-, OpenBSD and some Linux distributions (Debian) only come with /usr/bin/env. On the other hand, there's only /bin/env at least on OpenServer 5.0.6 and Unicos 9.0.2. (On some other Linux distributions (Redhat) it's located in /bin and there's a symbolic link from /usr/bin/env to it.) The env-mechanism is highly increasing convenience, but cannot strictly assure "portability" of a script.

  In practice, env must not be a script, because the #! mechanism usually accepts only binary executables (except on a few implementations like Minix and Linux 2.6.27.9 ff.).

• Splitting arguments

  Some systems split up the arguments like a shell to fill up argv[], e.g.,

  • FreeBSD before 6.0 (change pointed out to me by Akinori Musha)
  • MacOS X since 10.4(/xnu-792/Tiger), at least until 10.6
  • BSD/OS 4.1, and thus BIG-IP 4.2
  • Minix 2 and 3.

  For Linux, a patch was suggested on the Linux kernel mailing list, with an interesting discussion of some portability issues.

• Setuid (set user id) support
  • The setuid/gid-bit became ignored on many systems for security reasons. This is mainly due to the race condition between the kernel starting the interpreter and the interpreter starting the script.

  • SVR4 and 4.4BSD introduced a virtual filedescriptor filesystem which allows for avoiding this race: Here the kernel can hand over an open filedescriptor (e.g. /dev/fd/n) to the interpreter. However, 4.4BSD didn't support setuid scripts, yet. The UNIX FAQ claims this (4.7. "How can I get setuid shell scripts to work?"), but it's explicitly denied in kern_exe.c. (And 4.4BSD-Lite lost its execve() implementation due to the license issue.)
    Instead, it might have been added to a very early NetBSD release ¹.

    [1]

    NetBSD already implements it in the first cvs entry for exec_script.c (01/94), some time before release 1.0. The filedescriptor filesystem ("fdescfs") had been added with release 0.8 (04/93). NetBSD was influenced by 386BSD, but I couldn't find it there (including patchkit 0.2.4, 06/93). FreeBSD, which is a direct descendant of 386BSD, doesn't implement it either. OpenBSD forked off from NetBSD later (10/95) and thus implements it like NetBSD.

    Set user id support is implemented (always by means of the fd filesystem) for instance on:

    • Solaris (since birth)
    • Irix (at least since release 5)
    • UnixWare (since birth)
    • NetBSD (almost since birth; but only with the kernel option "SETUIDSCRIPTS" activated)
    • OpenBSD (since birth; but only with the kernel option "SETUIDSCRIPTS" activated)
    • MacOS X since 10.5(/xnu-1228/Leopard), earlier releases came without the fd filesystem

  • A sidenote: the SVR4 shell introduced the related flag -p. Without this flag, the EUID is set back to the UID if different.
    ksh88 and ksh93 in contrast activate this flag automatically if the euid/egid is not equal to the uid/gid.
    bash-1 behaved similar; but bash-2 ff. requires the flag to be set.

  • Nowadays many systems still ignore the setuid-bit with the #! mechanism, because you have to be aware of numerous issues. Keywords are:
    • the abovementioned race condition about the actual script being called (symlink attack)
    • some ksh88 (relevant on systems which do not use the /dev/fd mechanism) show this quirk:
      when opening a script, they look at PATH before looking at the current directory.
    • shell escape in subsequent commands
    • full control over data flow in all commands?
    • inherited environment
    • immunity against -i attacks
    • control over file name expansions, if used
    • overwriting of existent files
    • race conditions about internal temp files
    • safe understanding of script maintainance in future by other people

• Can you nest #!?

  Most probably there isn't any Bell-Labs- or Berkeley-derived Unix that accepts "nested" #!.
  However, Minix and Linux since 2.6.27.9 ² accept this.

  Be careful not to confuse whether the kernel accepts an interpreted script after #!,
  or if your shell silently tries to take over if the kernel refused it or returned with an error for another reason:

  • bash-1 always behaves so (the line length then is truncated to 80 characters and argv[0] is the invoked script.)
  • bash-2 and bash-3 do so, if the #! mechanism was not present at compile time (probably only in unix-like environments like cygwin).
  • If a filesystem is mounted with a "noexec" option the shell could take over as well (pointed out by Carl Lowenstein).
  • The original Almquist shell also recognizes #!, but only if "BSD" was not defined at compile time. Later variants de-facto do not recognize it.

  [2]	For more information about nested #! on Linux, see the kernel patch (applied to 2.6.27.9) and especially binfmt_script.c which contains the important parts. Linux allows at most BINPRM_MAX_RECURSION, that is 4, levels of nesting. (hint to me about the change by Mantas Mikulėnas.)

• Comments

  FreeBSD 4.0 introduced a comment-like handling of "#" in the arguments, but release 6.0 revoked this (see also a discussion on freebsd-arch).

  MacOS X introduced comment-like handling of "#" with release 10.3(/xnu-517/Panther)

• About the the maximum length of a #! line:
  • Originally (Research Unix between Version 7 and 8) it was 16 bytes.

  • 32 bytes on 4.xBSD, SunOS 4 and Ultrix 4.3. This is "sizeof(struct a.out/exec)".
    The reason is a union, which contains both struct a.out/exec and a string with the same length (to access the #! line).
    (On SVR3, earlier HP-UX and on Unicos it's the same limit; but I don't know if for the same reason.)

  • On some systems, the line isn't truncated if it was too long, but the kernel refuses to execute the file and returns with ENOEXEC.
    In some shells, you won't get an error or an according exit status in the command line in this case and it's a quite silent failure then.

  • For the history on FreeBSD, see imgact_shell.c and <sys/imgact.h> before 6.0
    and <machine/param.h> since 6.0 (MAXSHELLCMDLEN became PAGESIZE, which depends on the architecture),
    e.g., i386, ia64, sparc64, amd64, alpha (supported until 6.3) param.h, alpha_cpu.h).

  • For the history on NetBSD, see kern/exec_script.c (MAXINTERP in <sys/param.h> or PATH_MAX in <sys/syslimits.h>, respectively).

  • For the history on OpenBSD, see kern/exec_script.c (MAXINTERP in <sys/param.h>).

  • 127 bytes on Linux, see also load_script() in linux/fs/binfmt_script.c (and binfmts.h).

    On Linux, #! was introduced with kernel release 0.09 or 0.10 (0.08 had not implemented it, yet).
    In fact, the original maximum length was 1022, see linux/fs/exec.c from Linux 0.10. But on Linux 0.12, this was changed to 127 (parts of a diff).

  • On many other flavours, the maximum length varies between _POSIX_PATH_MAX (255) and PATH_MAX (f.i. 1024); see limits.h or syslimits.h on the respective system.

    Exceptions are BIG-IP4.2 (BSD/OS4.1) with 4096 and FreeBSD since 6.0 (PAGE_SIZE) with 4096 or 8192 depending on the architecture.

    Minix also uses the limit of PATH_MAX characters (255 here) but the actual limit is 257 characters,
    because patch_stack() in src/mm/exec.c first skips the "#!" with an lseek() and then reads in the rest.

• POSIX.2 or SUSv2 / SUSv3 / SUSv4 mention #! only as a possible extension:

      Shell Introduction
      [...]
      If the first line of a file of shell commands starts with the
      characters #!, the results are unspecified.
  
      The construct #! is reserved for implementations wishing to provide
      that extension. A portable application cannot use #! as the first
      line of a shell script; it might not be interpreted as a comment.
      [...]
  
      Command Search and Execution
      [...]
      This description requires that the shell can execute shell
      scripts directly, even if the underlying system does not support
      the common #! interpreter convention. That is, if file foo contains
      shell commands and is executable, the following will execute foo:
  
        ./foo 

  There was a Working Group Resolution trying to define the mechanism.

  On the other hand, speaking about "#!/bin/sh" on any Unix:
  This is a really rocksolid and portable convention by tradition, if you expect anything from the Bourne shell family and its descendants to be called.

• Possible errors:
  • If the interpreter is not found, the system returns ENOENT.

    This error can be misleading, because the shell applies it to the script called and not to the interpreter in its #! line.
    As exception, bash-3 subsequently reads the first line itself and gives a diagnostic concerning the interpreter
    "bash: ./script.sh: /bin/notexistent: bad interpreter: No such file or directory"

  • If the #! line is too long, at least three things can happen:
    • The system returns E2BIG (IRIX, SCO OpenServer) or ENAMETOOLONG (FreeBSD, BIG-IP4.2 (BSD/OS4.1)
      and you get something like "Arg list too long" / "Arg list or environment too large" or "File name too long", respectively.
    • You get ENOEXEC. In some shells this results in a silent failure. Some shells subsequently try to interprete the script itself.
    • The line gets silently truncated to the maximum length allowed.

Test results from various systems

I used the following as program "showargs":

    #include <stdio.h>
    int main(argc, argv)
	int argc; char** argv;
    {
	int i;
	for (i=0; i<argc; i++)
	    fprintf(stdout, "argv[%d]: \"%s\"\n", i, argv[i]);
	return(0);
    } 

and a one line script named "invoker.sh" to call it, similar to this,

    #!/tmp/showargs -1 -2 -3 

to produce the following results (tried them myself, but i'd like to add your results from yet different systems).

Typically, a result from the above would look like this:

    argv[0]: "/tmp/showargs"
    argv[1]: "-1 -2 -3"
    argv[2]: "./invoker.sh"

... but the following table lists the variations. The meaning of the columns is explained below.

Meaning of the columns:

• "maximum length of #! line": self explanatory
• "cut-off(c), error(err) or ENOEXEC ( )": see the selected issues above.
• "only the 1st arg": argv[1] from above contains only "-1" then
• "each arg in its own argv[x]": argv[1]:"-1", argv[2]:"-2", etc.
• "handle # like a comment": if a # appears in the arguments, the rest of the line (including this #) is ignored
• "invoker in argv[0]": argv[0] doesn't contain "/tmp/showargs" but "./invoker.sh"
• "not full path in argv[0]": argv[0] contains the basename of the called program instead of its full path.
• "remove trailing whitespace": self explanatory
  attribute pointed out by Matthew Garrett.
• "convert tab to blank": self explanatory
• "accept interpreter": the invoked program from the "#!"-line may be an interpreted script itself
• "search current directory": this means "#!sh" works if called from /bin
• "allow suid": this permission bit might be ignored for security reasons

Footnotes in the table:

And why shebang? In music, '#' means sharp. So just shorten #! to sharp-bang. Or it might be derived from "shell bang". All this probably under the influence of the american slang idiom "the whole shebang" (everything, the works, everything involved in what is under consideration). See also the jargon dictionary or Merriam-Websters for the slang idiom.

Sometimes it's also called hash-bang, pound-bang, sha-bang/shabang, hash-exclam, or hash-pling (british, isn't it?).

<http://www.in-ulm.de/~mascheck/various/shebang/>