Item — HN Reader

For everyone who's saying "if you need unit tests then you should rewrite it in a different language", you're not wrong but you're not right either.

Bash is a lowest-common-denominator language, available on almost all platforms, with a very stable (old?) API, which makes it ideal for broad distribution & bootstrapping of systems.

When you use another other languages, like Python, suddenly you're not only worrying about whether Python is installed, but also which version is available, and you still can't use anything that's not part of the standard library (because then you enter package management/dependency hell).

I agree that even Python with stdlib-only is still better than Bash, but I guarantee that you'll find Bash >=3.0 on every host.

6 comments

> I agree that even Python with stdlib-only is still better than Bash, but I guarantee that you'll find Bash >=3.0 on every host.

Shouldn't you go for POSIX at that point? I'm not an expert, but I still have some idea how to test that my script is (somewhat) POSIX compliant (e.g. use shellcheck, run the script with dash or a shell in POSIX-compatible mode). I have Bash 5 on my Debian, how do I test that a script I tested with this version works on Bash 3 as well? Or have the changes between Bash 3 and Bash 5 been so minimal that I would have to do it on purpose to find an incompatibility?

I'm also curious about the number of machines where Bash scripts are run but don't have, say, Python 3.4 installed, and projects where it's a better trade-off to spend time developing and testing in Bash rather than have a Python 3 dependency. I'd say it's relatively rare.

Bash also has a -o posix option, which changes a number of things which may be significant, but I rarely see it used.

But none of those 3 solutions really check for POSIX compliance (including -o posix, since of course you can use many non-POSIX features when it's on). I am not aware that ShellCheck actually checks POSIX compliance; I am pretty sure it just checks for common errors in your bash/sh scripts (which is what it says on the home page).

I think what you may mean is "portable shell", i.e. portable between shell X and Y and system X and Y. That is a decent goal but many people use bash just because it alone is portable enough! The system tools are often unportable though. Limiting yourself to POSIX in that case is pretty painful and also virtually untestable. It's better to say "portable between X and Y" because I think that's what you mean.

In other words, there is plenty of stuff that's portable but not POSIX that you probably want to use. local variables are probably the biggest example. Every shell I know of supports those, including dash, but it's not POSIX (though maybe they're thinking about adding it; the spec is pretty behind)

I addressed this somewhat here: http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#li...

> I am not aware that ShellCheck actually checks POSIX compliance; I am pretty sure it just checks for common errors in your bash/sh scripts

Bashisms are errors if you specify #!/bin/sh. If you want to try it out, you can load random examples on https://www.shellcheck.net/ until you get a #!/bin/sh. Then you'll get warnings such as "SC3010: In POSIX sh, [[ ]] is undefined."

To go back to the discussion: if you want to use Bash features, you're better off switching to Python.

OK interesting, yeah with #!/bin/sh it points out that 'local' is not POSIX.

But every shell I know of supports 'local'. It's not a "bash-ism' because it's universally supported.

I consider it essential; otherwise you might as well not use functions in shell.

This is a long argument (and is addressed in the same FAQ), but as someone who's written hundreds of thousands of lines of Python, I think shell is still better for a large set of tasks. A typical small project I write might have 300 lines of shell and 1000 lines of Python, rather than 5000 lines of Python.

But of course there are problems with using shell; if there weren't then Oil wouldn't exist :)

AFAIK unsetting the variables first works too.

For scripts with a #/bin/sh shebang line, ShellCheck points out bashisms.

I had something like this in my last job: I had to create an installer for a VCS aimed at chip designer.

The installer would be ignored as much as my coworkers could manage, and as many customers as possible had to be able to just use it unmodified. Since our supported OSes were RHEL6, RHEL7, SLES12, and SLES15, the best option was bash. Also had to build it on the off chance one of our more restrictive customers was still on RHEL5.11

Pain in the butt, but it worked.

>Bash is a lowest-common-denominator language, available on almost all platforms, with a very stable (old?) API, which makes it ideal for broad distribution & bootstrapping of systems.

/bin/sh is, bash isn't. Portability is a good excuse for using /bin/sh, bash is just a crappy middle ground, it's not as portable as the POSIX shell and it's still not a decent substitution for a proper scripting language.

Portability is not important in most cases. Most shell scripts are written for and run on a single platform usually as setup or teardown for something else.

I used bash to write a testing framework for a project I have that needs minimal dependencies. E.g. I don't want to depend on having Python (and the transitive closure of its batteries-included experience) installed. For that purpose, bash works pretty well. I also want to run on some older machines, like 32-bit macs.

Turns out that bash does evolve...slowly. E.g. I found out that the default bash on MacOS 10.4 doesn't support the "=~" operator...oi...

In addition, the default MacOS Bash is buggy :-/

What code are you writing that is pure Bash? Every Bash script I've written or find in the wild is always used to control subprocesses, and then you have to worry about compatibility of those programs, which are all over the place.

Python stdlib does way more than Bash, it's generally easier to just install Python and run a script that sticks with stdlib, than to install Bash and then figure out what subprocesses it invokes, what packages those map to in the OS package manager, what versions they are, if they're compatible with what you wrote, etc. And if you need more libs, Python also has a cross-platform package manager with a consistent interface. As an example, consider coreutils between macOS and Linux--macOS ships with dramatically different coreutils programs (such as "ls"). The only feasible way to get them consistent across OSes is to stuff everything in a Docker image, but that has its own problems and limitations. Or ask the user to figure it out for themselves, e.g. "use Homebrew".

My point is that Bash is already installed almost everywhere, including across WSL, Linux, and OSX.

Yes, when working with sub-processes you need to account for platform differences...which is also true when Python.

A Bash script "just works", while I have to direct customers to install the appropriate version of Python (or whatever else).

> Yes, when working with sub-processes you need to account for platform differences...which is also true when Python.

The cases in which you even need to use subprocesses is much smaller with Python, because the stdlib replaces a huge amount of external programs typically used with Bash (cat, grep, awk, sed, cut, paste, wc, ls, find, etc.). And so if you stick with stdlib without subprocesses, you don't need to account for platform differences. Python has already done that work for you.

In simple cases, or cases where portability doesn't really matter, then I agree with you. But once you start caring about portability, you have to be careful to consider the compatibility differences of external programs across distributions and OSs on which Bash runs, which vary dramatically among even the simplest or most common of programs such as "ls" or "grep".

Example:

    # ubuntu 20.04
    echo fooboo | grep -Eo '.+?oo'
    > fooboo

    # macos big sur
    echo fooboo | grep -Eo '.+?oo'
    > foo
    > boo

Bash scripts being portable has little to do with Bash itself, it's either deliberate by the programmer, or a happy coincidence. IME it's almost always the latter, if it's portable at all (which isn't uncommon in my work).

(Also wanted to add that this isn't a unique property to Python either. Any "real" language typically has these same properties.)

Totally agree. You can avoid lots of this by using Bash built-ins, but that's more advanced/esoteric Bash development that most people aren't familiar with.

Thankfully, with the official deprecation of Python 2.7 and the slow march of progress I hope to be able to better standardize on Python 3 + stdlib (or similar).

That is true, but most languages do work on all of those things, e.g. For parts of the D ecosystem we have moved over to build scripts actually written in D, and they almost always work even though the makefiles and bash scripts are sometimes completely fucked whenever I fiddle with my computer. The best part is with the scripts in a real programming language (Python is good enough too), you can actually read them and show how they work to people coming from visual studio who can't read bash or make

There are platforms where bash is not available. I wonder why people who want to run it on various platforms use bash.

ShellSpec (https://shellspec.info/) is a POSIX compliant testing framework supports all POSIX shells (Bash >= 2.0, dash, ksh88, zsh >=3.1, etc). I'm the author of ShellSpec.

Bash isn't installed default on OpenBSD or FreeBSD, and the bash on MacOS is a very old version. Also not installed by default on Alpine Linux, which would be fairly popular in the container world.