back
227 comments
It's still relevant if you use -a, -o, or parentheses (which I guess is reason to avoid them):

  $ a=1; b=1; c='!'; d='!'
  $ [ "$a" = "$b" -a "$c" = "$d" ]
  -bash: [: too many arguments
  $ [ '!' '(' "$c" = "$d" ')' ]
  -bash: [: `)' expected, found !
  $ echo "${BASH_VERSION}"
  5.1.4(1)-release
Note that it's not even a bug here; it's necessary for disambiguation.
Alternatively you can use the newer (and highly recommended) bash syntax that is also quote safe:

     $ a=1; b=1; c='!'; d='!'
     $ [[ $a = $b && $c = $d ]] && echo true
     true
For anyone writing bash scripts today I would avoid the old test/[ functions entirely.
The post mentions the issue is fixed by POSIX, which… notably does not standarize -o -a and parens. I guess now we know why POSIX didn't standardize those.
> Note that it's not even a bug here; it's necessary for disambiguation.

I would call it a bug when there's one valid way to read it, and [ fails to see it. I don't see the ambiguation. For example, zsh's [ doesn't have these errors:

  $ a=1; b=1; c='!'; d='!'
  $ [ "$a" = "$b" -a "$c" = "$d" ]; echo $?
  0
  $ [ '!' '(' "$c" = "$d" ')' ]; echo $?   
  1
Same result with FreeBSD's tcsh's [.

I think bash's [ is just lacking the ability to backtrack while parsing its arguments.

Curiously, GNU's [ fails to parse the arguments of the first, but not the second's:

  $ command [ "$a" = "$b" -a "$c" = "$d" ]; echo $? 
  [: extra argument ‘!’
  2
  $ command [ '!' '(' "$c" = "$d" ')' ]; echo $?   
  1
It seems its argument parsing works differently when using parentheses:

  $ command [ "$a" = "$b" -a '(' "$c" = "$d" ')' ]; echo $? 
  0
Well, never seen any reason to use -a or -o. To me they are less clear and you are prone to error. I find things like: [ "$a" = "$b" ] && [ "$c" = "$d" ] much more clear.
TIL
On a side note, I think this style tweak makes it more readable:

    [ x"$var" = x"val" ]
My eye doesn't have to separate "xval" into "x" and "val", so it's more obvious that the value being compared against is "val".

(It behaves the same way since the shell allows quoting to start in the middle.)

Related: Problems With the test Builtin: What Does -a Mean?

http://www.oilshell.org/blog/2017/08/31.html

The POSIX spec did indeed make things cleaner; the last section quotes it and gives some style advice.

As usual, the problem is that Sh & co. use in-band control, especially when talking to utilities. Just a mess of text being passed around, of which every program must make sense on its own—but this happening in shell's own functions takes the cake.

Moreover, similar problems are inherited by some programs that try to do shell-like commands but put their own syntax in the way. E.g. Ansible, where YAML, JSON, module call DSL and Jinja are all mixed on top of the commands so you can bump into any of them.

Interestingly enough, this isn't a mess in the shell's own builtins. The [ ] expression syntax is provided by the program /usr/bin/[ which is (often) a symlink to /usr/bin/test. It's just a clever trick! [ is a program that evaluates it's argv as an an expression, less the trailing ], and then returns the result as an error code.

Actually re-reading your post you probably know that already! Nevermind then. Maybe someone reading this will be enlightened

[ is not, in principle, the shell's own function. It merely happens to be implemented as one in most shells anyone actually uses as an optimization.
Indeed! As always, http://langsec.org/
The "POSIX prescriptions" mentioned in the quoted comment can be found here under "algorithm for determining the precedence": https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...
So, in conclusion it's a hack... but, it's also incredibly robust. It doesn't hurt, and we acknowledge there are cases where it could help. It's idiomatic enough that literally everyone who's done a nontrivial amount of scripting is familiar with it.

At some point, "decades old idiomatic hack that increases robustness" stops being a hack and becomes an idiomatic way to increase robustness.

A better conclusion would be "yes, keep using it".

> So, in conclusion it's a hack... but, it's also incredibly robust.

You know, shell scripts are a hack.

I really wish there was a middle ground between shell scripts and say python. Python has proper lists and hashes and loops and can manipulate paths with spaces or quotes or unicode. It doesn't get dragged down by the escape-an-escape-within-a-regex nonsense that makes things undecipherable. or ${VAR:-foo} or ${VAR##ugh}

But python is a bit fumbly when it comes to invoking external programs. try: ... subprocess.bleh except: ... ugh.

I learnt this trick from OpenBSD 2.5 (yes, a long time ago). I still remember getting root with a shell-script mp3 player that a friend coded because of this.

The exploit was based in inserting some \$\( make_a_new_root_account_in_etcpassw \) inside the variable inside the test.

As a result, both my friend and I stopped writing shell scripts as CGI and started learning perl just to do it.

Note that this is a relic of the past, as TFA explains. Modern bash can be totally manageable and even fun, if you take the time to learn and have a good style guide and shellcheck by your side.
I use this hack as a matter of course, having internalised it aeons ago. Bash does lots of things better now but as others have commented it is still so fundamentally weird that an innate conservatism is often a wise approach, especially when one lacks the time (or motivation) to explore its "newer" features.
I sometimes wonder why bash transpilers aren't more popular? Bash is such a strange language (or maybe just very old with lots of cruft) that it's impossible to write anything meaningful without check SO 5 times.

Given i haven't spent a lot of time writing shell script yet but the syntax is just super weird, like I recently learned(1) that when you When you write if [ -e /etc/passwd ]; then .. that bracket is not shell syntax but just a regular command with a funny name.

Is there a recommended bash transpiler like what babel is to Js? So i can write bash scripts using a more coherent language and it spits out .sh files.

(1) https://news.ycombinator.com/item?id=26701079

That's more or less what Perl is (a more coherent language).

Perl may look ugly, but it doesn't have the escaping problems that Unix shells have. And most of the usual shell features (listing files, launching commands, etc...) are built-in. It is also available in almost all Unix systems, maybe even more so than bash.

I wouldn't go that far as calling it impossible, but it sure is crufty and quirky seen with modern eyes.

My "favorite" (i.e. most terrifying) BASH fact is this: https://unix.stackexchange.com/questions/121013/how-does-lin...

`if` and iterations were two things that annoyed me about bash enough to write my own $SHELL. It's not a transpiler though so I can't realistically use it for production services. However for quickly hacking stuff together for testing / personal use it's been a life saver due to having the same prototyping speed of bash but without the painful edge cases.

Not suggesting it's suitable for everyone though. But it's certainly solved a lot of problems for me.

https://github.com/lmorg/murex

That's false, [ is a bash shell builtin (it's also a coreutils program, but the program is not called unless you specify /usr/bin/[ or use a shell that doesn't have it built-in).
> I recently learned(1) that when you When you write if [ -e /etc/passwd ]; then .. that bracket is not shell syntax but just a regular command with a funny name.

That’s how it used to be. Nowadays, most of the time it is a built-in, not it‘s own executable anymore.

Not sure how another standard would make it better. But sure maybe something that used dash as a backend. It does seems like it really would suck to actually use, if my little experience with Javascript eco system is anything to go on. So different perspectives maybe?
"This happened because the utility used a simple recursive descent parser without backtracking, which gave unary operators precedence over binary operators and ignored trailing arguments."

It is a bit sad that his problem has to be encountered and resolved over and over and over again. Granted, it was a SIMPLE PARSER because resources were extremely tight. Note, I don't mean shells, I mean working with projects that implement parsers. As soon as someone busts out lex/yacc I know I'm going to see at least one "ah, we didn't think of that" issue. Parsers are hard.

While I agree that there can be pitfalls to parsing, most of the time it isn't as bad as this. The "test" or "[" program has a unique challenge in that it never gets to see the quotation marks or variable references, because the shell already expanded them.

So when test sees that one of its arguments is the string "-f", it has no way to know whether that string came from a variable like $var, a quoted string literal like "-f", or just the text -f. Most of the time if you're writing a parser for your own DSL you don't have this problem. You can tell the difference between the string literal "+" and the operator +.

I first learned about this hack while reading a debian init script. Thankfully systemd has removed the need about this!
Init scripts are just executable files.

They can be written in any compiled language you like or even an interpreted language, if you are prepared to add the dependency of an interpreter for that language at boot time.

This is a lot simpler than migrating to (and getting locked in to) systemd.

Wondered for ages whether there is a real situation where this is needed. Good to see this publicly debunked. Now I have a resource to point to when I get the "better safe than sorry" argument.
What's been publicly and thoroughly debunked is the very idea of writing shell scripts in the first place, instead of using a non-ridiculous scripting language like Python, Perl, or JavaScript. If you want to be safe instead of sorry, then never write shell scripts.
People complaining about how underpowered shells are should have a look at Ion [1]. It's mostly the same as bash, but with first-class support for arrays, maps, byte slices, variable scoping, and (primitive) type-checking. Oh, and it's written in Rust.

[1] https://gitlab.redox-os.org/redox-os/ion

BTW Ion is mentioned here along with many other alternative shells: https://github.com/oilshell/oil/wiki/Alternative-Shells
Last release was 3 years ago...
> Amazingly, the x-hack could be used to work around certain bugs all the way up until 2015, seven years after StackOverflow wrote it off as an archaic relic of the past!

Oof. If people think JS is bad wait until they try to do anything moderately complex in shell script

Yup. These days if it's more than a few lines, I write it in Perl instead if it's for personal use, or Python if it's going to be shared (Python just takes me more effort still).

For all the vaunted Unix Philosophy it's amazing how clunky some of it is. Countless shell scripts still trip over spaces in filenames. Of those that don't, nearly all of them will still be confused by any unusual characters like newlines. If you want something done properly, you have to pull out a proper scripting language, to have readdir() and the ability to pass arguments directly to a process.

Even Perl, which generally excels in such tasks has weird lapses in convenience. You can run a command with one line of code without the possibility of confusion with `system("ls", "-l", $dir)`, but you can't get its output that way. There's no version of `system` that'd allow you to both explicitly specify each argument to the process, and obtain its output. You either use backticks and risk quoting trouble, or need to use `open`, which is a lot more verbose.

It's interesting that it took Microsoft to do a new approach in this regard. PowerShell has its amount of weirdness I really hate, such as that the environment escapes the program into the commandline, but it's really refreshing how it dispenses with brittle grep, awk and cut stuff.

To me the question always was what’s the point of [ba]sh for non-interactive shell scripting, when there is perl (I’m aware that it has issues with versions, portability to non-PCs, etc, so think “theoretical standardized lightweight perl mode for sh”, not a real one). Shell syntax and semantics are a pile of ugly hacks and landmines, which were done because everyone wanted to program cli. When you want to program, just take a programming language, not a poor excuse of it. The amount of time you end up wasting with bash is much greater than reading perldoc *, and mastering bash never returns an investment.
My second task at my first job was to connect up a horrible mess of bash, sed, awk, python and R to use a web service as input instead of local files. I ended up rewriting the whole thing to python. It was a good decision.
I still use this hack without thinking about it in case the variable on the left is empty. But I suppose I should actually be quoting.

  [ $X == "hello" ]
When X is empty, it gives:

  example.sh: line 5: [: ==: unary operator expected
So I fix it (apparently wrongly) with:

  [ x$X == x"hello" ]
I'll make a point of doing this instead:

  [ "$X" == "hello" ]
If you wish to support the author of ShellCheck, you can become a GitHub sponsor: https://github.com/sponsors/koalaman

Disclaimer: I am one of the current sponsors.

Amazing how people still put up with this nonsense. Bash really has no place in this age.
I thought it was about empty strings... Going to check that
Any time I have to do any serious shell scripting at work, I die a little inside. I do need to try out shellcheck thoroughly, it might catch some bugs in our pipelines
In my book with zsh being a very popular shell and it having been a bug as little as 5 years ago, I’d almost learn towards recommending it rather than recommending against? It seems like a relatively low impact fix that improves portability.
Seeing or typing x$foo is a reminder that shell scripts are unsuited for anything more than a few lines long. If you're tempted to switch to [[ ]], just switch all the way to a real programming language.
The only good part of sh/bash script is to run commands with piping/redirection support IMO,other than that I saw posts like this to explain sh/bash script’s puzzles/pitfalls all the time.

Maybe it is time to consider using rust instead: https://github.com/rust-shell-script/rust_cmd_lib ?