▲ 554 points
back
155 comments
I was privileged to be one of the technical reviewers for this book. There's a fair bit of the original content (which is still great), but Kernighan's done a great job with some good restructuring and some significant updates, too. The early chapters are very hands-on, with something of a focus on "exploratory data processing", particularly with CSV files. Big data with AWK, you could say.
Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode (parsing files with quoted and multiline fields per the CSV RFC). I'm really glad Arnold Robbins added a robust "--csv" implementation to Gawk, too, because that's really the most-heavily used version of AWK nowadays. I've already got CSV support in my own GoAWK implementation, and I'll be adding "--csv" to make it compatible.
I'm really glad this new updated version is coming out!
Its a crying shame we never settled on a control character separated text format. There's a ascii control characters for record and field (unit) separators. A bit of user space support for that would have been great.
> Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode
Awesome!!!! Super excited to see this!
Awk is really great, for those knowing nvm [1], I used awk to make `nvm ls-remote` run more than 10 times faster [2] by replacing the related shell script with around 60 lines of awk script [3], and I was quite happy with the improvement.
It's not really a one-liner, neither something big, but one can take that as an example regarding that awk is really not just for one-liners.
Meanwhile having `--csv` support is really nice. I'd also like to see things like a builtin `length` function to be standard.
[1]: https://github.com/nvm-sh/nvm/ [2]: https://github.com/nvm-sh/nvm/pull/2827/ [3]: https://github.com/nvm-sh/nvm/blob/9a769630d7/nvm.sh#L1703-L...
Our data product is delivered in CSV format. Even though I create user documentation mainly using csvkit, grep and sed, I would love to convert all those solutions to AWK. Sometimes AWK is more readable than sed and csvkit requires installation.
It will be nice to have a awk cookbook for CSV. In terms of CSV maniupulation and querying there is only a limited number of operations and I think there is potential to standardize those operation using AWK.
Ben is not just any old technical reviewer. He wrote a version of AWK in go and has done a ton of other work in the AWK ecosystem.
It's nice that everyone is supporting this, I've written a portable awk module that takes control of the parsing and it is SLOW (and a little buggy). I'm a little bummed that nobody will use it but this is truly a step in the right direction.
I guess for the people that are still using nawk, you can set up an AWK envvar so you can { awk -f $AWKU/ucsv.awk -f <(echo '{print NR, $1}') }
https://github.com/Nomarian/Awk-Batteries/blob/master/Units/...
Would you say the first few chapters are enough to get the 75-80% usefulness for mere mortals like me who will never try to master the full language? Or is the material fairly sprinkled throughout the whole tome?
Fantastic news. I’ve tried lots of new CLI tools but they always seem to fall between too little functionality (eg. xsv) and too much (VisiData). AWK is just right.
This is amazing, I may never use pandas again
Awk is awesome! Glad that they are looking to modernize the book. It wasn't really necessary, all the code examples in the original edition of the book still run just fine, although some are somewhat dated, like printing ASCII bar graphs. They also had examples of writing VMs, parsers and interpreters in the book, which run on modern implementations.[0]
The language has some quirks. To declare temporary variables, it's common practice to add extra arguments to functions that won't be used. And traversal of associative arrays is implementation-dependent. I'm not sure what the situation is regarding locale and UTF-8 support.
EDIT: Looks like Brian Kernighan added Unicode support last year.[1]
[0] https://github.com/siraben/awk-vm/blob/master/vm.awk
[1] https://github.com/onetrueawk/awk/commit/9ebe940cf3c652b0e37...
What would you suggest as an alternative to printing ASCII bar graphs? I do that all the time. Takes 20 seconds and often makes distributions, modalities, and patterns over time obvious right away.
Is there a particular benefit in writing a VM in AWK, placed in a big BEGIN block? Very similar code can be written in Perl or Python. Isn't the strength of AWK in its line-matching capability, being able to pattern-match a line against a block of code?
On wm's, why not a Z-machine? It's ideal for this.
I love awk. It’s everywhere and every time I am writing a shell script and work myself into a corner, awk has been the way out.
I know exactly enough to be dangerous and have meant to deep dive for almost a decade.
awk can be mastered by just reading the man page. The book doesn't take long to read either. Once you understand the simple principles, you can write an infinite number of scripts for all kinds of tasks.
See, when I'm writing a shell script interactively and work myself into a corner, I reach for awk, struggle with it for a bit, and then either:
1) succeed, and regret the messiness of the solution
or
2) fail, and find a non-awk way to handle it.
I really tried to like awk, but its portability hasn't been enough of a feature to raise it above other scripting languages for me. Especially if I'm going to end up in an editor
Thanks for your work! Awk is a rabbit hole.
"Dark corners are basically fractal - no matter how much you illuminate, there is always a smaller but darker one." - - Brian Kernighan (quoted in the GNU Awk book)
Awk has always been a language that I loved but I have struggled to use besides quick jobs for parsing text files. I understand it is meant to be use for exactly that, but the fact that is simple, fast and lightweight sometimes makes me want to do something more with it, but when I start trying to do something besides parsing text I find that it starts becoming awkward (pun intented?).
> but the fact that is simple, fast and lightweight
I see awk as a DSL to be honest. Yes, it can be used as a general purpose language, but that quickly becomes, as you say, awkward :D
Like many DSLs, it is simple, fast and lightweight as long as it is used for it's intended purpose. Once you start using it for something else, these advantages evaporate pretty quickly, because then you have to essentially work around the DSL design to get it to do what you want.
I find it pretty nice for writing simple preprocessors. For example I have one which takes anything between two marker lines and pipes it through a command (one invocation per block). Awk has an amazing pipe operator which lets you do something like this:
... {
print $0 | "command"
}
And of course, no mention of awk is complete without the "uniq" implementation, which beats the coreutils uniq in every way possible (by supporting arbitrary expressions as keys and not requiring sorted input):
!a[$0]++This is exactly why I moved from AWK to Perl for these quick jobs a couple of years ago. If you stick to an AWK-like subset, Perl is also simple, fast and lightweight. If you want to grow your scripts (and you have a lot of discipline) Perl – in contrast to AWK – gives you enough noose to hang^W^W^W^Wthe tools you need.
I have found a handful of unconventional applications for awk -- I once needed a tiny pcm pulsewave generator, and awk was surprisingly decent for the job [1].
Aside from that I've mostly been using it for quick statistics [2], but it quickly moves into perl territory...
1: https://github.com/9001/asm/blob/hovudstraum/etc/bin/beeps#L...
It's a language for creating quick alternative views from line- and column-oriented text streams. That means, take the output of another tool and represent it in a different way.
Ok, dumb question: Is the link supposed to link to the actual book (i.e., is the book free and/or open source) or is this just a page of miscellaneous interesting links about the book (which we can pay for, later, when it's published).
I was expecting the book, but the page itself says "This page is a placeholder for material related to the second edition of The AWK Programming Language."
It's fine if this is a placeholder page (and an awesome excuse to read talk about AWK here on HN :) ) but I want to be sure that I'm not missing the book itself.
Amazing, takes me back.
~
One of my first big projects at my first job fresh out of college was using sed & awk to semi-automate the transformation of semi-unstructured data into a database.
IIRC I couldn't completely automate because it contained author names, from global naming conventions. (parsing names correctly is deceptively complex) They had somewhat arbitrary #'s of initials ranging from 0-3.
Again, IIRC, I could easily accommodate 0 or 1 initial (followed by \.) but trying for more would make the regex I was using too greedy and pull in part of the article abstract. These were scientific books and journals.
So I scripted a sed & awk program to detect the possibility of > 1 initials and when that occured, I'd pipe the record into nano for a quick review where I manually inserted the correct \. characters for the initials.
It was decades of back-catalogue publications for digitization so I sat there for days, listening to music on an original 1st gen iPod, waiting for my duct-taped kludge of a program to pipe one of thousands of records into a nano session every few minutes. This was on an Apple G4 workstation running OS X, where I earned my real bash scripting chops. It was an awful hack by today's standards, but at the time, accomplishing what was expected to be a 1-year long project in ~1 month, it was seen as nearly miraculous.
I know lots of people like awk, but I pretend it doesn't exist. Why? Here's my comment on this from 6 years ago[0],
>I used awk until I learned Python (long ago). For me, awk was yet another example of the "worse is better" approach to things so common in unix. For example, if you make a syntax error, you might get a message like "glob: exec error," rather than an informative message. "Worse is better" is probably a good strategy in business and for getting things done, but still, mediocrity and the sense of entitlement that so often goes with carelessness, sickens me.
[0] https://news.ycombinator.com/item?id=13457265
Long live the Unix Hater's Handbook! (Unix is fine, and so are the criticisms herein. Some of these criticisms have been eclipsed by ongoing development.) https://en.wikipedia.org/wiki/The_UNIX-HATERS_Handbook
Seems like the best time to ask since this is an awk thread: if anyone has a line on the original artwork or a source for the awk t-shirt please let me know. From memory it's of a gangly bird jumping / parachuting from an airplane (DC3?) and captioned with awk's infamous catch-all error message: "Awk: bailing out near line one".
Currently looking @ alternatives (not that I dislike AWK, far from it):
Have to wait, as "The book will be available by the end of September"
See https://hn.algolia.com/?q=The+AWK+Programming+Language for discussion on the first edition
Didn't know there was a list of `awk` implementations: https://www.gnu.org/software/gawk/manual/html_node/Other-Ver...
One of the first utilities I had to get to grips with way back was awk, and it serves me well to this day. Best bang for buck investment of time in my entire career. Even today I still use some variant of awk -F(x) '{print $x}'.
This is good news, because you have to pay a lot for a used copy of the first edition nowadays. I hope the spirit remains the same as in the first edition.
I read the first edition so many times as a young kid... AWK was just such a cool name when I would go to the library and grab a book out of the stacks trying to learn something new.
I don't know about Awk, but I feel the urge to write a library named "ward" for it.
Also watch his recent interview on Computerphile: https://www.youtube.com/watch?v=GNyQxXw_oMQ
And: Brian Kernighan adds Unicode support to Awk https://news.ycombinator.com/item?id=32534173
I wish awk had support for addressing a range of fields, like from $1 to $7. `cut` supports it, FWIW.
Awk is old but great, designed to chew through lines of text files with ease, and has great defaults that minimize the amount of awk code you actually have to write to do anything. It's underrated.
I love using Awk, the only thing I miss is that it can't handle complex csv files. Does anyone know how to handle quoted CSV strings like
> "foo","bar,baz"
I FINALLY started learning awk in the past couple weeks. I think I was intimidated because awk can be very terse, and there are some default actions that aren't clear when you first start looking at awk scripts.
My other problem is that I want to accomplish things, not learn a tool, and it generally takes me a bit longer than it should to decide to actually learn something and not just hack at it.
Is it still worth it to be "the awk guy" at work?
Who wrote the second edition?
I wish I use awk all the time but everytime I use it the knowledge I gain doesn't stick. Could be due to its arcane syntax which is just too hard for me to remember.
Wow, hyped for this.
I picked up this little book from my University library once, and it was a fantastic read.
Awk was great at its time, but when you need at write more than 5 lines of awk code please consider using python since
1. It is a lot faster than awk/perl/grep/sed combos
2. Way a lot readable and maintainable
3. More powerful than awk with it's string functionalities
4. Same availability as awk in OSs since last decade
Find and AWK together, a match made in heaven. Thanks for the link.
I love the csv-mode. It obviously takes some time
I am looking forward to this coming out.
Can I preorder this?
Awk and ChatGPT are best friends.