back

by jeffreyrogers·12y ago·view on hn ↗
> kOS is coming. Nothing will be the same afterwards.

There seems to be this strange idea going around that if we just get the right tool, everything else is going to change forever. I see this a lot with people trying to create IDEs that let non-programmers create programs without really knowing how to code.

But the thing is, most people just don't have anything worth coding. The problem isn't that the tools don't exist. They do, even if they aren't perfect. It's that making something that matters isn't an easy thing to do. And no tool can change that.

3 comments
If I am from another planet, and say I don't know why programs are so big and slow and buggy, and the most complicated program you see I've produced is a glorified calculator, it's too easy to be patronising and say well, that's because you haven't done anything complicated.

However if I then show you a programming language, a database engine (similar in capability to SQL but around 1000x faster), a graphical desktop with icons, mouse, editors, filesystems, ISR, and so on- and it's still under 400 lines of C code, then maybe it's easier to have this conversation that I think we need to have: That maybe all you programmers have simply been doing it wrong, and there's something fundamentally wrong with the way you and everyone else programs computers.

I think most programmers take so much offence from the thesis: that everyone programs wrongly (or badly) and that a fundamental shift in the way we program could make programming much better, that it's been difficult to actually work on this problem. Just look at all the people who are complaining about the text editor being difficult to read without saying I can't read this, and I want to get better.

And I think it should be obvious: You've built bridges for thousands of years, so you expect you're pretty good at it now, and yet there are still improvements in bridgemaking today; why do you expect programming to be any different? Why do you have "best practices" for programming if you've only been doing it for a few decades and don't really know how to do it yet?

Maybe it's much easier to think we're working on giving non-programmers the ability to program, but we're not: we're trying to make programming suck less.

I'm hesitantly excited about this language, after reading this article. The power increase I've gained going from an "OO" imperative mindset to functional has been huge. The simple volume of code from other styles just boggles my mind. I can't understand why people prefer their verbose code with so much edifice. Making another leap like that sounds very promising.

At the same time, it does sound slightly off. What's the catch? In the text editor example, how capable is it? Often, such claims rely on a trick, like saying "<textarea/>" is a text editor in 11 characters. Or the tiny Haskell quicksort that's actually rather inefficient.

I think I'll try playing with it. How long should a Logo interpreter in K be?

I wanted to be able to type K and press a button and get the result, so the first bit of k code I wrote was the following:

    np::*((l$[=/k;(*i),0;i]),j)_a;ci:{{J y;x:3:. x;J y;kx x;K(y,j)}[np;j]}
When you type !10 and press control-I it prints the result where the cursor is and selects it. This way you can press backspace to delete the output and change the code easily.

How much code should that take? Probably less than that (I didn't know about bin yet), but I think that's part of the learning process.

I think a minimal logo would only take a couple lines.

that's awesome. It's a one line REPL.
I think Douglas Adams' SEP[0] field is a good description of this. Most people ignore what doesn't fit with their logic - doubly so if fitting it in would show them that they have been wrong/wasted money/wasted time in the past. (The Upton Sinclair quote about "it's difficult for someone to understand something when their salary depends on not understanding it" is also relevant).

It's that way with religion, paradigms in medicine, and even chemistry/crystallography (see e.g. Shechtman vs Pauling[1]). It's human nature.

It is also worth pointing out that while this specific version of K is new, K itself is 20, and it is mostly purified APL, itself over 60 years old. People have been ignoring this since forever (APL found stellar success in the OR community in the 70s and 80s on its own, in trading floors in the 80s and 90s thanks to Arthur Whitney, but has since mostly disappeared)

[0] http://en.wikipedia.org/wiki/Somebody_Else%27s_Problem#Ficti...

[1] http://www.theguardian.com/science/2013/jan/06/dan-shechtman...

Nothing encapsulates PNAS' worthlessness as a journal more than that fiasco.
If the SQL is I/O bound, the kdb version cannot be 1000x faster. And if it's not I/O bound, you're not pushing it right. (Nite: compare columnar SQL here.) K can't make your I/O subsystem faster, and the new OS probably doesn't natively support 99.9% of existing high performance hardware.
The SQL was not I/O bound, per se - that is database/implementation-specific. The benchmarks were evaluated on "hot" data with plenty of RAM available (like any good production system).

Some databases don't manage that situation as well as they should. I.e. - they were developed in an era of small RAM & large disk.

> And if it's not I/O bound, you're not pushing it right. That's exactly the point. k is "pushing it right" while the others don't do nearly as well.

> the new OS probably doesn't natively support 99.9% of existing high performance hardware.

At <500 LOC of ANSI C there's not much to port to new hardware, given a decent C compiler.

>And I think it should be obvious: You've built bridges for thousands of years, so you expect you're pretty good at it now, and yet there are still improvements in bridgemaking today

Right, and I'm not saying programming won't improve. It obviously has. No one wants to write a web app in C++ or assembly. What I'm specifically taking issue with is the idea that there is some sort of monumental change out there that is going to enable us to do... what exactly?... well no one can really answer that, but the claim is that it is big and exciting and will change everything.

VPRI (vpri.org) was already mentioned here. The goal of Alan Kay, as I understand it, to reduce complexity of computer systems - by making them smaller. k shows you potential results with this approach - doing things differently, you can have small and functional systems, so they could be easier to understand while remaining feature-rich.
When I can fit my program on the screen, I don't make any mistakes.

I think this is true of most programmers.

While "hello world" type programs tend to be the pedagogical example, kOS demonstrates that the complexity of such a one-screen low-defect program is much higher than people previously thought.

Not being facetious, but does this extend to using larger screens, multiple screens, wider columns or multiple windows, and smaller fonts? Or was one screen just meant as an estimate for 200 lines or so?
I think it is the act of scrolling or window-switching or head-moving that is the cost. To that end, I find using a MBA screen to be perfectly adequate for programming because it fits entirely in my field of view.

My screen is about 55 lines tall and maybe 170 characters wide. I sometimes shrink or increase the font slightly to make programs fit on screen.

I'm just a regular web and app developer, but would it be at all possible to write a layer on top of k that makes it at least more readable while retaining most of the speed advantage?

To me, part of being an engineer is not taking offense when being told there are potentially better ways to do things, but at the same time I can understand why some people jump to that kind of statement since at first glance it seems like a regression of sorts, especially when the industry has a continuing history of trying to lower the entry barrier. Even though underneath the programming concepts may be revolutionary, people may also be quick to balk at it when it's in such a form (hence the first question).

Industry does try to lower the barrier to entry. At the same time it strives to reach ever-higher tops. APL family of languages have a high barrier to entry - which is of course a problem - but even remaining novice in the language you can be routinely more efficient working with APL than working with another language. So - that high barrier to entry is a disadvantage, but for that you can have pretty high levels of efficiency in the same language going from newbie to intermediate to advanced levels.
I think this "readable" bit is the theory behind q which you can download from kx.com; Some people think it is more readable. q/kdb+ supports websockets and has a built-in http server so it is very possible to make web apps with it.

However I think there is value in the dense coding style that has nothing to do with it's performance: it's that it reduces bugs and helps you think about the problem.

You've sparked my interest, for sure. I'd love to pull it down and give it a whirl. Hell, one could conceivably write those things with K as well.

Thanks for the perspective. Good luck to you all with kOS and others.

Having studied the History of Mathematics, I think that the same discussion/argument must have occurred in Italy when transitioning from Roman to Arabic numerals.

Would you rather do:

IXDCCCLVI * VIIDCCCXLIX

or

9856 * 7849 ?

> similar in capability to SQL but around 1000x faster

If it's really similar in capabilities, why not strap an SQL parser on it and sell it? You can do it client-side to avoid wasting performance where it matters. Surely there's money to be made in the database business if you are an order of magnitude faster than everyone else -- let alone three.

You don't seem to be aware that Arthur did exactly that:

http://code.kx.com/wiki/Startingkdbplus/qlanguage http://www.kx.com/q/d/kdb+.htm http://www.kx.com/q/d/kdb+1.htm

It's a commercial product and I understand kx does pretty well.

I was not. Thank you. What is the source of the performance figures?
http://www.listbox.com/member/archive/107499/2014/05/sort/ti...

The list is subscriber-only, and I apologise that the information isn't very easy to get at.

I don't know, think bigger. The internet changed things forever, and before that computers. Revolutions have been happening at exponentially increasing rates. The explosion of software complexity we see today is perfectly ripe for innovation and maybe even could cause another revolution. The software industry isn't exactly mature when you take a longer time perspective.
Part of what made the internet such a big change to "things" was that it didn't attempt to replace "everything" wholesale. I'm fairly sure that form of the idea was tried -- for example, PC/GEOS -- but the idea that succeeded was the one that allowed the most people to try it out and see what it did.

I'm trying not to add more words to that paragraph, though I could/should.

Not knowing how to code is not really an issue because we can and do teach people how to code the same way we teach people how to count and do basic algebra. What we can't do well is teach people how to solve novel problems that they haven't encountered before.
The reason k, q or other array languages like J (which I use) help, not teach, people to solve novel problems is that they give you abstractions to quickly play with in order to tackle novel problems. My take on this whole thing is that these array languages (APL, k, J, q) are a perfect fit for things that are essentially arrays - big data, images, etc... And to those who shy away from the seemingly 'noisy' syntax, I point them to mathematics. Until you understand the sigma symbols and others, it is noise too. Why are all the new languages getting hot on 'vectorization' - Julia and company - because GPUs and data are all array-based. To have the array as your lingua franca puts you in a good position for all sorts of attacks on modern computing needs. And for those who cannot live with the syntax, and say how can you create a dsl - you can. Here is a J example of computing averages:

  +/%#
You can set this up as:

  avg=: +/%#
or

  tally =: #
  divide =: %
  sumall =: +/
How's that for a dsl? Which allows this now:

  avg=: sumall divide tally
  avg 2 4 6
  4
EDIT: I forgot to add that these operations, as others in J or APL, work on scalars, vectors and arrays without any special typing or handling. See this presentation for at least the first 5 minutes to see J in action with explanation: http://www.infoq.com/presentations/j-language The conciseness allows you to start seeing the small patterns in the small expressions just like mathematics, hence bringing clarity and speed of abstraction to your concept manipulations.
I think part of the difficulty here is that it's almost like Haskell's point free style, but not quite. It isn't really clear where the arguments to avg go. It seems that it's meant to be a bit like this:

    avg list -> (sumall list) / (tally list)
I guess you just need to know how the argument you give to avg when you use it distributes over the functions that comprise the expression. The list argument to tally isn't a problem, but why does sumall get it's own copy of it? Or is the execution model something else entirely?
In J there are two special ways to combine functions which are written using special syntax. Namely,

1) when you want to calculate f(y, g(y)) , you write (f g) y - this is "hook" of one argument (monadic, in J terms)

2) when you want to calculate f(x, g(y)) , you write x (f g) y - this is "hook" of two arguments (dyadic)

3) when you want to calculate f(g(y), h(y)) , you write (g f h) y - this is monadic "fork"

4) when you need f(g(x, y), h(x, y)) , you use x (g f h) y - dyadic fork

5) when you have a train - say, (a b c d e) x - longer than 3 elements, then you consider rightmost 3 functions (functions are called verbs in J) as a single fork - say, f - and then consider (a b f) x . So (a b c d e) x is

b(a(x), d(c(x), e(x)))

If the train length is even, the last operation becomes hook - so x (a b c d) y is

a(x, c(b(x, y), d(x, y)))

You can express any computation as a sufficiently complex train.

A mite of pedantry: the J train (a b c d) is a hook with a fork on the right, and so dyadically acts like

a(x, c(b(y), d(y)))

Roger himself has dismissed [0] hooks as an unfortunate result of J4's myriad train rules, made in the name of tacitable everything, which I lament because for some reason, tacit programming is just so much more satisfying than normally solving the problem.

[0]: http://www.jsoftware.com/jwiki/Essays/Hook%20Conjunction%3F

Yes, my mistake regarding 4-element dyadic train.
That takes some time to digest, but I guess is absolutely crucial to be able to do anything with this language.

Would you say that this is 90% of the reason it looks so uncomprehensible?

No, it's not absolutely critical. You can do a lot of your own programming without using hooks and forks. Your programs will be somewhat simpler - and whenever you need to use a variable in several places, you'll have to explicitly name it - but still.

Incomprehensibility happens in part because all ASCII characters are used - so you have a lot of differently-looking symbols; because [ and { aren't paired with ] and }, and " isn't paired either; because you often have . and : as the second character of built-in entities, so you've got a lot of dots... APL used the symbols invented explicitly for those purposes, but the price - non-ASCII alphabet - was considered too high. So - no, I wouldn't say forks and hooks are the source of 90% of uncomprehensibility.