Item — HN Reader

I get the sense that most engineers would almost never use, and never need to use, most things beyond advanced-beginner in this sheet. It may be fun to brag about knowing how to use "embedded DSL with combinators," but is that really the best thing to help your startup succeed?

I suppose I'm slightly bothered by the fetishizing of challenging knowledge for challenge sake. Most of the people I know who learn about "Embedded DSLs with combinators" and set theory seem to not be able to stop rubbing in how smart they find themselves, and yet strangely they never seem to be the most productive engineers (in terms of delivering useful and reliable code).

I know it can be a little disappointing to feel like there isn't a pot-of-gold at the end of the rabbit-hole, but this is no different than most disciplines. You use arithmetic daily, algebra weekly, calculus monthly, imaginary numbers annually or less.

4 comments

A lot of these techniques are things that take a bit more up-front effort but pay off over the long (or even medium) term. I think that's why it might make people seem less productive—although of course a lot depends on your perspective and experience. Measuring productivity is an open problem in software engineering and, in my experience, people's intuitions about it are all over the board, which means that nobody's any good at it.

DSLs are a great example of this long-term dynamic. Compared to throwing together an ad-hoc library, a DSL approach takes a lot more design effort but, ultimately results in a system that's more coherent, elegant and expressive.

I've worked on both kinds of projects and the difference is palpable: with the first style, productivity is constant at best—getting things together in the first place is a struggle, and then adding features or fixing bugs continues to be a struggle. On the other hand, the second style of project is even more of a struggle at first, but once it works it's like magic: new features are easier to add than you'd expect, and I've had way more things work after my first attempt than anyone has a right to expect.

I know which style I prefer, and it's definitely not because I "fetishize challenging knowledge for challenge sake"—it's because I'm willing to put effort up front for a long-term reward. And it's not even that long-term—I've found these things pay off over weeks or months, not years, so I'd take the same deliberate approach unless my deadlines were literally days away.

Well, I can't speak to your career experience, but I can speak to mine.

My experience has been (31, SF) that companies are fundamentally disorganized, frequently reinvent things (a behavior seen at all levels), are skewed by politics, and ultimately are rarely successful due to the code. The fact is, that for most startups it's not the code that's make-or-break (twitter, snapchat, facebook, airbnb, uber), it's the business execution.

It's been my career experience that the only way to be a 10x engineer is to prevent management from engaging in unnecessary projects (Bob wants to rewrite X in node, Joe wants to make a service that only has the responsibility of CRUD to 1 table, Sally wants to move it all to noSql). It's been my career experience that soft skills give the best ROI.

So when I see a list like this, I find it hard to imagine how "Profunctor Optics" is what Zynga (or any company I've worked at) needed to be successful.

My experience has been (34, Earth) that companies fail to succeed for a variety of reasons. Some of those reasons are poor technical choices. And some companies succeed in spite of said choices.

What functional programming brings to the table, aside from dense jargon, is the tools to build systems that are _correct by design_ and have certain, provable properties. For businesses this means they can spend less money fixing errors in their software and avoid losing revenue if they gain a reputation for releasing unreliable software. For programmers it means focusing on delivering instead of fussing around with runtime type errors, deadlocks, and the like.

Where this is useful is reducing the risks associated with failure: _when_ your software fails, what is the worst that could happen to your users or your business? If the answer is, "well some people might see the wrong blog article or have to re-submit their comment" then you have your answer. If your system is handling orders on a trading platform where an error could cost someone a few hundred million dollars... well it might be worth the effort to eliminate the possibility of as many errors as possible by using a better tool to help you with that.

The success of some companies in spite of not using _strict_ functional programming languages doesn't disprove anything the FP zealots have been saying for years. It only demonstrates how much money and time we invest in absorbing the cost of developing and operating software with innumerable, unknown errors.

>So when I see a list like this, I find it hard to imagine how "Profunctor Optics" is what Zynga (or any company I've worked at) needed to be successful.

I mean, by that criteria, why learn anything?

The world is not just startups btw, I work on products that have very very defined requirements. Hell, sometimes I'm implementing an RFC. FP is a huge win for us, and yes i've used profunctor optics in a high performance network application.

>I mean, by that criteria, why learn anything?

This might be tongue in cheek, but I agree with the parent. Engineering excellence will not further your career much, companies tend to oversell meritocracy. Salaries flat out quickly, and there is a huge supply of good enough developers, if you step out of the line too much.

> Salaries flat out quickly

What other professions besides lawyers or doctors regularly have a 3 or 4x range for salaries of people more or less with the same job description?

In addition to those? Actors, musicians, sport people. In the office? Managers. Hence the advice, focus on soft skills. I would wager the top 1% on the developer track earn nowhere near to the top 10% on the managerial track. And your best bet is to regularly change jobs otherwise good luck getting anything more than a 5-10% increase, while managerial salaries and bonuses scale with the size of the organization. Also, there is a lot to be said about the difficulty of the work.

I have had a number of projects that were absolutely successful from a technical standpoint because of FP methods. No one knew except team members that were unfamiliar with the techniques. While they were not full converts, many "saw the light" and reduced mutation and wrote purer functions after that.

The business itself was oblivious to why the project was successful. Or how we were able to extend it so quickly and retain stability.

Even on that project, 90% of the difficulty was social/political. Just as Java allowed average programmers to write above average code, the same thing can be said of FP.

> with the first style, productivity is constant at best—getting things together in the first place is a struggle, and then adding features or fixing bugs continues to be a struggle. On the other hand, the second style of project is even more of a struggle at first, but once it works it's like magic: new features are easier to add than you'd expect, and I've had way more things work after my first attempt than anyone has a right to expect.

I would like to challenge that by pointing out that not a single piece of large software (except maybe a compiler or two) has ever been written using the skills/concepts of the higher levels of this chart. What you say may be true if generalized to better upfront design etc., but there's also no evidence that this is better done using the approaches discussed. It's not even about measuring productivity; it's about arguing over the properties of an empty set. When it comes to small (<100KLOC) and non-large (<1MLOC) programs, then the difficulties are not that hard to begin with, and depend mostly on the essential complexity of the problem, rather than abstraction/code organization etc..

> it's because I'm willing to put effort up front for a long-term reward.

You're equating upfront effort with a particular choice of technique, and one that has never been put to the test.

Again, I answered this for you last week. Both Standard Chartered Bank and Barclays have many millions of lines of very successful production Haskell code. Both codebases are solving complex problems not solved at other banks and both make extensive use of types and purity. We do not regard pure functional programming as "aesthetics".

You are not addressing my point at all, and millions of lines of code (that include a Haskell compiler) do not necessarily make a large software system. If you are building software that is significantly more complex than could be achieved by other, simpler means, or you're building comparable large software systems for significantly cheaper, you'd only do Haskell and the world a great service by collecting and publishing relevant data. Without it, I have no idea how big your systems are, how complex they are compared to other projects, and how costly they are to build and maintain. I have no doubt you're happy with Haskell, but there are plenty of far more mainstream languages that people are happy with. Without any data, we can't make any comparison, so claims of superiority are nothing more than just claims. I find it puzzling that you don't see my reaction as a reasonable one for a marketing campaign that has been going on for nearly two decades, that has so far produced no evidence and negligible adoption. I truly and honestly want to be convinced. I am familiar with the theory of FP (up to about monad transformers, but not profunctor optics) and have programmed in SML. I just don't see any signs whatsoever that adopting Haskell can improve the bottom line significantly enough to be worth the cost and the risk, so the most logical thing to do is to wait and beg people for data.

You said:

"I would like to challenge that by pointing out that not a single piece of large software ... has ever been written using the skills/concepts of the higher levels of this chart."

To which I gave two counterexamples (today and on other occasions). All software systems should be modular to some degree, so I am not clear exactly what your criteria for a large software system is. In both my examples, the Haskell codebases are monolithic repositories where everything is typed-checked and built together.

You keep asking for quantitative data for a comparison with other languages. But my answer is the same as last week. It's good people that make software efficiently and cheaply. Give engineers PHP and they'll still manage to build something good. Haskell is just a tool, but it's a tool that increasingly good people are asking to use. The system at Barclays and Standard Chartered were built and are currently maintained very cheaply because good people were hired. Haskell just happened to be their preferred tool.

> I am not clear exactly what your criteria for a large software system is.

Let a "software system" mean any assembly of processes that communicate to provide some shared functionality, with the components being coupled to one another in some non-trivial way (i.e., there are correctness conditions that cross processes boundaries, so that changes to one program may necessitate changes in others). Excluding the compiler, how many lines of Haskell code do you have in your largest system (using my definition)?

You just described the entire bank! I prefer my single codebase/single build definition. I forget the current figure, but StanChart has at least 3M lines of (dense) Haskell. More details here: https://skillsmatter.com/skillscasts/9098-haskell-in-the-lar...

>out that not a single piece of large software

What are you talking about? I know for a fact this is untrue. How do you continually justify making up things and passing them off as facts?

> I know for a fact this is untrue.

Why, because I missed a point of data in the ocean of hype and there is one? Two? Please, set the record straight, and provide us with some facts. Maybe in Haskell's 20 year history as the world's most hyped language there was one or two or maybe three non-compiler programs written in it that aren't very small. Maybe there is even one anecdote out there with some actual information in it, although I've looked for one -- a lot -- and couldn't find any. So right now, the people who are making stuff up and passing them off as fact are those who claim significant impact over and over and over without a shred of even the tiniest of anecdotal data. How is that justified?

I have absolutely no problem saying that Haskell works better for large interactive software than other languages, once there are a couple of anecdotes around, but the fact is that currently there aren't any (at least none that I could find). What we have at the moment is a lot of vague claims with zero metrics.

But perhaps I should explain the source of my skepticism. First, it stems from the almost unprecedented gap between hype and evidence. I would imagine that after so many years, given the grand claims there would at least be some good anecdotes. That there aren't any, inspires skepticism. Someone here likened Haskell's abstractions to load bearing materials as opposed to other language's mud. If that were true, the reasonable prediction would be to see Haskell skyscrapers towering over a sea of mud hats; that we actually observe the opposite, inspires skepticism. Second, it stems from my general skepticism (based on CS theory and 20 years of experience) towards the impact any language can make. I.e., I have not encountered a case where the choice of language was the determining factor. Haskell is perhaps the most notable example of a language that claims to make a significant difference by virtue of language-level features (as opposed to runtime features, like GC). I would very much like to see how big that contribution is, if it exists at all.

I'm not talking about Haskell, i'm talking about Scala written in a functional style where we've used a lot of the concepts from Haskell. We're using it at Verizon for extremely large projects and it's working quite well. I know you've been pointed that out by others, so when you say there's no anecdotal data, how can you justify that? I've talked to others who work at extremely large corporations finding success with it too in extremely large projects.

I am not looking for anecdotes that it is possible to write large programs in a pure functional style. I know it is possible. But, given the high cost of the approach (training, new libraries, maybe a new language and even a new platform) I am looking for anecdotes that the approach provides benefits that significantly outweigh its cost. I have not found any.

BTW, "extremely large projects" are anything above, say 20MLOC. What projects of that size have been written in pure functional style? What large projects (>5MLOC) have?

> not a single piece of large software (except maybe a compiler or two) has ever been written using the skills/concepts of the higher levels of this chart.

I wonder: to what extent this is because those higher level concept tend to shrink programs in the first place? I mean, big is bad, there's no debating this. Big programs can only arise from necessity or stupidity. And if those fancy features are any use, they must be reducing the need for big.

I would ask myself a slightly different question: what kind of program still have to be big, even if you have all these fine concepts at your disposal?

And of course, big is risky, and that tend to make us chose more conservative options. Why use Haskell where C++ has shown in the past it could do that kind of job? Not to mention the network effects: even if Haskell as a language was better than C++ as some specific job, you may still choose C++ because of libraries, commercial support, and developer availability.

> When it comes to small (<100KLOC) and non-large (<1MLOC) programs

Oh, so that's how you're calibrated… My, to me, small means <10KLOC. 100KLOC is already big, and 1MLOC is gigantic. Besides, I've seen a couple multi-million lines programs, and I don't believe for a second they had to exceed 100KLOC. Such programs are more about piling historical accident on top of historical accident than about encoding a complex problem domain.

A small compiler takes a couple thousand lines of code. With the proper tools, it can often be squeezed into 2KLOC (Source: the STEPS project from http://vpri.org). It won't optimize like GCC, but you rarely have to anyway. Now you have a DSL for writing an executable specification. What kind of specification is so complex that it cannot even fit in 50 books?

Of course you have to keenly understand the problem domain to pull that of, and that often means solving it the crappy way the first time around. Which means you're never going to rewrite it the proper way: it would be way too risky. So of course it has seldom been put to the test. One does not just hinge an entire business on original research.

> You're equating upfront effort with a particular choice of technique, and one that has never been put to the test.

No, not never. I have at least one example: one of my uncles once had to write a number of database transactions. It was one of his first job. He had 1 year to do it. Seeing how tedious it would be, he first though about the problem, then devised a DSL in which he could write the damn transactions. Since he wasn't exactly senior, and had crappy tools (he used B), the DSL took him about 6-7 months to perfect. (During which management were on the verge of panic, because no transaction has been done yet.)

At the 8 month mark, all transactions were done, tested, and had a surprisingly low bug count. The client was very pleased and the contract was renewed for another batch of transactions. Same size, but to be done in 8 months this time. That was given to a co-worker, who used the my uncle's DSL to perform 8 months worth of contracted work in 1 month.

DSLs sometimes work.

> to what extent this is because those higher level concept tend to shrink programs in the first place?

None at all. No one, AFAIK, even claims a reduction of even a single order of magnitude in code size.

> I would ask myself a slightly different question: what kind of program still have to be big, even if you have all these fine concepts at your disposal?

All those that are big now. I'm not talking about a single executable, but about an entire system (specification, development and debugging have always been done modularly, regardless of whether there's a single process or a distributed system). I see no way to implement the requirements of, say, an air-traffic control system, complex avionics or a banking system in software that isn't very big.

> 100KLOC is already big

A mid-sized enterprise software system is ~5MLOC. Google and Facebook have codebases that are measured in the hundreds of millions LOC. You can use whatever definitions, but if you look at software actually being constructed, much (if not most) of development effort in the industry is systems that are about ~5MLOC.

> What kind of specification is so complex that it cannot even fit in 50 books?

My guess? The majority of software written today is part of such specifications. I once worked on a medium-sized air-traffic control system designed for a relatively small area and number of planes, whose informal functional specification was ~10 books. The specifications for the avionics software of a fighter jet developed in the 80s were ~2000 pages of structured natural language (source: http://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.H...)

> No one, AFAIK, even claims a reduction of even a single order of magnitude in code size.

http://vpri.org claims about 3, though most of it is not because of the languages, but because of the removed redundancies. The languages do seem to be responsible for at least 1 order of magnitude.

About the rest, of what you say, I won't claim anything. It just make me feel… uneasy. Okay, those specs are that big. Do they have to, though? I've seen a big fat list of requirements in my last gig, and many were duplicated into 2 slightly different versions. As were some pieces of the code. And that's the obvious stuff. There were more subtle waste, where simpler alternatives would have fit the bill, but weren't applied because of historical reasons (I asked the architect, there was a reason for everything).

Almost everywhere I look, I see a wasteland of useless code, and even the specs aren't that clean to begin with. It feels like proper DRY alone would have reduced the size of this stuff by 2 or 3. Maybe I was unlucky enough to work in especially crappy environments. But from what I hear, that's the norm. So far, you're the only one I met that challenges that perception.

> The languages do seem to be responsible for at least 1 order of magnitude.

Compared to what? C? I'm not talking about C, but about any modern language.

> Do they have to, though?

Yes. Or, at least, everybody (including me when I first saw them) says "this cannot possibly be this complicated", and after understanding them, everybody says, "oh, OK". But let me put it another way: if the modern world really only requires simple software, then our work is nearly done. Some would use Python, some would use Java, some would use Haskell -- but if the largest software system needs to be ~100KLOC, then none of it matters too much. Writing such a piece of software isn't hard regardless of what language you use (or, rather, the difficulty is in the essential complexity; there's not much accidental complexity in 100KLOC). Even assuming one methodology would be 15% than another, it wouldn't make much difference to the bottom line because producing 100KLOC software is cheap anyway. And if that were the case, investing in programming languages would be even a bigger waste, as investing in simplifying specifications would have a much bigger impact, and would cost a lot less.

But just to get a sense, GHC -- which is a compiler, and, as you've noted, compilers tend to be small -- is about 400KLOCs of Haskell. The Linux kernel is over 15MLOC of C. Reduce that by an order of magnitude, you still get 1.5MLOC, and that's just for an OS kernel.

> So far, you're the only one I met that challenges that perception.

I don't argue that there isn't a lot of waste. I argue that even without all that waste, we'd still need software that's very large (or that, alternatively, waste is unavoidable). There are no signs that Haskell reduces this waste at all, or that it dramatically reduces the size of programs.

> Compared to what? C? I'm not talking about C, but about any modern language.

The domain they tackle is personal computing, which means Kernel, Windowing system, remote communication (web/mail), multimedia… So, yeah: mostly C and C++, by the look of currently popular programs.

That said, much of the (apparently) needlessly complex stuff I have seen was written in C++, and it did look like they didn't have the real-time requirements or resources constraints that would justify the use of such a monster of a language.

> Or, at least, everybody (including me when I first saw them) says "this cannot possibly be this complicated", and after understanding them, everybody says, "oh, OK".

I have yet to reach the second stage. And on one occasion, I did reach a reasonable understanding of the whole system. It definitely had to be very complex to match the specification, but the specifications themselves didn't match the end user's needs.

> (or that, alternatively, waste is unavoidable)

That's the alternative I'm most scared of. I cannot comprehend unavoidable waste, but I can't rule it out either.

Alternatively, I've came across the idea of not solving some problems¹, because the return on investment is just crap. Okay, when safety is involved, you probably cannot do that. Still, the idea that the 80/20 rule is sometimes more like 99.9/0.1 is enticing. Sometimes, full automation is not best. For instance last I checked, the best Chess player ever is a human-computer team, not a computer.

[1] Stop Over-Engineering https://www.youtube.com/watch?v=GRr4xeMn1uU

> if the largest software system needs to be ~100KLOC, then none of it matters too much. Writing such a piece of software isn't hard regardless of what language you use

I think this is absolutely the difference between you and proponents of Haskell. Personally I have found working on 100k LOC codebases very hard in Python and easy in Haskell.

Two potential counter-examples to your productive software claim:

Opaleye [0], an Arrow-based [1] DSL for Postgres SQL that allows the user to create type-safe, composable, and (generally) optimally fast SQL queries.

Halogen [2], a type-safe frontend framework for PureScript [3] that models view component interaction as an algebraic datatype containing query actions (among other fairly advanced concepts).

If someone is used to a certain level of abstraction, then the higher levels might seem unnecessary until they invest the time into learning them. The only thing unique to functional programming is that it has inherited a lot of terms from theory (profunctor, monad, algebra) rather than inventing friendlier ones.

[0] https://github.com/tomjaguarpaw/haskell-opaleye

[1] https://www.haskell.org/arrows/

[2] https://github.com/slamdata/purescript-halogen

[3] http://www.purescript.org

> Two potential counter-examples to your productive software claim

How are those counterexamples? Do they provide benefits that significantly outweigh their cost?

> the higher levels might seem unnecessary until they invest the time into learning them

Except that this can't go on forever. Because we know that programming productivity has a theoretical upper bound, therefore there must be diminishing returns, and at some point the cost of more abstraction would outweigh the benefit. We just don't know where that point is. Finding out is not a matter of faith or even personal feeling, but of actual results.

Now, I know that collecting meaningful productivity data is hard, but that evidence of a claim is hard to come by doesn't make that evidence any less necessary for the claim. You're free to say that you like FP because you enjoy it more, or even that it makes you feel more productive. But you can't make actual empirical claims without actual evidence.

> The only thing unique to functional programming is that it has inherited a lot of terms from theory (profunctor, monad, algebra) rather than inventing friendlier ones.

I'm not entirely sure that's the case. FP is often its own theory. I'm not sure many of those concepts were studied heavily outside the context of FP, but I could be wrong about that.

Profunctor and Monad are heavily studied in CT and many related mathematical fields that use category theory. For instance, algebraic geometry and logic. Algebra is incredibly well studied under the name "abstract algebra" and is one of the core concepts in modern mathematics since Bourbaki.

But this is just a side point to what you're saying.

I don't understand why this doesn't have more votes. Asking for evidence is good, not bad. It isn't in any way self-evident that FP increases productivity. That's not to say that I don't like it, on the contrary.

This is not about being smart. This is mostly about low defect rate and ease of modification and maintenance.

Hopefully the smarter bits van be factored out as easier-to-use libraries.

> "embedded DSL with combinators,"

I agree that this wording makes it sound pretentious, but combinators are actually a very useful, practical technique. Parser combinators, in particular, are awesome.