▲ 192 points
back
167 comments
> For example, a trivial hello world program in Julia runs ~27x slower than Python’s version and ~187x slower than the one in C.
I don't think it makes any sense to speak of "__x slower" for hello-world. Clearly, this is just a benchmark of startup time, so you only pay it once per program. It should be reported as "__ms slower".
Julia startup (according to this post) takes 371ms. That's 357ms slower than Python, and 369ms slower than C. Faster is always better, but this doesn't seem so bad to me.
For comparison, on my old workstation here, starting a Swift repl takes 2724ms, and starting a Clojure repl takes 4792ms.
Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second.
Moreover, even in some server-side applications it's super neat to have the luxury to spawn a new process to service certain requests, not having to worry about memory leaks. It's a perfect "API" which allows multiple languages to interact together.
It always bothered me when people dismiss the start-up time by adding "just" in front of it. We're not talking about web frameworks, these are _general purpose_ programming languages, and horrendous start-up time automatically disqualifies them from being general purpose and places them into a niche category, in my humble opinion.
To be fair, the article says:
> If you ignore startup time, Julia might have good performance for simple array/matrix operations and loops, but we already know how to make them fast in Python and other languages.
> And it’s not just scripts, Julia’s REPL which should ideally be optimized for responsiveness takes long to start and has noticeable JIT (?) lags. What’s even more worrying is that there doesn’t seem to be much progress there. The REPL was a pain to use a year ago and it still is.
> In addition to that, Julia programs have excessive memory consumption. The above hello world example in Julia uses 18x more memory than Python and 92x more memory than the C version.
You can't compare starting up a statically-typed, compiled Swift repl to those other dynamic, interpreted languages -- it's doing a lot of things that running a normal, compiled Swift program wouldn't do, and that dynamically-typed Clojure repl doesn't do (even though it's slower).
The Swift repl is essentially a debugger running a compile-run cycle on each entered expression. Some statically-compiled languages have slow compilers but extremely fast runtime execution (Rust, and to a lesser extent C++, come to mind).
> It should be reported as "__ms slower".
That would be dependent on hardware, while the relation will be constant.
It’s relevant if you are discussing cli scripts. It’s often something I write and usually something I consider.
its a systemic problem, not just startup time:
I want to know what business anything running on a modern computer has taking that much time to start up. Aside from the fact that 5 seconds is an insane amount of time, this is before any user code has been loaded, so presumably your Clojure repl is executing the same 5 seconds of work every time it starts. Can't that be cached?
Developing tools for devs suck as they will use "hello world" as benchmark. And the tech stack will be decided by the non technical founder. Devs are more concerned about what tools others use rather then the pro et contra. /rant
This article is from 2016. The OP probably should indicate that in the title. As far as I have skimmed read the article, some of the claims the authors make have already been resolved in Julia, and I haven't used the language yet, but I will soon in the next couple of weeks.
Indeed, and while there is plenty to still be worried about, a lot has been addressed, too. Further we are now starting to see the pay-off with amazing algorithmic innovations being implemented in Julia.
We have working code to code automatic differentiation:
https://github.com/FluxML/Zygote.jl
Finally, the core team is paying attention to the problems and addressing them. All my top complaints are already mentioned in the laundry list:
Like other people, the startup isn't much of a problem to me since I can easily adapt (and even prefer) a REPL based development and for the stuff I use Julia that one time cost is more than worth it, but the problem of the initial JIT lag is that it affects the first impression of the language, which probably has a significant effect in adoption and good word of mouth from people who didn't stay enough to appreciate the compromises being done.
Julia language development seems to be guided towards avoiding making damaging mistakes on the long run (for example focusing early on having a stable programmable multi-stage compilation, very efficient union-types and state of the art multithreading so the library ecosystem can grow with the correct assumptions). But the "time to first plot" is something that can be solved (regardless of it being hard or easy) at any time "with no consequences", since libraries written with the expectation of a slow startup will work just as well when that problem is solved. Though the compiler team did acknowledge the issue and are prioritizing it post 1.3.
I do wonder if focusing on doing the very simple things well first (like python does right now), would be a better strategy, or it would just make people ignore the language because the simple things are already covered everywhere and not interesting enough for bringing people (compared to the amazing stuff Julia can do with supercomputers/clusters and state of the art stuff like Zygote).
>I do wonder if focusing on doing the very simple things well first (like python does right now), would be a better strategy, or it would just make people ignore the language because the simple things are already covered everywhere and not interesting enough for bringing people (compared to the amazing stuff Julia can do with supercomputers/clusters and state of the art stuff like Zygote).
This is just the difficulty of open source. I am a mathematician by trade, so I work on DifferentialEquations.jl, Zygote.jl, new forward-mode AD, and showcase this stuff in neural differential equations as part of my work. While it would definitely be nice if someone focused on startup speeds, it's not going to be the DiffEq/Zygote people, because it's not the stuff that I/we know. Julia has tons of great scientists and mathematicians, so the libraries in that area are pretty fantastic already, but we do need to find some people who know how to write apps and do devops. I plan to actually find and hire some devops people to help out Julia here. Julia has shown from its libraries that it's worth supporting, so now we should tie a bow around it for less hardcore folks.
,,Text formatting and unit testing are two areas that should be relevant to almost any project''
This is just blatantly false.
I love Julia for statistical analysis.
I never needed speed when printing out stuff.
REPL should be much faster (especially importing libraries), and I'm not a fan of 1 based indexing, but everything else is awesome if you need great performance.
Also I'm missing a great Julia native graphing library with zooming support (Matlab's is far better).
Production binaries are very hard to make, but for statistical research I don't know any better software.
Also for unit testing asserts are good enough for me.
> I never needed speed when printing out stuff.
> REPL should be much faster (especially importing libraries), and I'm not a fan of 1 based indexing, but everything else is awesome if you need great performance.
Then don't you want fast printing / string ops? REPLs heavily rely on that.
> but for statistical research I don't know any better software.
R?
These complaints almost seem like a troll to me.
Complaining about start up times for a JIT implies your numerical work isn’t that heavy (and why not compare it to C compilation times in that case).
That sprintf comparison is ridiculous as it doesn’t show how much assembly follows that jump, it’s just argument packing.
And calling C can indeed segfault.
Exactly. This article is so ridiculous. Benchmarking "Hello world" performance? What a joke.
Lots of the complaints in the article seem a bit contrived or not too relevant. To mention a few:
Performance: It's been mentioned by others that JIT and Julia is a thing. An important point that many here seem to be unaware of is that you can compile your Julia code [0], and then end up with a fast Hello World, if that kind of thing floats your boat.
Language style: This is more or less a subjective thing, right? However, I've found that optimising for JIT forces me into writing short and pure functions. I.e., optimising my Julia code for speed also forces me to write clean code. This is a really nice byproduct of the language design.
Libraries: The complaints seem somewhat thin: there is no mention of the type of unit tests that the author is missing, just a complaint that the library is less featured than some in C++ or Java. Is comparing a pre-1.0 language's unit testing libraries to those of C++ and Java a fair thing anyway? Finding that the generated instructions of a print statement are too long for your liking also does not seem to me to be a fair criticism of the language libraries.
Development: Complaining about the codebase being a mishmash seems unfair to me as well. I find myself browsing source code in Julia much more than other languages. With Julia I can just dive in and generally find that what's relevant to me underneath is also Julia.
These are some strange complaints. As a C++ developer optimizing numerical code, I have never once worried about any of the things mentioned here. I care about instruction folding, vectorization, loop reordering, and efficient instruction emission. If your optimized program is worried about startup latency of 100s of ms, you are doing something very very wrong.
I guess negativity gets clicks.
Julia is really good for that.
There's a macro @simd which lets you write a for loop and it will convert it to simd.
> One-based indexing is another questionable design decision. While it may be convenient in some cases, it adds a source of mistakes and extra work when interoperating with popular programming languages that all (surprise!) use 0-based indexing
The benefit is that it's more familiar to scientists who have experience with Mathematica, MATLAB, R, or Fortran. It's fair to compare the pros and cons of this choice, but you have to at least mention the pros.
The revulsion a lot folks express at one based indexing is always bizarre to me. I write code in c, python and matlab. Switching between these is really not that hard. The two indexing models just seem to be convenient/painful for different things.
And yes, perhaps one based indexing introduces a class of bugs when you need to call into c. But zero based indexing has the same problem if you need to call into fortran, and calling fortran is really common for numerical code.
I have started to learn Julia recently after some following the news. The addition of a debugger finally decided me to give a try. For the record I have been using Python for scientific programming before numpy existed, also a little Matlab too.
Since my experiments are just some simple implementations of kmeans and bandits epsilon greedy algorithms take what I'm going to say with a grain of salt. Anyway:
I find Julia very interesting. I managed to make kmeans fast with type annotations. If I were to summarize I would say that Julia is a much better cython. I never managed for example to debug cython. Profiling also seems to work in Julia, another thing very hard to do in Python. On the other hand, as in cython, sometimes you don't know if some missing type annotation is slowing your program. It remains to be seen if the "verbosity" of type annotations is going to help or slow its adoption.
>sometimes you don't know if some missing type annotation is slowing your program
The lack of type annotations won't slow down your Julia program, since the compiler will infer them anyway (types are for multiple dispatch, documentation or to assert types, not speed). What will affect it is if the type can be inferred or not. For example, if you have a variable that is sometimes an int, sometimes a float, sometimes a string it will force the compiler to put the checks on runtime, dropping performance to CPython level (although the compiler optimizes small unions, such as Union{Int, Nothing} for a nullable Int). That's what the community calls type-stability, and the first step of profiling a function is usually using the macro @code_warntype to see what the compiler is inferring. See:
https://docs.julialang.org/en/latest/manual/performance-tips...
You need types. It's the secret sauce behind multiple dispatch.
The article was written in 2016 (from the URL). Have some of the authors criticisms been addressed in the latest release? I know that startup time for Julia is now much faster (and also faster than my IPython profile from anaconda).
Some have been partially addressed.
But the main point is that the critique is misguided in its generality. If the author cares about running many small scripts that each take a handful of milliseconds, then julia is just not the right tool for his job. No need to write overly general angry posts like "julia is slow"; instead write "julia has sluggish startup time". This is to some extent unavoidable, since julia has a quite heavy runtime (need to load llvm). For some workloads, bash / python / perl are more appropriate tools.
To give you an example, `$ time julia -e "print(5)"` gives me about 230 ms, compared to python 35 ms.
The language is designed for longer running programs that compute heavy stuff. And it performs very well at its intended use.
JIT overhead/startup time is still comparably large. There is https://github.com/JuliaLang/PackageCompiler.jl that helps reduce this overhead in user libraries. The base library precompiles quite a few methods already, so the performance deficit relative to C and Python on Julia 1.2 is half that quoted in the article, and unchanged by statically compiling.
Personally, I do a lot of computational geometry in Julia and I really don't care so much about these kinds of small overheads since actual computation time is the dominant factor. I imagine if Julia was designed for scripting in Unix environments this would be a bigger deal, but I think most people in the Julia community care more about how to manage several gigabytes of data in RAM/cache and run some analysis quickly, e.g. composable multithreading in 1.3.
The article may be old, but as if 6 months ago (the last time I tried Julia), the complaints about the jit were still valid. Typing something into the repl with a syntax error took tens of seconds to produce an error. Creating an array with 3 elements took over a second. Plotting took forever. It was a very frustrating experience.
Here’s the HN discussion from 3 years ago:
This article is both outdated and misguided. Really? You are going to do perf measures on hello world?
I've been writing julia for almost a year. For this purpose, no other language comes even close.
Why is this being posted? It was written 3 years ago, has been discussed before on this site, and the language has changed enormously in those 3 years.
I think this post is pretty outdated... Julia has come a very long way since v0.4.
I gave up on Julia at 0.6 for these reasons a couple of years ago, before giving it another try at 1.1
Unfortunately, while the startup time has improved, I still find the JIT compiler overhead makes it unsuitable for most shorter scripts - the sort of thing where I would normally use python or R. Taking a few minutes to produce the first plot of the day while julia recompiles the plotting packages dependency tree is just a pain.
Edit: After a fresh install of 1.1, I found loading the Plotly package takes 4 seconds from a fresh repl, while the first plot with the Plotly.jl backend (plot(rand(5,5),linewidth=2,title="My Plot")) takes 14s, and subsequent plots take less than 1s on my computer. This is a marked improvement on 0.6, which is where my few minutes came from. Couldn't get PyPlot to work.
I've been working in the data space with R since 2014 and Python/Pyspark since 2017 and I'd love to switch to Julia, but whenever I look up for "production" examples I don't feel safe enough to start a real, paid project on it.
This article talks about either the insignificant (startup time, syntax), the fairly straightforward to fix (FFI, documentation), or both (printf performance). The title of the page implies that these are intractable problems, and nothing in the content suggests otherwise.
As far as Julia criticisms go, the Dan Luu post felt like it focused more on the right things (https://danluu.com/julialang/, circa late 2014). Of the two, that's the one I'd like to see a follow-up for.
I really hate how printf and sprintf are macros. I have no idea why they would have these in the standard library instead of replacing them with function forms.
Julia isn’t yet ideal (but looks like this will be addressed soon) if your code will run in less than 30 seconds in eg Python, but anything taking longer than that would probably benefit from using Julia. So for most machine learning/data science projects Julia may offer a significant advantage over Python or R.
Julia 1.2 release is almost[1] there, with many small improvements. And Julia 1.3.0-rc1 available[2] for testing as well.
[1] https://discourse.julialang.org/t/julia-v1-2-0-rc3-is-now-av...
[2] https://discourse.julialang.org/t/julia-v1-3-0-rc1-is-now-av...
This seems to be a review of Julia as a general-purpose programming language. For example the reviewer points out early that the review is not targeted at people who like Matlab's syntax.
So this reads a little bit like yet another review of a math DSL, written by and for people who don't need or like math DSLs.
Not surprisingly, it comes out negative.
A good opportunity to point out this Juliacon 2019 video:
What's Bad About Julia (https://www.youtube.com/watch?v=TPuJsgyu87U)
TL;DR: Julia is not that bad.
This is subjective but, I've been working and prototyping in a LOT of languages in the past years in various domains. I have to admit that Julia was one of the very few that actually got me excited and love writing in it ^_^. As with everything, it has it's pros and cons - you just have to find the right balance that fits your needs.
What problem does Julia solve again?
you should try Nim.
I like Julia a lot, but the lack of row indices for the data frames is very annoying. I’m used to pandas and saddle and row indices are essential. Can they express anything just columns cannot? No, but they are a useful idea. I just use the Julia pandas wrapper when coding Julia and it works better than the native one. Also the performance is pretty good to.
Also I don’t like the begin/end block delimiters, but it’s not a huge deal. The Julia people keep repeating that curly brackets are too valuable, but I’m not at all convinced.
In addition, Julia should just give up on dynamic typing and make static typing mandatory. I understand why they allow it, but I still don’t agree.
At this point, when I hear about "C-like performance", I already know somebody's trying to bullshit me.