▲ 149 points
back
27 comments
My group is developing Clasp (github.com/clasp-developers/clasp.git). It's a Common Lisp implementation that interoperates with C++ and uses llvm as the backend. It's something you might want to look at because it could change your options in the following ways:
(1) Clasp interoperates with C++ - C++ libraries like graphics and physics engines can be easily exposed within Clasp Common Lisp. (See https://github.com/clasp-developers/demo-expose-bullet-physi...)
(2) Clasp uses llvm as the backend - it generates llvm-ir from Common Lisp code and uses link time optimization to link C++ and Common Lisp generated llvm-ir.
(3) Clasp uses the Memory Pool System garbage collector, which can be configured to minimize GC pause times to a programmer-defined level.
(4) Clasp generates debug info and can be profiled with standard tools like dtrace and C++ functions, Common Lisp functions and GC functions are all profiled against each other.
I'm a professor of Chemistry at Temple University - I'm developing Clasp to design molecules and materials - but it could be applied to a lot of other problems.
Caveat: Clasp is not yet as performant as Steel Bank Common Lisp, but it uses the open source and highly configurable Cleavir compiler developed by Robert Strandh and we have incorporated inlining and are currently incorporating type inference.
If you want to see a little bit more - I have a 25 min talk up on youtube that I presented a couple of weeks ago at the 2018 llvm developers meeting (https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be)
Does clasp still support all the metaprogramming features of lisp while using llvm, and how does it do it?
Given that clasp is a Common Lisp - that passes 97%+ of CL ANSI tests on par with other CL implementations like SBCL - then answer is obviously yes.
You need to go into more detail about why you think llvm would make a difference. Common Lisp is usually compiled to native code and the compiler is - unlike most other languages that compile to native code - exposed and available at runtime. When you program interactively in a natively-compiled Common Lisp implementation like clasp or SBCL or ClozureCL, your code is compiled to native code on the spot. You can even customize the code generation via compiler macros. LLVM is just another compiler backend that can slot in so in that regard, conceptually it's seamless.
Well, there are a lot of those, they need to be looked at separately.
What I (and the QPX school if I might use the term) use is compile-time computing, aka a programmable programming language. LLVM never gets to see this, by the time you pass it to the compiler it is dumb code. QPX after macroexpansion looked like C.
When it comes to executing code that only becomes available at run time (repl, eval, from a string, from a file etc) you generally just kick off the compiler, which is that Clasp and LLVM do right now. You can also use the interpreter which again both Clasp and SBCL have.
When it comes to the dynamic nature of CLOS method invocation, that is generally accelerated with special dispatch mechanism that live in the compilers. You can implement CLOS without any compiler support, but teaching the compiler about method dispatch makes it faster. The Lisp compilers emit abstract assembly code for LLVM or the last stage of SBCL.
When talking about dynamic typing the same applies. The compilers that do Lisp emit abstract machine language, pretty much the same way for Clasp and SBCL.
To Clarify, Clasp uses various LLVM compilation targets including LLVM's "JIT" for runtime known code. I am not sure I'd call it a JIT because it doesn't compile functions when they are used, but they are compiled into memory (as opposed to into object files).
Hope this helps.
Hi, I watch your llvm talk but, correct me if I'm wrong, it seems you didn't really have enough time to cover all you wanted. Do you think you'll have another conf to do it fully ? or maybe find time to do it on another video platform ?
oh also, someone on your talk main thread suggested to use more GPU (https://news.ycombinator.com/item?id=18707771 <= get ready for some harsh adjectives). I wonder if you're already doing so or planned ? Maybe it's just a backend parameter (llvm-to-gpu-cluster..)
Yes, there is never enough time to cover all that I want. There will be more talks in the future and more as Clasp gets more exposure.
Well, I'm super eager to see more clasp/cando videos on frontpages.
Wonderful and inspiring work! I wonder if it would be possible to create a similar interop story with the Rust programming language, which also targets LLVM -- ideally without having to redo all the hard work that went into Clasp. I have a Prolog interpreter/compiler I'm developing in Rust right now. One of my aims (eventually) is to compile Prolog to fast native code with it. It would be nice to have convenient access to it from a Common Lisp environment.
Rust can already talk C ABI so I think the dream is already reality.
Rust does not have native C++ interop.
C ABI is different to what clasp has done (native C++ interop). All widely used Common Lisp implementations have seamless C ABI interop through their FFIs / CFFI [1].
Clasp goes above and beyond that and natively interfaces with C++. LTO works with mixed Common Lisp and C++ code. Exception handling works as expected. You can mix Common Lisp and C++ stack frames. Correct me if I'm wrong but Rust does not have anything equivalent. The language that comes closest is D I think [2].
Rust can inline cpp code via rust-cpp crate. I've been playing with it several months ago and successfully used dear-imgui and other stuff in some of my rust tech.
See https://news.ycombinator.com/item?id=1998321
"I even went so far as to write an entire compiler and environment (a several man-year endeavor) to design two different runtime oriented dialects called GOOL and GOAL used in 8 Naughty Dog Crash Bandicoot and Jak & Daxter games. All 30-60 FPS action games -- 40 million copies sold. All written in Lisp! (more or less)"
That's pretty interesting too! Although I think the advantage of the game here is that it's not only written in Lisp (or CL as the case may be) but the client is actually going to be open source, and it sounds like they're incorporating a (subset?) of the Lisp language as a component of playing the game.
"Abuse" by Crack.com way back in 1995 was powered by Lisp, as I recall (and also moddable via it):
The source, maintained by Sam Hocevar: http://abuse.zoy.org/browser/abuse/trunk
Of all the code that might have to be written for a non trivial game, a skeletal animation system doesn't strike me as particularly difficult. Curious why that was called out specifically as something not to try and implement yourself?
I agree that skeleton system is not very hard to implement. Anecdotally, one of the first CL based apps I wrote were to implement Quake MD5 (.md5mesh and .md5anim) parsers to do skeletal animation.
I think he might be categorising it as hard for the following: - Defining a simple, easy to navigate data structure with the bone and weight mappings across vertices, to do the math for pose calculation - Having a highly compressed binary format for animation data, which is definitely hard but required for good FPS - Doing it on CPU or GPU
The article talks about one of the primary motivations for using Lisp being that it's 'fun,' which is a comment I've heard about Lisp elsewhere, and which makes sense to me since I've noted a difference in funness between other languages.
So that's pretty appealing to me, but I haven't quite found the right Lisp to use. It seems like Clojure (probably ClojureScript actually) would be the most practical for me to use on some projects—but I'm curious whether others have found Clojure/ClojureScript to have that same element of fun Lisp is sometimes known for.
Clojure is practically tied to the JVM and the Java ecosystem (ClojureScript is different enough to count as a different language so I'll focus on Clojure).
To me that's not fun. It's also limited by design to fit the development model Rich Hickey wanted for his consultancy. It is nowhere near as encompassing or paradigm agnostic as Common Lisp. Moreover, you can take old Lisp code that's 40+ years old and run it today on top of Common Lisp with no/minor changes. Clojure breaks compatibility with Lisp code in many different ways. For me these reasons are enough not to consider Clojure a Lisp.
Still, in cases where one is tied to the Java ecosystem for whatever reason, surely Clojure is better than no-clojure?
I'd still pick something else in that case too since I don't really like the decisions behind Clojure. But hey go for it if it fits the problems that you find yourself solving.
There's always Armed Bear Common Lisp. which runs on the JVM, if you want to stay in it. And you can also try [Linj](https://github.com/xach/linj) which is similar to Common Lisp an transpiles to Java.
Interesting. Can ABCL be used for the same sorts of use cases as Clojure?
if by limited we mean full ecosystem compatibility with jvm and javascript, then yes, Clojure is limited. Which 40 year old CL codebase is it that i am trying to run?
The decisions behind Clojure worked out well for Rich Hickey and his consultancy but they don't fit the programs that I find myself writing all the time (compilers, assemblers, virtual machines, tight control of code generation and memory operations and so on). I can do all of that - and more - in SBCL and it's a pleasure to use as the I don't have to bend myself to work around language shortcomings. Common Lisp gives me a multitude of programming paradigms and an enormous arsenal of approaches to use. When working in exploratory domains, that I do all the time, nothing else comes close. Here is an example of using SBCL for exploratory lowlevel programming by Paul Khuong [1].
I've written performant and lightweight TCP/IP network stacks in Common Lisp. Clojure is just too constrained.
As far as old Lisp code goes, it's everywhere - if you care to look -. The lambda papers, McCarthy's original papers, the CMU CL repository (still of practical use today), vast archives of Lisp code from MIT and so on and so forth. Not to mention useful frameworks like Screamer [2] which is close to 30 years old and Maxima [3] which is even older.
[1] https://www.pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembl...
> if by limited we mean full ecosystem compatibility with jvm and javascript, then yes, Clojure is limited.
I don't think that was he meant. Probably he meant that Clojure took a bunch of things from Lisp, but there is a lot in Lisp, which is not in Clojure - for example because of the limitations of the JVM eco-system (for example creating an executable image in SBCL of a running Lisp is an easy task, whereas in Clojure it involves a complex & slow machinery like GraalVM) or design limitations imposed by its developers (like a compiler written in Java and no interpreter) - plus it seems not to be an open language design - more or less the language is driven by a single company listening to its community and sometimes not.
I find it fun, repl driven development, immutability, functions and maps as first class, practical in many situations
Also Datomic hyperfiddle and specs