"How is Lisp useful?" -- a blog post in which I explain how I use Lisp

Note: this was originally a comment I wrote on Lemmy in answer to the question “what type of problems do you solve using Lisp?”. The post got to be a bit too long, and I am re-publishing it here as a proper blog post. I am also including some of a post I wrote on Mastodon which touched on some of these same issues.

So to answer the question: I have known about Common Lisp and Scheme for years, but only recently started using them. This is the story of the 3 Lisp dialects that I use.

Emacs Lisp

I use Emacs and Emacs Lisp to manage my tens of thousands of text files, I write Emacs Lisp scripts to automate simple tasks like searching for pieces of information, formatting it, and outputting it to a report that I might publish on my blog or send in an e-mail. I also use Emacs to help with data cleaning before running machine learning processes. Emacs helps with navigating CSV and JSON files, it also is a really good batch file renamer.

Scheme

I have recently started using Guile Scheme to do some personal projects. I went with Guile over the myriad other Scheme dialects because it is the implementation used for the Guix package manager and operating system.

• Also, there the Goblins, which is a distributed object-capability programming system is officially supported on the Guile platform, and I have been really wanting to write applications using this programming style ever since I first learned about it.

• Also, there is the G-Golf foreign interface layer allows Guile to automatically use an C library that implements the GObject Introspection interface. So through Guile, like with Python, you can use any C code library used to create of all native apps in the Gnome, MATE, Cinnamon, or (my personal favorite) the Xfce desktop environments. This potentially makes Guile a viable alternative to Python scripting across all of those Linux desktop environments.

Of all the Lisp dialects, Scheme is my favorite, for a few reasons:

• It is absolutely tiny. Guile is relatively large (not as big as Common Lisp), but other implementations are unbelievably small. for example the Chez Scheme “petite” interpreter is fully compliant with the R5RS standard, and the executable is like 308 kilobytes on a 64-bit Linux computer system.

• Hygienic macros with syntax-case

• Recursive functions over using the loop macro of Common Lisp. When writing algorithms, I personally find it easier to reason about recursive functions than loops. Scheme also provides me the ease-of-mind that comes with knowing the optimizing Scheme compiler will ensure recursive loops will never overflow the stack.

• Pattern matching is well supported by most Scheme implementation.

• It is a "Lisp-1" system, meaning there is only one namespaces for variables and functions, as opposed to Common Lisp (a "Lisp-2 system") which allows a name to be either a variable, a function, or both. I personally find it easier to reason about higher-order functions in Lisp-1 systems.

• Support for Delimited Continuations, which is a fairly recent discovery of computer language theory (first being discussed back in the 1990s), but is available across a few Scheme implementations.

Common Lisp

That said, I am also starting experimenting with Embedded Common Lisp (ECL) because it is a lightweight standards compliant Common Lisp implementation that compile your program into C++ code, and this is useful to my professional work.

The modern software industry, especially in the realm of big data and machine learning, has mostly settled on a pattern of using C++ for creating performance critical libraries, and creating Python binding to the C++ libraries for scripting. I was hoping languages like Haskell and/or Rust might come along and change all this, but it will take decades (if ever) for the software industry to turn in that direction.

The problem with Python, in my experience (and I believe many other software engineers would agree) is that it does not scale well to larger applications at all, whereas Common Lisp does. This is for various reasons, but mostly due to how Lisp does strong dynamic typing, and also the CLOS implementation of the meta-object protocol. Yet too many companies waste time writing large applications in Python — applications that are much larger than the scripting use cases that Python was originally intended to be used. I believe this is time and money better spent on other things.

So I see Common Lisp, and the ECL compiler, as a potentially viable alternative to the sub-optimal status quo of Python as a scripting layer around C++ code libraries, at least perhaps for my day job, if not being more generally true industry-wide. Mostly, ECL would allow me to write a program in Common Lisp instead of Python, but deliver to my clients the C++ code that ECL generates to be used in their machine learning projects. (I have not actually done this yet, I am still investigating whether this would be a viable solution to any of my projects).

ECL makes it easy to use C++ libraries through Lisp instead of Python. And there are so many good C++ libraries out there: Qt, OpenCV, Tensorflow, PyTorch, OpenSceneGraph, FreeCAD, Godot game engine, Blender. And it compiles easily on Linux/Unix (GCC), Windows (MSVC), and MacOS (via Clang++), so good for cross-platform development.

Conclusions

So in spite of Lisp being such an old family of languages (its earliest incarnations dating all the way back to 1958), and being superseded in popularity and widespread use by languages like Python and JavaScript across the software industry, Lisp is still a modern, relevant, evolving, and very useful family of programming languages. At the same time, a Lisp such as Scheme or Common Lisp would even be a better choice of programming language in many applications where Python is currently used.

I just hope I eventually find the time to try out all of these Common Lisp and Scheme related ideas I have. I especially hope ECL turns out to be a profitable technological choice for the professional work that I do. But only time will tell.

Please feel free to comment here, or on Mastodon