BQN's development history

I (Marshall Lochbaum) began designing BQN as a "fixed APL" in collaboration with my colleagues at Dyalog, and decided to take it on as a personal project when I chose to leave the company several months later in early 2020. BQN is influenced by my array language background, previous work in programming design, studies of APL history, and design discussions before and after starting work on the language. I developed most of the novel functionality in BQN, and am at the end of the day the one who writes the spec, but it includes significant contributions from collaborators, most notably dzaima and Adám Brudzewsky.

Background

I learned J as my first computer programming language in 2009, and it's been my language of choice for personal projects from then until I started working with BQN. My first exposure to APL was Dyalog APL, which I began learning gradually after starting work at Dyalog in 2017; while I understand every primitive in detail (I've substantially reimplemented most of them), I've never written even a medium-sized script with it. I studied APL's history and many other APL dialects while helping to create the new APL Wiki in late 2019. In particular, I found A+ to be a very sound design and nominally took it as the starting point for BQN. As a result, BQN draws more from a general concept of APL than any particular dialect.

I have been working on programming language design since 2011 or earlier. The start of my first major language, I, might be placed in 2012 when I published a paper on its approach to mapping, unifying J's trains and function rank. By this time I had worked with several languages including Python and Factor, and implemented little interpreters in Java, Scala, and Haskell. There are many new ideas in I and some of them have made it to BQN, but the language has served mainly as a warning: its pure and simple syntax and approach to array and tacit programming lead to a rigidity that seems to take over from any human designer. And the end result is not usable. So I sought out constructs that would give back some control, like APL's two-layer function/operator syntax and explicit functions (although Dyalog has lexical scoping, it is crippled by the lack of closures and I ended up learning proper use of lexical scoping from Javascript—I've never done real work with any actual Lisp). Another language that I envisioned before BQN, called Iridescence, goes further in this direction, with Python-like syntax that is "noisy" relative to APL. It remains purely theoretical (I'll implement it some day) but has already had an influence on some BQN primitives.

The idea of a "fixed APL" is always percolating in the APL community, because of its long history and strong backwards compatibility requirements. BQN arose out of sessions by the Young APLers Group (YAG, unfortunately) inside Dyalog after I suggested that wild ideas for the future of APL might be a good topic for meetings (the previous order of business, right at YAG's formation, was creating the APL Wiki). At these meetings Adám, Richard Park, Nathan Rogers, and sometimes others discussed and proposed many ideas including the sketch I created that ended up as the blueprint of BQN. When I left Dyalog (putting an end to any notions of using those ideas at the company), I joined The APL Orchard forum, which most YAG members already used, to post about BQN. dzaima quickly began building his own BQN implementation using the existing dzaima/APL, and has discussed and contributed to most BQN design decisions since.

Shapecatcher is an essential resource for finding appropriate unicode characters. I've been using it heavily, and so has everyone else interested in glyph choices.

Timeline

The table documents when I encountered features or interesting decisions in BQN. The "source" field indicates how I became aware of the idea while the "person" field gives, to the best of my knowledge, the original inventor (in blank entries and those that start with "w/", I am an inventor).

Features

Discussion of specific features from the timeline above, with more detail.

Structural Under

I developed structural Under in 2017 and 2018. By Dyalog '17 I had figured out the basic concept, and sometime later I discovered it could be unified with J's Under operator, which was being considered for inclusion in Dyalog.

Prefix, Suffix, and Windows

I discovered Prefix, Suffix, and Windows while thinking about Iridescence, probably in 2019. They are influenced by J's Prefix, Suffix, and Infix operators, but in Iridescence, with no distinction between functions and arrays, Prefix is just the Take function, and Suffix is Drop!

Array notation

APL array notation has been developed mainly by Phil Last and later Adám Brudzewsky. The big difference from array literals in other languages is the idea that newline should be a separator equivalent to ⋄, as it is in ordinary APL execution including dfns. The changes I made for BQN, other than the ligature ‿ discussed below, were to use dedicated bracket pairs ⟨⟩ and [], and to allow , as a separator.

I picked out the ligature character ‿ between YAG meetings, but I think Richard Park was most responsible for the idea of a "shortcut" list notation.

Double-struck special names

There was a lot of discussion about names for arguments at YAG (no one liked alpha and omega); I think Nathan Rogers suggested using Unicode's mathematical variants of latin letters and I picked out the double-struck ones. My impression is that we were approaching a general consensus that "w" and "x" were the best of several bad choices of argument letters, but that I was the first to commit to them.

Assert primitive

Nathan Rogers suggested that assertion should be made a primitive to elevate it to a basic part of the language. I used J's assert often enough for this idea to make sense immediately, but I think it was new to me. He suggested the dagger character; I changed this to the somewhat similar-looking !. The error-trapping modifier ⎊ is identical to J's ::, but J only has the function [: to unconditionally throw an error, with no way to set a message.

Context-free grammar

In YAG meetings, I suggested adopting APL\iv's convention that variable case must match variable type in order to achieve a context-free grammar. Adám, a proponent of case-insensitive names, pointed out that the case might indicate the type the programmer wanted to use instead of the value's type, creating cross roles. Although I considered swapping subjects and functions, I ended up using exactly the conventions of his APL style guide.

Headers

The idea of dfn headers is very common in the APL community, to the extent that it's hard to say which proposals lead to the form now used in BQN. A+ has headers which are similar but go outside the braces, and BQN headers aren't all that different from tradfn headers either. I found when creating BQN2NGN that ngn/apl allows dfns to include a monadic and dyadic case, separated by a semicolon. Some time later I realized that the ability to include multiple bodies is very powerful with headers because it enables a primitive sort of pattern matching, something I already wanted to squeeze into the language. I discussed this with dzaima, who added header support to dzaima/BQN almost immediately and was thus able to investigate the details of the format.

A "return from block" syntax Name→, creating a monadic function that cuts execution short and returns its argument from the block with the given name, was specified to go along with block headers, and is the main reason name-only headers (labels) exist. It was never implemented anywhere, and eventually I decided it was probably not worth the complication and removed it from the spec.

Group

I've been fiddling with the idea of function or map inversion (preimage creation, really) for several years, and in fact independently discovered something very similar to K's Group function =, which is an excellent tool for languages that have dictionaries. I liked this approach as it didn't have all the ordering issues that J's Key has. However, I also didn't really want to introduce dictionaries to BQN, as they have a very strange relation to multidimensional arrays—are arrays like dictionaries with multiple keys, or dictionaries with a single vector key? I've been a proponent of /⁼ as a programming tool for much longer. I'd also developed a sophisticated view of partition representations while studying an extension to Dyalog's Partitioned Enclose proposed by Adám and included in Dyalog 18.0. I finally put all this together while fighting with Key to develop BQN's compiler: I realized that if the "key" argument was restricted to array indices, then it would make sense for the result to be an array, and that this was simply the "target indices" partition representation minus the requirement that those indices be nondecreasing.

Before ⊸ and After ⟜

It happens that BQN's Before (⊸) and After (⟜) modifiers are identical to I's hook (h) and backhook (H), but it took some time to arrive at this point. The hook function in I comes from J's 2-train, also called hook (I had probably seen Roger Hui's remarks that he would prefer hook to be a conjunction, with 2-trains indicating composition instead, but I don't think Roger has proposed a reverse hook). But the model for Before and After was initially APL's Compose (∘) and the complement Reverse Compose that Adám created for Extended Dyalog APL. I noticed the similarity to Bind and decided to unify Binds and Composes at around the same time that I picked the symbols ⊸⟜. However, I kept the idea that the one-argument case should be simple composition unless the bound operand had a subject role. Eventually I decided the violation of syntactic type erasure was too inconsistent and settled on the current definition. Now I think these forms are better even ignoring constant functions, although I do occasionally run into cases where I'd like to use APL's Compose.

Constant modifier

The idea of a constant function is nothing new; I named it k in I, taking influence from the SKI calculus. It was actually Adám who suggested adding it to Dyalog with the glyph ⍨, although I was the one who campaigned for it and introduced it to the public in version 18.0. It wasn't initially clear that a dedicated modifier was needed in BQN because the treatment of data types as constant functions seems to fill this role, but I eventually found that I needed a constant function returning a function too often to leave it out.

Pair

Enlist/Pair (⋈) is an obvious enough primitive, and was in Adám's Extended Dyalog APL as ⍮ in 2018, and in I as ; in 2012—but because I uses nested lists, its version can also be interpreted as ≍. Initially I wanted to see if just ≍ would work out, although now I think only ⋈ would be better than only ≍. When dzaima pointed out its absence just hours after learning about BQN, my reply also stated I had no character for it. I don't know when exactly I hit on ⋈, but I first posted about the character in April 2021 (a feature I liked about it is that it also suggests ⋉ and ⋊ for prepending and appending an element to a list), and added it to the language in October.

Namespaces

I designed the namespace system as a way to pack code into modules, in particular to allow a file to define multiple exported values while having its own privately accessible scope. There was some initial complication to avoid immediately exposing namespaces as first-class values, requiring each namespace to be unpacked where it was created, but this restriction was lifted before long. I consulted with dzaima on various details of the destructuring syntax, particularly aliasing.

High-rank array notation

While array notation with ⟨⟩ was initially supported in the language, the flat rank-increasing version [] was reserved but not implemented until June 2022. Most of the delay was just laziness in specifying and implementing a minor feature, but I did question whether special syntax just for >⟨⟩ was worth it (I'm still not sure, really). The main value is that it can be used as an assignment target like {𝕊[a,b]:…}, a feature that also makes the design slightly less trivial.