Morloc Technical Manual

morloc allows functions from polyglot libraries to be composed in a simple functional language. The focus isn’t on classic interoperability (like calling C from Python), or serialization (like sending JSON between programs) — though morloc implementations may use these under the hood. Instead, you define types, import implementations, and compose everything together to build complex programs. The compiler invisibly generates any required interop code.

Do you want to write in language X but have to write in language Y because everyone in your team does or because your expected users do? Love C for algorithms, R for statistics, but don’t want to write full apps in either? morloc lets you mix and match, so you can use each language where it shines, with no bindings or boilerplate.

Tired of learning new benchmark and testing suites across all your languages? Is it hard to benchmark similar tools wrapped in applications with varying input formats, input validation costs, or startup overhead? In morloc, functions with the same general type signature can be swapped in and out for benchmarking and testing. The same test suites and test cases will work across all supported languages because inputs/output of all functions of the same type share equivalent morloc binary forms, making validation and comparison easy.

Tired of writing wrappers, CLIs, APIs, and bindings for every tool? With morloc, just write clean functions and compositions — the compiler can generate the rest. Rather than porting and maintaining a complex application, with all its fragile interfaces and idiosyncracies, you can focus on the algorithms.

Are you accustomed to chaining tools with text files (think bioinformatics pipelines)? Maybe you even thought this was "the UNIX way" and therefore a "good thing". But after writing your hundreth GFF parser that builds gene models from inconsistently formatted attribute fields, maybe you’ve started to have doubts. morloc lets you drop the fragile formats. Instead, compose functions that share clear, unambiguous data structures that can be serialized unambiguously to JSON, MessagePack or morloc binary format.

Tired of reimplementing everything in every language? In morloc, we can define functions, typeclasses, and hierarchies of types. Then build universal, (optionally) polyglot libraries that compose across language boundaries. These libraries may be searched by type as well as metadata (ranging from mundane licensing info to exotic empirical models of performance and certificates of correctness). This makes it possible to build safe, verifiable, and even AI-assisted compositions.

The easiest way to start using morloc is through containers. I recommend using podman, since it doesn’t require a daemon or sudo access. But Docker, Singularity, and other container engines are fine as well.

An image with the morloc executable and batteries included can be retrieved from the GitHub container registry as follows:

The v0.53.7 may be replaced with the desired morloc version.

Now you can enter a shell with a full working installation of morloc:

The --shm-size=4g option sets the shared memory space to 4GB. morloc uses shared memory for communication between languages, but containers often limit the shared memory space to 64MB by default.

Alternatively, you can set up a script to run commands in a morloc environment:

Name this script menv, for "morloc environment", make it executable, and place it in your PATH. The script will mount your current working directory and your morloc home directory, allowing you to run commands in a morloc-compatible environment.

You can can run commands like so:

The generated executables may not work on your system since they were compiled inside the container, but you can run them in the container environemtn as well:

More advanced solutions with richer dependency handling will be introduced in the future, but for now this allows easy experimentation with the language in a safe(ish) sandbox.

The menv morloc or menv ./nexus syntax is a bit verbose, but I’ll let you play with alternative aliases. The conventions here are still fluid. Let me know if you find something better and or if you find bugs in this approach.

The inevitable "Hello World" case is implemented in morloc like so:

The module named main exports the term hello which is assigned to a literal string value.

Paste code this into a file (e.g. "hello.loc") and then it can be imported by other morloc modules or directly compiled into a program where every exported term is a subcommand.

This command will produce two files: a C program, nexus.c, and its compiled binary, nexus. The nexus is the command line user interface to the commands exported from the module.

Calling nexus with no arguments or with the -h flag, will print a help message:

The command is called as so:

In morloc, you can import functions from many languages and compose them under a common type system. The syntax for importing functions from source files is as follows:

This brings the functions map, sum, and snd into scope in the morloc script. Each of these functions must be defined in the C++ and Python scripts. For Python, since map and sum are builtins, only snd needs to be defined. So the foo.py function only requires the following two lines:

The C++ file, foo.hpp, may be implemented as a simple header file with generic implementations of the three required functions.

Note that these implementations are completely independent of morloc — they have no special constraints, they operate on perfectly normal native data structures, and their usage is not limited to the morloc ecosystem. The morloc compiler is responsible for mapping data between the languages. But to do this, morloc needs a little information about the function types. This is provided by the general type signatures, like so:

The syntax for these type signatures is inspired by Haskell, with the exception that generic terms (a and b here) must be declared on the left. Square brackets represent homogenous lists and parenthesized, comma-separated values represent tuples, and arrows represent functions. In the map type, (a → b) is a function from generic value a to generic value b; [a] is the input list of initial values; [b] is the output list of transformed values.

Removing the syntactic sugar for lists and tuples, the signatures may be written as:

These signatures provide the general types of the functions. But one general type may map to multiple native, language-specific types. So we need to provide an explicit mapping from general to native types.

These type functions guide the synthesis of native types from general types. Take the C mapping for `List a` as an example. The basic C list type is vector from the standard template library. After the morloc typechecker has solved for the type of the generic parameter a, and recursively converted it to C`, its type will be substituted for `$1`. So if `a` is inferred to be a `Real`, it will map to the C `double, and then be substituted into the list type yielding std::vector<double>. This type will be used in the generated C++ code.

Functions can be composed:

These morloc compositions will be internally rewritten in terms of the native imported functions, for example:

So in the final form, all functions in morloc are imported from foreign languages.

morloc also supports partial application, eta reduction, and the dot-operator for composition. So sumSnd can be simplified to:

But what code is generated from this? Remember, we imported functions in Pythong and C++ for each of the three native functions above. This problem is addressed in the next section.

morloc supports a kind of language or implementation polymorphism. Each term may have many definitions. For example, the function mean has three definitions below:

mean is sourced directly from C++, it is defined in terms of the sum function, and it is defined more generally with sum written as a fold operation. The morloc compiler is responsible for deciding which implementation to use.

The equals operator in morloc indicates functional substitutability. When you say a term is "equal" to something, you are giving the compiler an option for what may be substituted for the term. The function mean, for example, has many functionally equivalent definitions. They may be in different languages, or they may be more optimal in different situations.

Now this ability to simply state that two things are the same can be abused. The following statement is syntactically allowed in morloc:

What is x after this code is run? It is 1 or 2. The latter definition does not mask the former, it appends the former. Now in this case, the two values are certainly not substitutable. morloc has a simple value checker that will catch this type of primitive contradition. However, the value checker cannot yet catch more nuanced errors, such as:

In this case, the type checker cannot check whithin the implementation of add, so it cannot know that there is a contradiction. For this reason, some care is needed in making these definitions.

In addition to language polymorphism, morloc offers more traditional ad hoc polymorphism over types. Here typeclasses may be defined and type-specific instances may be given. This idea is similar to typeclasses in Haskell, traits in Rust, interfaces in Java, and concepts in C++.

In the example below, Addable and Foldable classes are defined and used to create a polymorphic sum function.

The instances may import implementations for many languages.

The native functions may themselves be polymorphic, so the imported implementations may be repeated across many instances. For example, the Python add may be written as:

And the C++ add as:

Up to now we have ignored the method morloc uses to allow communication between languages. We’ve simply asserted that there was a "common type system". We’ll now give a quick peak into how all this works (finer details will be reserved for technical sections later on).

Every morloc general type maps unambiguously to a binary form that consists of several fixed-width literal types, a list container, and a tuple container. The literal types include a unit type, a boolean, signed integers (8, 16, 32, and 64 bit), unsigned integers (8, 16, 32, and 64 bit), and IEEE floats (32 and 64 bit). The list container is represented by a 64-bit size integer and a pointer to an unboxed vector. The tuple is represented as a set of values in contiguous memory.

Here is an example of how the type ([UInt8], Bool), with the value ([3,4,5],True), might be laid out in memory:

Records are represented as tuples. The names for each field are stored only in the type schemas. morloc also supports tables, which are just records where the field types correspond to the column types and where fields are all equal-length lists. Records and tables may be defined as shown below:

The morloc type signatures can be translated to schema strings that may be parsed by a foundational morloc C library into a type structure. Every supported language in the morloc ecosystem must provide a library that wraps this morloc C library and translates to/from morloc binary given the morloc type schema.

By itself, this system allows any type that is comprised entirely of literals, lists, and tuples to be translated between languages. But what about types that do not break down cleanly into these forms? For example, consider the parameterized Map k v type that represents a collection with keys of generic type k and values of generic type v. This type may have many representations, including a list of pairs, a pair of columns, a binary tree, and a hashmap. In order for morloc to know how to convert all Map types in all languages to one form, it must know how to express Map type in terms of more primitive types. The user can provide this information by defining instances of the Packable typeclass for Map. This typeclass defines two functions, pack and unpack, that construct and deconstruct a complex type.

The Map type for Python and C++ may be defined as follows:

The morloc user never needs to directly apply the pack and unpack functions. Rather, these are used by the compiler within the generated code. The compiler constructs a serialization tree from the general type and from this trees generates the native code needed to (un)pack types recursively until only primitive types remain. These may then be directly translated to morloc binary using the language-specific binding libraries.

In some cases, the native type may not be as generic as the general type. Or you may want to add specialized (un)packers. In such cases, you can define more specialized instances of Packable. For example, if the R Map type is defined as an R list, then keys can only be strings. Any other type should raise an error. So we can write:

Now whenever the key generic type of Map is inferred to be anything other than a string, all R implementations will be pruned.

Here is an example showing a parallel map function written in Python that calls C++ functions.

This morloc script exports a function that sums a list of lists of real numbers. The sum function is implemented in C++:

The parallel pmap function is written in Python:

morloc the inner summation jobs will be run in parallel. The pmap function has the same signature as the non-parallel map function, so can serve as a drop-in replacement.

Here we’ll cover the basic morloc data types that directly map to morloc binary. For these types, there are fairly direct representations in most languages. The basic types are listed below:

All basic types may be written to a schema that is used internally to direct conversions between morloc binary and native basic types. The schema values are shown in the table above. For example, the type [(Bool, [Int8])] would have the schema at2bai1. You will not usually have to worry about these schemas, since they are mostly used internally. They are worth knowing, though, since they appear in low-level tests, generated source code, and binary data packets.

A record is a named, heterogenous list such as a struct in C, a dict in Python, or a list in R. The type of the record exactly describes the data stored in the record (in contrast to parameterized types like [a] or Map a b). The are represented in morloc binary as tuples, the keys are only stored in the schemas.

A table is like a record where field types represent the column types. But table is not just syntactic sugar for a record of lists, the table annotation is passed with the record through the compiler all the way to the translator, where the language-specific serialization functions may have special handling for tables.

Both are defined in similar ways.

Function definition follows Haskell syntax.

morloc supports the . operator for composition, so we can re-write foo as:

morloc supports partial application of arguments.

For example, to multiply every element in a list by 2, we can write:

General type declarations also follow Haskell syntax:

Where a is a generic type variable. morloc supports [a] as sugar for List a.

The general types may be translated to concrete types by fully evaluating them with a set of language-specific type functions. For example:

Language-specific types are always quoted since they may contain syntax that is illegal in the morloc language.

Type functions may also map between general types.

Why do I call them type functions, rather than just aliases? There is a lot more that can be done with these functions that I am just beginning to explore.

Sourcing a function from a foreign language is done as follows:

Here we state that we are importing the function mlc_foo from the C++ source file foo.h and calling it foo. We then give it a general type signature.

Currently morloc treats language-specific functions as black boxes. The compiler does not parse the C++ code to insure the type the programmer wrote is correct. Checking a morloc general type for a function against the source code may often be possible with conventional static analysis. LLMs are also quite effective at both inferring morloc types from source code and checking types against source code.

For statically typed languages like C++, incorrectly typed functions will usually be caught by the foreign language compiler.

A module includes all the code defined under the import <module_name> statement. It can be imported with the import command.

The following module defines the constant x and exports it.

Another module can import Foo:

A term may be imported from multiple modules. For example:

This module imports that C++, Python, and R add functions and exports all of them. Modules that import add will import three different versions of the function. The compiler will choose which to use.

morloc supports ad hoc polymorphism, where instances of a function may be defined for multiple types.

Here is an example of a simple type classe, Sizeable, which represents objects that have be mapped to an integer that conveys the notion of size:

Instances of Sizeable may be defined in this module or in modules that import this module. For example:

Where in C`, the generic function `size` returns length for any `C size with a size method. For Python, the builtin len can be directly used.

morloc also supports multiple parameter typeclasses, such as in the Packable typeclass below:

This specific typeclass is special in the morloc ecosystem since it handles the simplification of complex types before serialization. Instances may overlap and the most specific one will be selected. Packable may have instances such as the following:

Each supported language has a base library that roughly corresponds to the Haskell prelude. They have functions for mapping over lists, working with strings, etc. They also contain standard type aliases for each language. For example, type Cpp ⇒ Int = "std::string".

The root of the current library is the conventions module that defines the core type classes and the type signatures for the core functions. The conventions library does not, however, load any foreign source code, so it is entirely language agnostic.

Next each language has their own base module — such as pybase, rbase, and cppbase — that import conventions and include the implementations for all (or some) of the defined functions and typeclasses.

Finally, a base module imports all of the language-specific bases. Currently, there are only three supported languages, so importing all their base modules is not impractical. In the future, more selective approaches may be used.

TODO

TODO

An "object" is a somewhat loaded term in the programming world. As far as morloc is concerned, an object is a thing that contains data and possibly other unknown stuff, such as hidden fields and methods. All types in morloc have must forms that are transferable between languages. Methods do not easily transfer; at least they cannot be written to morloc binary. However, it is possible to convey class-like APIs through typeclasses. Hidden fields are more challenging since, by design, they are not accessible. So objects cannot generally be directly represented in the morloc ecosystem.

Objects that have a clear "plain old data" representation can be handled by morloc. These objects, and their component types, must have no vital hidden data, no vital state, and no required methods. Examples of these are all the basic Python types (int, float, list, dict, etc) and many C++ types such as the standard vector and tuple types. When these objects are passed between languages, they are reduced to their pure data.