My simplicity toolkit: Programming (part 2)

Info

This is the second part of two talking about my simplicity toolkit when programming. If you have not read the previous post I highly encourage you to do so first.

This post is part of my blog series about My Simplicity Toolkit.

At the beginning of my previous post about simplicity within programming, I talked about the numerous benefits that bringing simplicity to programming can be: Easy to understand, use, debug, etc.

Choosing simplicity can be radical, and choosing it means taking a stance against something. I have therefore decided to formulate all my simplicity tools as “X over Y”. The following preferences are a continuation of the previous post:

SQL over ORM Link to heading

SQL is the lingua franca of working with data stored in databases. SQL is interactive, which makes it easy to manually build exactly the queries we want. SQL is also powerful & flexible, allowing us to transform, filter, sort, group, pivot our data, and more.

The most basic idea with ORMs (Object Relational Mappers) is to map SQL query results to objects. While that can be a bit of manual work, let’s be honest, manual mapping is not a lot of work and is usually done once. And assuming you have automated tests of your persistence layer (right??), typos should be caught before being shipped.

The problem with ORMs is that they have vastly expanded beyond the mapping functionality. They support lazy loading, proxy objects, eager loading, cascading deletes, have giant interfaces, and more. Most ORMs are not simple anymore and I keep ending up googling how to do various tasks in them.

Oh, a note on SQL injections: There are DSL query builders that protect you from that. And linters. You don’t need a big ORM to avoid SQL injections.

In short, I believe there is generally so much complexity in most ORMs that we are better off not using them.

Finally, SQL is timeless and ORMs are not. SQL has stood the test of time. ORMs are 3rd party libraries that come and go or get modified over time. I prefer to invest my knowledge in timeless solutions first.

High-level code up the callstack over low-level Link to heading

Here is a small Python program:

from functools import reduce

def calculate_sum_of_squared():
    numbers = []
    
    while True:
        user_input = input("Enter a number (enter '0' to finish): ")
        num = float(user_input)

        if num == 0:
            break

        numbers.append(num)
    
    result = reduce(lambda acc, value: acc + (value ** 2), numbers, 0)
    
    print("Sum of numbers:", result)

if __name__ == "__main__":
    calculate_sum_of_squared()

It does everything in one function, calculate_sum_of_squared(). Compare that to:

from functools import reduce

def get_numbers_from_terminal():
    numbers = []
    
    while True:
        user_input = input("Enter a number (enter '0' to finish): ")
        num = float(user_input)
        
        if num == 0:
            break
        
        numbers.append(num)
    
    return numbers

def calculate_sum_of_squared(numbers):
    return reduce(lambda acc, value: acc + (value ** 2), numbers, 0)

if __name__ == "__main__":
    numbers = get_numbers_from_terminal()
    result = calculate_sum_of_squared(numbers)
    
    print("Sum of numbers:", result)

The latter example has high-level code at the top of the callstack in if __name__ == "__main__". You can easily see that the program is split into two stages, first asking the user for numbers, followed by making a calculation.

Another thing that is different is that the most low-level, the reduce(...) function call has been moved into a smaller function as deep into the callstack as possible. By doing so, we can put a name on what it does. Parsing what reduce(lambda acc, value: acc + (value ** 2), numbers, 0) means is not simple.

Layered software architecture over mixing transport, business logic, or persistence Link to heading

Most applications I have worked with have three layers:

The external API layer. This is the layer that handles the input and output of your application. It works with data-transfer objects (DTOs) and business models and converts between them. Data-transfer objects are usually used for JSON marshaling/unmarshalling. Business models are the most natural representation to implement your business logic.
The business layer where all my business logic is implemented. It works with business models only.
The persistence layer which deals with storing and reading up things from a database. It takes in and returns business models. It can optionally use persistence objects if an ORM is used. The persistence layer is where a Repository implementation resides.

Over and over again I have seen classes conflate (or shall I say…complect?) DTOs, business models, and persistence entities, and being used in multiple of these layers. An example:

import com.fasterxml.jackson.annotation.JsonProperty;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import java.io.Serializable;

@Entity
public class ConflatedClass implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private long databaseId;

    @JsonProperty("fullName")
    private String name;
    private int age;

    public ConflatedClass() {
        // Default constructor required by Hibernate
    }

    public ConflatedClass(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public long getDatabaseId() {
        return databaseId;
    }

    @JsonProperty("fullName")
    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setAge(int age) {
        this.age = age;
    }
}

By mixing JSON, and persistence and using this object for business logic, most of our commits must work across all the layers of your application from the REST API down to the persistence layer. This leads to larger commits and more bugs. It also makes it hard to migrate to a different type of database or switch JSON library. It also usually creates a strong coupling between which database is used.

The alternative would be to have three classes:

import com.fasterxml.jackson.annotation.JsonProperty;

public class PersonDTO {
    @JsonProperty("fullName")
    private String name;
    private int age;

    public PersonDTO(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }
}

public class Person {
    private String name;
    private int age;
    private String occupation;  // Additional business model fields if needed

    public Person(String name, int age, String occupation) {
        this.name = name;
        this.age = age;
        this.occupation = occupation;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public String getOccupation() {
        return occupation;
    }

    public void setOccupation(String occupation) {
        this.occupation = occupation;
    }
}

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Entity
public class PersistedPerson {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private long databaseId;

    private String name;
    private int age;

    public PersistedPerson() {
        // Default constructor required by Hibernate
    }

    public PersistenceModel(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public long getDatabaseId() {
        return databaseId;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setAge(int age) {
        this.age = age;
    }
}

A common complaint about having three different classes is that mapping between these classes is cumbersome. That’s a feature, not a bug! By explicitly mapping, you understand what is happening, and it gives you more flexibility. Besides mapping is simple and quick - and there are libraries for it if you do need it.

The Internet is full of articles about layered architecture, but if you would like to get started I highly recommend reading about The Clean Architecture (a superset of all variants of layered architectures), and possibly the Domain-Driven Design book.

CI over startup Link to heading

I prefer to do things at CI than when my application starts if I can. Examples:

Package static resources.
Compress resources if I can.

By doing this, I catch issues earlier at CI than after deployment. This also has the added benefit that my application tends to start up faster.

This blog is actually a great example of this. It is a set of static HTML files that get generated by the CI. This means I don’t need any runtime to serve it and can publish it using Github Pages for free. ✨

Boot phase over request phase Link to heading

I tend to think of REST services as generally having three stages:

Boot. When the application starts.
Serve. When the application is serving requests.
Shutdown. When the application, refuses new requests, drains ongoing requests and shuts down.

I try to do as much as possible during the boot phase instead of when serving. For example, if I can read up some small data from the database and store it as an in-memory immutable object for the rest of the application, I prefer that over having to read from the database on every API call.

This has the following benefits:

Fewer incidents. If anything goes wrong during boot, the application shuts down, health checks don’t pass, and no users are impacted.
Easier to test. Testing code run when booting doesn’t require setting up an HTTP server.
More performant. By moving work to boot, each request needs to do less work. By definition, the API becomes more performant.
Thread-safety. I tend to store the things done at boot as immutable datastructures during the serving phase. By doing so, I know they are thread-safe and can be read by multiple concurrent threads.

You can read more about this in I’m a State Engineer. Are you, too?.

Thread-unsafe over thread-safe code Link to heading

Writing thread-safe code is hard and error-prone. On top of it, it usually comes with parameters that must be tweaked in terms of queue limits, ordering promises, and concurrency limits. Thread-safe code also tends to be less performant; locks need to be taken and synchronization needs to happen.

This is why I try to push thread safety up the stack. Unless it’s a concurrency-related library, I expect the user of my library to deal with concurrency, not me.

Synchronous over asynchronous Link to heading

Our REST API is slow! We need to make it asynchronous!

Congratulations! You now have many new problems to solve:

How should the user know that the API request is finished? Webhooks? Long-polling? An e-mail? Polling?
- If a user submitted 100k async requests, will you be able to handle the pressure of all the poll calls?
If the user makes two requests to modify the same resource, which request wins?
How will you handle if your async process always crashes, or a new deployment happens while it’s running?
How will you handle if the user submits 100k async requests? Will Redis be able to store all the tasks?
How will you make sure that one user doesn’t saturate the queue for 30 minutes impacting all other users?
You will eventually need to start maintaining priorities between different types of tasks.
You will now need to maintain concurrency-related limits such as queue depths and concurrency limits.

On top of the above, asynchronicity spreads like wildfire to too many other systems depending on your newly async API. This means that they, too, must handle many of the problems above.

Running my code is slow! I need to parallelize it!

Congratulations! Similarly to the above, you now forever need to maintain concurrency-related limits such as concurrency, and queue depths.

The alternatives to the above solutions are:

Reconsider your data model. You likely got it wrong.
Auto-scaling (but it’s not a panacea).
Load shedding.

Working code over pretty code Link to heading

Code that does not work has no value to your business or your customers (however, the process of creating a spike that doesn’t work might have!).

Pretty code over performant code Link to heading

Premature optimization is the root of all evil.
Sir Tony Hoare

Code is read much more often than it is written. Pretty code is readable. It is also implicitly easy to modify since it’s easy to understand.

When it comes to performance, 99% of the time a program is spent in 1% of its source code. This means that performance usually is not of concern. Instead, figure out where your program spends time and optimize that only.

Standard library over 3rd party library Link to heading

Reusing code is not an end goal and will not make your code more maintainable per se. Reuse complicated code but be aware that reusing code between two different domains might make them depend on each other more than necessary. Also remember that the more you reuse code, the more it needs to be well-tested, otherwise you risk breaking lots of things with a tiny change.

Every third-party library is also a liability from a security perspective. Hello TypeScript and Leftpad! 😄

Finally, if you want to invest your knowledge in something, invest in something more timeless. The standard library will prevail but 3rd libraries will not. They all eventually get replaced or changed.

My own interfaces over 3rd party interfaces Link to heading

In a previous post, I wrote about the repository pattern. It talked about how the repository pattern allows me to replace the implementation of a repository. For example, at a previous employer, I migrated data from MySQL to Cassandra without having to rewrite any business logic. This was possible to do since I owned the repository interface myself.

However, there are libraries that ship with interfaces. The problem with such an interface is three-fold:

Firstly, the external interface is usually much larger than what you actually need. You break the Interface Segregation Principle. For an ORM, maybe you are fine with a simple key/value interface, but it gives you everything from support for transactions to iterating over all the rows.

Secondly, when upgrading the library, you risk breaking a lot of code if the upstream interface has changed. If you own the interface yourself, you will only need to update the [bridge][bridge-patterm] between your interface and the third-party library.

Third, a library interface also usually comes with implementations of its interfaces. This means that the interface usually is highly coupled for the union of all these implementations. For example, there is an ORM called TypeORM that ships with support for a fixed list of databases. However, that list does for example include Apache Cassandra. The interface for their “repository” might therefore not fit well with Cassandra. Had you owned the interface yourself, you could cater it for the database implementation you would need.

Simple code over annotations Link to heading

A common thing many frameworks and libraries do is to use @Annotations to control validation, HTTP routing, SQL generation, OpenAPI documentation, and more. I think annotations are an excellent example of where our industry is choosing easy over simple. At a first glance, an annotation looks simple, right?!

The thing is, it’s usually very hard to understand the implication of an annotation. The actual code handling the annotation is hidden inside a foreign framework or library, leading to Action at a Distance. Ever wondered in what order the annotations are handled in? Or the implications of adding an ORM annotation to your SQL? Good luck.

The alternative is boring code, usually in a function. Want to validate your DTO? How about implementing validateMyDTO(): []error and calling it when you want to validate it? By doing this, you can debug your validation, write (simple) tests for your validation, understand exactly where you validation is being called in your IDE, and more. Less magic, more simplicity.

Conclusion Link to heading

This post concludes summarising my simplicity toolkit when programming. Next up is I will write about testing from the perspective of simplicity.