A critical look at the modern programming landscape
This post is inspired by the languages Lisp, Smalltalk, io, Unison. The IT industry changes faster than any other industry, due to the flexibility of how information is exchanged and processed. And so do the languages, with which programmers express their intentions to a computer. But it is my impression, that the development in this space becomes more and more like searching for a local minimum, in a space constrained by previously adopted abstractions, and the limitations that come from them. There is nothing wrong with that, but what interests me is - can we have revolutionary progress, instead of evolutionary? If it is possible, it would take questioning the assumptions made until this point, challenging our very viewpoint. So, let us engage in some critical thinking. State Image source Programmers avoid state at all costs. With stateless computations - you can scale out, you can retry, you can do the obvious thing, and expect it to work. When you try managing state - the ground slips from under your feet, you cannot trust anything, your tools work against you. Evolution of the state is a nightmare of it's own. If you thought about this ahead of time at all, you can choose the least worst option. The language dealing with state is like an intern that was given a task it was unfit for - best it can do is pass it on like a hot potato. To a framework, to a pattern, to a database. Why can't the language itself express state clearly? Have objects, that encapsulate state. Have them expose an API, that performs state changes while maintaining internal consistency. Have them compose with other objects, to provide higher-level APIs. Have the objects persist across program restarts, across version upgrades. Hell, do we even need the concept of "program restart", if our state handling is actually good? Distributed programming Image source Programmers were ambitious in the 90s. They dreamed of objects that could transparently move from server to server. Of function calls that were transparently remote. Of the code itself being remote, with code-on-demand principle in the original REST specification. Now best they can do is spin up VMs or containers behind a load balancer. Oh, but function calls can fail at any moment, due to a network outage! We have error handling for that. But if you try to do it properly - all the edge cases will obscure the actual logic of your code. The fault is with the language, because it fails in decoupling the edge cases from the happy path, which should be straightforward. But function calls can hang indefinitely! Putting timeouts on requests should be straightforward and ergonomic. But for that the language would need to have a concept of a "request". But how do you wait for the result of a remote function call? Do you keep the connection? Do you poll? Do you wait for a reply? How do you do idempotency? True, with modern languages it is all a pain - we should be able to do better. You can forego a lot by sticking to a "local" mentality when programming. But good distributed programs are also good local programs. Local programs are an optimization, a corner-case. If a language helps you make good distributed programs without footguns - it will help you with local ones just as much. Languages should step up and solve the general problem, instead of optimizing the local corner-case. Distributed state Image source A good general solution for distributed state is, at the time of writing, event sourcing, eventual consistency, the saga pattern. But constructing, operating and evolving such a system with currently available tools is a nightmare. Again, state being consistent, and not needing a generalised mechanism for rollbacks is an optimization, not the general problem. And it is a problem worth solving. Concurrency Image source When coroutines are executed, they allocate space on the heap for storing their state. When different coroutines execute at different times, the processor operates on different areas of memory, depending on what the event loop decides to execute next. But if we know some coroutines to depend on each other in series, we could allocate their internal state sequentially within memory, as well as avoid jumping into the event loop altogether, because we know what we want to execute next - we could just jump from one coroutine to the next. Thus we invent the stack, and saving the return address to it. Which demonstrates, that sequential execution is, once again, a particular problem, and concurrency is the general one. I'm a fan of validation-based concurrency control myself - when a distributed transaction modifies state, it places a condition that has to be valid for the duration of the distributed transactions. If another transaction modifies state as well and violates the condition, the initial transaction gets rolled back. For example, a payment in progress places a condition that balance has to be above z
This post is inspired by the languages Lisp, Smalltalk, io, Unison.
The IT industry changes faster than any other industry, due to the flexibility of how information is exchanged and processed. And so do the languages, with which programmers express their intentions to a computer. But it is my impression, that the development in this space becomes more and more like searching for a local minimum, in a space constrained by previously adopted abstractions, and the limitations that come from them. There is nothing wrong with that, but what interests me is - can we have revolutionary progress, instead of evolutionary? If it is possible, it would take questioning the assumptions made until this point, challenging our very viewpoint. So, let us engage in some critical thinking.
State
Programmers avoid state at all costs. With stateless computations - you can scale out, you can retry, you can do the obvious thing, and expect it to work. When you try managing state - the ground slips from under your feet, you cannot trust anything, your tools work against you. Evolution of the state is a nightmare of it's own. If you thought about this ahead of time at all, you can choose the least worst option. The language dealing with state is like an intern that was given a task it was unfit for - best it can do is pass it on like a hot potato. To a framework, to a pattern, to a database. Why can't the language itself express state clearly?
Have objects, that encapsulate state. Have them expose an API, that performs state changes while maintaining internal consistency. Have them compose with other objects, to provide higher-level APIs. Have the objects persist across program restarts, across version upgrades. Hell, do we even need the concept of "program restart", if our state handling is actually good?
Distributed programming
Programmers were ambitious in the 90s. They dreamed of objects that could transparently move from server to server. Of function calls that were transparently remote. Of the code itself being remote, with code-on-demand principle in the original REST specification. Now best they can do is spin up VMs or containers behind a load balancer.
Oh, but function calls can fail at any moment, due to a network outage! We have error handling for that. But if you try to do it properly - all the edge cases will obscure the actual logic of your code. The fault is with the language, because it fails in decoupling the edge cases from the happy path, which should be straightforward.
But function calls can hang indefinitely! Putting timeouts on requests should be straightforward and ergonomic. But for that the language would need to have a concept of a "request".
But how do you wait for the result of a remote function call? Do you keep the connection? Do you poll? Do you wait for a reply? How do you do idempotency? True, with modern languages it is all a pain - we should be able to do better.
You can forego a lot by sticking to a "local" mentality when programming. But good distributed programs are also good local programs. Local programs are an optimization, a corner-case. If a language helps you make good distributed programs without footguns - it will help you with local ones just as much. Languages should step up and solve the general problem, instead of optimizing the local corner-case.
Distributed state
A good general solution for distributed state is, at the time of writing, event sourcing, eventual consistency, the saga pattern. But constructing, operating and evolving such a system with currently available tools is a nightmare. Again, state being consistent, and not needing a generalised mechanism for rollbacks is an optimization, not the general problem. And it is a problem worth solving.
Concurrency
When coroutines are executed, they allocate space on the heap for storing their state. When different coroutines execute at different times, the processor operates on different areas of memory, depending on what the event loop decides to execute next. But if we know some coroutines to depend on each other in series, we could allocate their internal state sequentially within memory, as well as avoid jumping into the event loop altogether, because we know what we want to execute next - we could just jump from one coroutine to the next. Thus we invent the stack, and saving the return address to it. Which demonstrates, that sequential execution is, once again, a particular problem, and concurrency is the general one.
I'm a fan of validation-based concurrency control myself - when a distributed transaction modifies state, it places a condition that has to be valid for the duration of the distributed transactions. If another transaction modifies state as well and violates the condition, the initial transaction gets rolled back. For example, a payment in progress places a condition that balance has to be above zero. Other techniques are also possible. But it needs to be at the language level, it needs to be correct, and ergonomic.
Trees
Scope forms a tree. Function calls form a tree. Object references and their lifetimes form a tree. Modules form a tree. Files and directories form a tree. The program is just one big tree.
In the real world, things relate to each other, forming a free-form graph. Our programs model the real world - why do we stick to a restricted graph, and not reach for the real thing?
And no, files and directories (and symlinks, and named pipes, and unix domain sockets, and character and block devices) are not a good abstraction. A file is everything, and nothing. For managing documents, you need one set of features. For devices - another. For program state (including configuration) - a third. Lumping all of it together into "the filesystem" does not make anyone's job easier. Speaking of which,
Text files
Text files are useful enough, and have a lot of tooling built around them. The filesystem itself, git, emails, parsers, formatters - work well enough. But we can do more, much more.
A program goes from text, to binary image, to a running process - which is what we need, in the end. Why not skip the extra steps? Like Lisp and Smalltalk - have just objects, with them being the program, and the thing that is running. Diffing, merging, sharing, perserving history - all of that falls in the realm of state management. And if the state management is good - it should not be a problem. In fact, because we operate on the "real thing", instead of text - we could do much more, because it's the right abstraction. For example, diffs that are meaningful for the language, and not just lines of text.
A lot of the problems which are relevant to the current way of programming - dependency management, cyclic module imports, factories - all have to do with bringing a system to life "from nothing". But if we don't construct and tear down a system each time a program is run, if we evolve it incrementally - those problems become much, much simpler.
A deep integration with the IDE
But navigating a graph of objects, which aren't just a template, but are "alive" - takes deep integration with the IDE. But in a way, the line between the IDE and other programs is blurred. If the documents you edit are just objects, which form program state - how is a document editor not an IDE? If the entire system is described in terms of the language, without arbitrary boundaries - all you do with that system, is modify program state. This opens possibilities for "IDEs", that are tailored for a specific job. Which makes sense - if a good program reflects the domain that it models, why shouldn't the IDE do the same?
Conclusion
A lot of my remarks, and the ideas I presented are by no means new or original. But I believe, a lot of these ambitious undertakings failed due to being too early for their time, and deserve proper consideration. As programming languages of this generation go from one standard to the next, they suffer from diminishing returns - why not take a leap, and start a new generation? Thanks for reading.
