Introducing Restspace - but what's the elevator pitch?

2 May 2020

So what is Restspace? Right now that's the problem. I think it's something which is genuinely a new approach to building software on the web. But that makes it very hard to explain. Right now I'm trying to work out how to engage people with the idea, because if I can't do that this will never get off the ground. Getting people to try out the product, getting investment, getting anyone interested in it in any way needs that.

I'm going to start this blog with how Restspace started. I'm planning on covering a lot of different subjects not just Restspace, but this is where I'll start.

I've been working on the web for many years, but I came from a hardcore computer science background. The first time I worked with a commercial CMS I was a little shocked at how badly it was put together and how much was being charged for it. I spent the next 5 years building one of my own and had the chance as CTO to build a large content site with it: greatbritishchefs.com. It was built on the MS stack, and I updated it to run on .Net Core. It's open-sourced now, you can find it at lynicon.com.

I remember my second-in-command at Great British Chefs saying when he saw it, how can a CMS be so simple and do what we need? I think that was the most gratifying feedback I had. Although it seems quite a few people use it now, it's had about 8000 downloads from Nuget, I get next to no feedback from users.

So it was successful in a very limited way. The problems it had were first, the agency CMS market at which it was aimed is hard to break into because the commitment involved in the choice of a CMS on which to build a large website is major. I've seen a few times what happens generally when people try to use an underspecified and undersupported CMS for a commercial project and it's not pretty. Also it didn't help that since it was the kind of CMS that required programming, it was locked to the ASP.Net Core framework and it's not the hugest or most novelty-loving community.

It had a couple of features that were very powerful. One is it was possible to build plug in modules with a lot of power. I think that extensibility is key to any good platform for writing software: it solves the issue of: this platform saves loads of work but if I ever have a requirement it simply can't do, I'll have to throw it away and start again. The other was that I built a very flexible data layer that could use drivers (technically, the Bridge pattern) to allow you to plug in different data sources, whether a SQL database, a JSON database, or even to run the editor demo on the site for the CMS, I got it to use a cookie on the user's machine to store the content.

The serverless CMS

As time moved on and I learned more about working in the cloud, about serverless, headless CMS, Javascript frameworks and the emerging ideas in the JAMStack movement, I could see the possibility for architecting the functions of a CMS in a new way. It would have 3 layers: a SPA front end would talk to an API. The API is the most interesting part: this would essentially be a framework of plugins for sourcing JSON or flat table data from different back end services, then layering CMS style services on top of that using plugins providing things like URL mapping and versioning. Then the final layer would be those back end services, whether SQL tables (which could be in a preexisting database) or JSON data stores, or even files. One advantage of this setup is the API layer, needing to do no persistence of its own, could run as a serverless function.

For some months I was looking at this idea as a possible basis for a new serverless headless CMS product, which would also avoid the absolutist route that headless CMS vendors have tended to take that they shouldn't put web-based functionality into a headless CMS. This viewpoint conveniently makes it very easy technically to build a headless CMS and charge service prices for basically simple infrastructure. I thought this was unjustifiable particularly considering headless CMSs are accessed over the web, but deign to provide web-related functionality (like url mapping).

Infrastructure as a package

I realised the headless CMS idea was not necessarily the best as the market is stuffed with headless CMSs right now as they're so easy to build. I then had a new idea which was to try and find some way to make infrastructure as code as easy as adding a new package to a codebase. So for instance you might add a package called 'SQL Server instance' which would go to the cloud, set you up a cloud SQL server, and generate the code to transparently connect to it. It was an interesting concept but tricky to make workable. I spent a while thinking around this problem which is essentially making the cloud more programmable and easy to access for people building stuff on the web who don't want to learn loads of server and architecture related knowledge in order to have a cloud back end.

JSON on urls

Then I had another idea while thinking around the concepts of JAMstack and what could be an ideal CMS for use with an SPA. It's a very simple idea, which is instead of using a Static Site Generator to compile markup, why not just have static data files. That way you could get all the caching, simplicity and security advantages of static files, while not forcing your SPA to work on some imposed page model. And if your CMS just reads and writes JSON files, you don't need to compile your site ever. Which is obviously more convenient but also bypasses the problem SSGs have with scaling, which is the time needed to compile a very large site.

I realised that what this would be was an HTTP database. A system where you read and write data to HTTP resources (which live on urls) is incredibly congruent with the core ideas of HTTP. When you build an abstraction over infrastructure, it is always much better for the abstraction to be aligned with the concepts and structure of the infrastructure. A great example of this if you know it is the difference between ASP.Net classic and ASP.Net MVC. The latter is far more aligned with HTTP and consequently simpler, more flexible and more powerful.

But how do you do queries? Well a solution for that I'd found before in my Lynicon CMS was to have a kind of multi-field index of your data in memory. This works surprisingly well because if you limit the fields in each data item which you can query, you can have an entry of less than 1K for each item so you can happily manage maybe 100,000 items like this but then you start to hit a scaling limit.

The other option is to duplicate your data to a JSON data store, and be able to run queries against that via another API.

Building Restspace

So I got the opportunity to build something like this. Initially I built it on Node on Express. If you don't know Express it does fairly standard MVC-style Controller based routing, it's very clean and nice to work with. So I had some controllers mapping for instance a path beginning /json/ into a file space on the server where the rest of the url path was the file path you used to find the JSON file in the file space. So if you had the path /json/a-directory/a-file.json as the url, this would map to a file space on the server at say /var/www/restspace/data/json-files so the full corresponding file path would be /var/www/restspace/data/json-files/a-directory/a.file.json, like you'd expect on a static site.

And then I needed authentication, and I wanted to store the user records in the JSON file space. So let's say you have a user stored on the url /json/users/fred@a-domain.com. Then you would log in via /auth/login/fred@a-domain.com. This suggests that you need to do a url transformation. This is an important concept in Restspace which has a syntax for this called a url pattern. You'd use the url pattern /auth/login/$<0 here where the $<0 pattern means use the rightmost path segment from the original url.

This suggested the key idea of Restspace. If you're configuring urls, why not configure your routes using a JSON file? You could configure a server from prebuilt services. I played with this a bit and started to think it had some possibilities. Maybe you could build most of the things you need in a back end without ever writing code, just using prebuilt services. I was working on a web app at the time where clients would log in, enter data and upload files, quite standard in this respect, so I thought I would try and build the back end just using reusable generic services.

I went through a few versions. I started off using Express, but then I got a clearer idea of the abstraction I was building and realised I needed an internal request bus.

I also built a front end. This is essentially a modular editor for data held in HTTP resources. It also lets you edit the configuration of your server via a GUI. One of its key features is a form generator which uses JSON schema to define the form. The JSON service uses JSON schema to define how JSON data is structured, so to build a CMS style form to edit JSON data, you just read the schema, build the form and populate it with the current data. So here is the core of a headless CMS. But also, the front end can write JSON data. There are quite a few modules out there to generate forms from JSON schema, so you can read the schema to generate a form on the front end. So here is the core of a form builder. The power of interoperability.

Restspace services are best viewed as HTTP Functions which take an HTTP message as input and return another HTTP message as output. For services to call other services, they make an HTTP request. But if the other service is on the same server, this can just be a call made in code, so I needed a setup where external and internal requests were handled the same in the routing system. So the second version was built on the raw Node HTTP library.

Why is this useful?

So I built the whole webapp using just generic HTTP functions. The closest to code is a generic JSON transformer where you express the transform in another JSON file. It let the client do the CMS functions they needed as well as being able to view data and files the user entered and uploaded. I realised I'd got something interesting.

So why is this useful? Obviously it is much faster to build a back end using prebuilt components and writing a minimal amount of code. It's also much easier to experiment and make changes. Also what you find when you start using a platform like this is that you can build a lot of the functionality you would normally get from external services and tools. You can build a CMS. You can build a form builder. You can build an email system and a PDF converter. When you do this, you have all these tools in the same place and built on the same blocks, so they are extremely easy to integrate with each other. When you add the possibility to wrap external APIs like postcode lookup, payment systems etc and have them integrate super easily, you start to see something very compelling.

But what's the elevator pitch?

If you've got this far and understood what I was discussing I'm hoping you will agree with me that Restspace is something original and worthwhile for you to investigate further (you can do that in the Restspace docs). But the problem I have now is distilling what this is and why it's useful into a couple of sentences.