Get blog updates
Now that 2020 is here, let’s look at what we can expect from the next decade in software. As Web developers, our solutions can help shape the organizations we work for. The tools we build and the architectural decisions we make have a compounding effect on society at large. What are the new trends, and will they help empower or enslave people?
The Old Trends
The trends in the last 10-20 years have led to more and more centralization of the Web, consolidation of power in the hands of the largest services (Facebook, Google, Amazon, Reddit) and their extended ecosystems. Between these and the large publications, the “independent Web” has suffered a tremendous setback. Most people and organizations trust large corporations with proprietary algorithms to manage their data, identity and brand. This has led to massive new issues for individuals and society, involving governments and corporations, and how we all relate to one another. Attempts to resolve these issues have spawned some projects to decentralize the Web.
When the Web was born, browsers rendered HTML documents, and there was very little support for client-side programming. Whatever Javascript support was introduced over the next decade was inconsistent because of the browser wars, and led to Javascript libraries to bridge the gap, of which jQuery emerged as the winner. The last 10 years saw the rise of Angular and React, new versions of Javascript and HTML5 Web APIs, which finally made front-end Web programming a powerful proposition on the most widely deployed platform in the world.
Client-First Web Apps
As Javascript was maturing, a conventional wisdom has developed among most Web developers, that you should render the HTML on the client side, and then progressively enhance it with Javascript. This was considered best practice and recommended by pretty much every authority from 2009 to 2018.
In this decade, Web developers will turn this conventional wisdom on its head, and start to consider progressive enhancement to go the other way:
- First, develop static HTML, CSS and Javascript
- Make Javascript fetch data from servers, render it on the client
- Progressively enhance the site for older environments (Server-rendered HTML)
What follows are multiple reasons for why this is the better approach going forward. This one shift in how we approach Web development will have profound technological and societal implications.
Distributing Software
1. Separation of concerns. It pays to decouple the rendering of an interface from the delivery of code / markup. That way we are not tied to one type of app delivery — that of a server on the web sending our executable code. We are able to sideload apps, download them from app stores, and update specific files when they have changed. And we use one language for each task, too: JS is the code. HTML / Handlebars / etc. can be used for templates / markup. CSS is used for presentation. JSON or XML is used for data. After you have done this, if you want to pre-render HTML on a server for AJAX, you can, but will start to feel “dirty” as you’ll be coupling things unnecessarily again. Things are going this way as headless CMSes are making an appearance, while Cordova, Ionic and React Native represent other ways of delivering code through app stores.
2. Trust. You can’t trust what code is running remotely (although Signal has been experimenting with using Software Guard Extensions by CPU makers, originally designed for DRM, to go the other way and ensure what code runs on a server). But even if you can, you have no guarantee some other process won’t steal or corrupt your data. The Trusted Computing Base should not include arbitrary amounts of remote sites shipping code at any time. Decoupling how the code is loaded (see point #1) onto your client allows you or your user agent to verify checksums and certify that it is indeed the code you think it is. And it is that code that should be managing your data and using the personal keys on your device. Package managers and app stores will be able to distribute code that has been audited by third-party security firms, and people will be able to trust them.
3. Decentralization of Code. As the next decade unfolds, we will find that code bases don’t necessarily have to live shrink-wrapped on a specific server. Rather, clients can use multiple interoperable software modules and versions and can have multiple app stores and distributors in the future helping maintain repos and package managers for end-users and organizations. We will probably see automated package management become more user friendly in the 2020s, as we already have a docker container culture, we have browser based package managers etc.
Data Ownership
4. Decentralization of Data. This is the big one in terms of effect on society. By having web servers render your webpage, you are implicitly locking yourself and your organization into the type of model where the servers store and access the data in a private database. They have enough data to render everything, apply access control rules to manage what you can read and write, and so on. Instead, we as a society need to empower people and their client side apps, and push the logic of fetching data, caching it and assembling it to the user agents. We can use capabilities / access tokens for data instead of a centralized site rendering HTML. In this way, always inverting the progressive enhancement is an activist position to change society against the abuses of power like the ones listed here.
5. Reliability After the 2015 ISIS attacks in Paris, countries around the world expressed solidarity with the French people. French colors were flown, but similar-sized attacks at the same time by ISIS in Beirut were totally forgotten. Facebook rolled out a feature to customize one’s profile with the French flag superimposed, but only the French flag. So we used the Qbix Platform to quickly build a small app called customizemypic.com to allow anyone to change their profile picture to a flag of their choice. The goal was to make a statement and express solidarity with people in Beirut, Baghdad and other areas hard hit by terrorism. Today, that same app is no longer able to do its core function because Facebook removed any way for users to give permissions to apps to upload a photo on their behalf. This is what happens when you rely on third parties to announce what you can and cannot do with your own profile picture. The most extreme reliability is achieved by an offline-first approach, which is a close cousin of the client-first trend that will grow in the 2020s.
6. End to End Encryption. Server-side rendering perpetuates a culture where the server has all the data unencrypted. Even if the data is encrypted at rest, the served holds all the decryption keys and is one central target for hackers, government agencies, and advertisers. Rendering things client-side goes hand-in-hand with a culture of people storing their own keys on devices of their choice, and letting key management and password management be the domain of operating systems and trusted computing bases, not random websites.
Data Delivery
7. Bandwidth. Ever since Steve Jobs presented WebObjects, we have wanted sites to render dynamically. Well, that often involves looping through various items and rendering each one. It is extremely wasteful to send the HTML results of rendering hundreds of items to a client, when you could have just sent the data, which would then be “hydrated” into 5. templates by the client. However, I can understand pre-rendering just the items above the fold (if one could estimate this number, not knowing the size of the window on the first request).
8. Caching Issues. Often, you have subtle and pervasive changes on every page when a person is logged in vs out. (I should remark that “logging in” into a site itself is an artifact of “centralized” thinking, but I digress.) Their avatar might be rendered in various places. New links are shown that might not be available otherwise. And new information may be shown that access control and discovery suggestions determines they can see. If you render everything on the served, there is no way to cache most of the fragments of the page, because they are changing. If you render client-side, all this comes for free.
Next Steps for Web Developers
So by now you may be convinced that “client-first” is a good design pattern and progressive enhancement can be implemented later, by “speeding up” the first render, and by making it available to “dumb” crawlers and user agents who don’t execute Javascript in 2020. Here is how that would actually look, in actionable terms:
9. Preloading. Okay, now that you are rendering everything client-side, you can implement a mechanism to preload data from the server. Perhaps put all the JSON in one file and send it over on the first render. Which — remember — happens only when you use a Web Browser to visit a page directly, a very specific scenario. Every other request besides that, including subsequent requests from a web browser, don’t need this preload. It’s an extra flair you can add for that specific use case. So the preloaded data comes and your Javascript will already have it and will render the HTML synchronously and quickly.
10. Static Site Generation. The most popular static site generators today are still pretty narrow in their use-case. They help with blogs and publishing, eg Jekyll, Hugo, etc. But if you already have a dynamic site, you can sort of transform it into a static site by having a server-side script request some (dynamically specified) set of pages and render them to some related static html documents, and then begin sending 301 redirects on the dynamic pages to permanently tell browsers to go to the static pages in the future. (Because rewriting all links in your site may be infeasible). This approach runs into problems I described in point 5 — so a naive approach would only work for publicly accessible pages that don’t change when someone is logged in. You may still need to add client side JS to “fix up” at least the basic affordances for logged-in users.
11. Caching, Throttling, Batching. Every function that issues a GET to a server could be made smarter to take into account caching, throttling, and batching of request. In Qbix Platform we have middleware methods like Q.getter and Q.batcher which take any getter functions and transform them into ones that do the above things. They work with instances of the Q.Cache class which have adapters to cache in the current document, in sessionStorage or localStorage. (And later maybe in IndexedDB etc.) They have hooks for additional steps when serializing and unserializing objects etc.
12. Deployment. We have built various scripts in the Qbix Platform to help not just Qbix apps but general websites be deployed, including:
- combine.php (which fixes css files to have absolute URLs, minifies, combines etc. all scripts and stylesheets you tell it, and Qbix Platform rewrites links when you add scripts and stylesheets)
- urls.php (which goes over all the static files inside the publicly accessible web directory, checks their modification time and builds indexes in php and JSON as to what changed since the last time it was run… allowing clients to eg download a diff of what files changed since last release. If also calculates hashes for SRIs and Qbix apps automatically append the correct SRI information when rendering certain tags, and the correct timestamp for cache-busting.)
- static.php goes over your site, whatever it is, and generates a static version that you can serve, and 301 Redirects for the rest. It helps turn a lot of your public pages into a static site.
- bundle.php (which builds a bundle for Cordova app deployment. The client-side Cordova plugin from Qbix intercepts requests and sees if there is already a local version in the bundle, and serves that if needed. This works with https as well. One of the many client-side Cordova features that allows us to turn websites into native social apps. Sadly, Apple doesn’t support Service Workers or AppCache for WebView.)
These PHP scripts are part of a larger open-source ecosystem that we developed to help Web Developers take advantage of modern security practices and build advanced social web / native apps like this one for Andrew Yang.
The Platform is free and open source, with tutorials and a guide on how to use it. If you are interested in learning more about the Qbix Platform, then reach out to us. And if you have a meetup or group that you’d like us to present and answer questions, let us know.