Notes from a Programmer

View over Marsden in the snow

Website Development

This website is in its infancy. Eventually it’ll be a blog about developing software, especially high-performance websites. Initially, I’ll be writing about the development of the Ulverston Fallen site.

In preparation for the centenary of the end of World War One in 2018, my cousin Janet, researched the 175 people (known as the fallen), who were killed in action or who died of wounds inflicted during the war. She wanted a website, so I agreed to write one. It needed to give details of each of the fallen; allow people to filter them by criteria such as their date of death, rank and regiment; and show the location of the fallen in Ulverston. The initial version of the website is here.

This presented a lot of challenges. I had written a few websites before but nothing this challenging. Also, I’m not a website designer, I’m a programmer.

There was virtually no budget. Not a problem for developing the site as I gave my time for free (it was a learning project for me), but it could be a problem running it.

In writing the site, I have concentrated on its performance, I wanted the website’s pages to have sub second display times, even on a mobile phone. There is nothing worse than having to wait several seconds for a page to render.

I also needed to be able run it for as little cost as possible. Initially I have concentrated on reducing the load on the web server to an absolute minimum, so it can be run from a very small web server. Ultimately, I want to investigate using Microsoft’s Azure cloud to remove the need for a server altogether.

The following sections give an outline of the approach used to develop the website. Most of the sections assume a knowledge of the products/techniques used.

Microsoft Asp.Net Core

The website was written using Visual Studio 2017/19, the programming language is C# and the .Net Core framework (release 2.2) is used. The website pages were developed as MVC Razor pages. The website currently runs on an Ubuntu web server, using Nginix and Asp.Net Core Kestrel.

Website Database

This is a database of the pages of the website, the following is stored for each one:

Caching Web Pages

In order to reduce the load on the web server, web pages are cached on the browser for a short period.

Whenever a page is sent to the browser:

Whenever the browser needs a copy of the page, it checks to see if it’s in its cache:

If not: the browser requests the page from the server. If it can receive it encoded as UTF-8 and Gzip compressed, the pre-compressed version is sent, which happens most, if not all the time. Otherwise, the page is generated in the usual way using ASP.NET Core MVC, a much slower process. The browser is also sent the page hash and the last modified date time.

The compressed pages for Ulverston Fallen are just under 3MB in size and are loaded into memory (as a single byte array) whenever the web server is started. This makes sending the page highly efficient as it is cached in server memory and does not need to be generated.

If it exists and has not timed out: the browser can use the cached copy, nothing is sent to the server, the server is not involved.

If it exists but has timed out: the browser requests the page from the server and sends it the hash and last modified date and time of its copy. The server uses these to determine if the page has changed. If not, the server tells the browser to continue using its cached copy until it times out again. Otherwise it needs to send the new version of the page.

This technique reduces the load on the web server and improves the performance of the website. However, caching times needs to be kept short. While a page is cached on the browser a new version will not be seen, since the web server is unable to clear a browser’s cache. The new version will only be seen when the cached version times out (or becomes stale).

How web pages are cached on a user browser (or privately cached) are specified by the developer in the website database.

Caching Web Page Subsidiary Files

A website page can use several subsidiary files, these files contain things like images, CSS and the Java Script. These files are cached on the browser forever.

Because the file contents can change (say a new image is used on a web page, or the Java Script/CSS changes), the file path always contains a version number. If the file’s contents change the version number in the path is changed, and the new path is used in the web page. This changes the hash of the page, so the next time the browser asks about a cached copy it is sent the new version of the page. It then requests the new version of the file.

A lot of the files are common to several pages, such as header images, the CSS and Java Script. These files will be downloaded the first time they are required. For subsequent pages, the cached copy can be used, resulting in a fast rendering of the page.

The static files are also cached on a content delivery network (CDN). This means that the web server is usually not involved in sending these files. Ulverston Fallen currently uses Cloud Flare as its CDN, it’s free.

Cascading Style Sheets

Not my favourite, difficult to understand and use.

Up until earlier this year I used bootstrap for all my websites. It gave me a grid system, but I always seemed to be fighting it. I just assumed it was a CSS library, but it turns out that it is more of a user interface library. I was probably using just 10% of the library’s code.

Earlier this year I did a lot of research into CSS, one of the best sites about it is Harry Robert’s CSSWizadry. If you are writing or starting to write CSS it is well worth visiting. I also found the websites of Rachel Andrew, Brad Frost and Cathy Dutton very useful.

I now write my own CSS and feel much, more in control of the look and feel of the site. I also feel I am beginning to understand it. I’ve still got a way to go, but I feel like I am now getting there.

I used bootstrap because of the problems writing CSS for the different types of browser out there in the wild. Sometimes browsers implement CSS in different ways, bootstrap took care of those differences, which meant I did not need to worry about them. It turns out that it is not really a problem these days, most browsers pretty much conform to standard. You just need to take care what CSS features you use, some browsers are more up to date than others. I found this website extremely useful for giving me an idea of features I could and could not use.

My CSS is now split up over several files rather than having it all in one file at the top of the HTML. Jim Nielsen explains why very well, another useful website.

Java Script

Not one of my favourite programming languages.

For the last fifteen years I have been programming with the C# programming language, which is a strongly typed language. I’ve used Java Script on and off for the last ten years or so, but I’ve never really got used to it, especially the fact that it isn’t strongly typed. When defining a variable, I like to specify the type of data it holds, it makes debugging much easier.

There is also a problem running the code on the browser. If the code fails to compile, nothing happens; if the code fails to run due to an error nothing happens; unless there is proper error handling. All the Ulverston Fallen Java Script code runs in try/catch blocks in order to catch errors. Error details are always sent to the web server to be logged, for investigation.

Microsoft TypeScript

Last year I came across Microsoft’s TypeScript language. I now write all my Java Script as typescript and then let the TypeScript compiler compile it into Java Script. It also means I can use Visual Studio to develop the code. This is good, because it does some of the code checking for me.

Like CSS, one of the problems with writing Java Script, is the different types of browser out there, using different versions of the language. With TypeScript I can use some of the more recent features such as classes and interfaces and I can state the type of data I want my variables to hold. I then let the compiler create Java Script code that conforms to the ‘ES5’ version of Java Script, which seems to be supported by most modern browsers.

One thing I want to do soon is to compile my TypeScript code to multiple versions of Java Script code, such as ‘ES5’, ‘ES6/ES2015’, ‘ES2016’, etc. Then, when a Java Script library needs to be loaded for a web page, I can run a small function to detect what version of Java Script the browser implements and then download that version. Then, I can have one code base, to create multiple versions of Java Script code.

Disabling Java Script on a Browser

People can and do turn off Java Script, but statistics show that not many do, very few in fact. The approach taken for Ulverston Fallen is that the site is still usable when Java Script is turned off. However, some of the functionality like filtering and the map are not available.

This is also important for SEO (search engine optimisation); Java Script is never used in the initial rendering of a web page; it is only used to add functionality. This means that failing Java Script does not result in SEO failing.

Without Java Script the main Fallen detail page shows the details of a given fallen person plus a list of all the fallen. Someone using the site can click on a person in the list and a new page is then displayed, resulting in a round trip to the web server, unless it is cached.

When Java Script is available, the list of people can be filtered, for example to only show people who died in August 1915 or were in the King’s Own Regiment. This filtering is only available when Java Script is enabled in the browser. The same applies to the map. These features are covered in more detail below.

When Java Script is available, the list of people can be filtered, for example to only show people who died in August 1915 or were in the King’s Own Regiment. This filtering is only available when Java Script is enabled in the browser. The same applies to the map. These features are covered in more detail below.

JQuery

When I started developing websites, I used the JQuery Java Script library. This was mainly due to the differences in how the various browsers implement Java Script. A similar reason to the one for using the bootstrap CSS library.

Research showed me that most browsers now implement Java Script correctly, to standards. Because of this, I no longer use JQuery or any other Java Script library; I write my own Java Script from scratch. There is a lot of information on the internet about how to do this.

Filtering

Filtering created some interesting problems.

For efficiency, I did not want to keep returning to the web server for decisions. I wanted all filtering to run on the browser, controlled by Java Script.

I did not want the filtering to be part of the web page itself, it just clutters it up. It just would not work on small devices such as mobile phones. For this reason, I decided to implement it as a series of pop up forms. This allowed me to create a single ‘filtering’ or ‘fallen detail’ web page. The ‘fallen detail’ page for Private John Akister can be viewed here.

The ‘fallen detail’ page contains a list of the fallen (by name), and the full details of one of the them, displayed in a ‘detail’ area, and that’s all, nice, simple and fully responsive. Each name in the list can be clicked in order to display their full details. When a person’s name is clicked, an Ajax request is used to get that person’s full details, which are then displayed. The page is never fully re-displayed, only the ‘detail’ part of it. Allowing state to be kept.

At the bottom of a fallen person’s details, there are buttons to allow the user to display the details for the previous, next, first and last person in the list.

Filtering is activated by clicking a button at the bottom of the browser. This results in a series of popup forms being displayed, by which the user enters their filtering criteria.

When the Apply button is clicked, the names in the list are displayed or hidden so that only names matching the filter criteria are visible. The full details of the first person in the list is also displayed. This is all done with Java Script, the web server is not involved (the filtering process makes use of the data attribute of an HTML Element).

Currently, should a user move away from the ‘fallen detail’ page their current filtering criteria will be lost. When they return to the page, all 175 fallen will be listed. This was done because of the infamous EU Cookie law. I did not want, nor see why, I needed to display a cookie message when I am not storing any personal or tracking data. The data is part of the site’s functionality. However, soon, a version of the site will appear, where the filtering criteria are stored locally. On return the same filtered list will be displayed, there will be no cookie message.

Contact Page

This is the only page of the website where data is posted back to the server for processing.

There are currently two types of contact page.

This first is Java Script driven and is a popup. Once the contact details have been entered, the send button is clicked to send the detail using Ajax, to the server. On return a response message is displayed and the user can return to the page they were viewing, without it having moved.

The second version is used when Java Script is disabled and is a full web page, which gets displayed in the browser. When the contact details have been entered, the form is posted back as an HTTP Post request. The user needs to move away from the page they were viewing.

Currently, the contact page needs a web server. In the future I will change the processing to use an Azure Function to remove the need for a server.

Map

Currently full mapping is not implemented.

A lot of thought was put into this, I looked at Google Maps, Bing Maps and Open Street Map. They were all a possibility but were modern; what we needed was an old map of Ulverston. Janet tracked down a map of Ulverston in 1913 which we could use for our website for free (once we had bought it).

The map came as a ‘tif’ file and is massive, 16800 by 11656 pixels, 560MB, so it cannot be used on the actual website. The displayed version is 1,000 pixels wide with a 50% quality, it displays OK, but you cannot really zoom in on it.

The next stage of the project is to produce tiles for the map so that it can be used in a similar way to Google Maps. I will be writing about how I did this soon.

Future

As I said, this site is in its infancy.

Over the coming months I will start expanding the above into a series of articles explaining the points in more detail, starting with the mapping.

Paul Chappell

16th August 2019.