Setting Up This Wiki

Dear future hypothetical reader of this wiki page. You might be needing to set do work on this wiki and would like to know where all of the pieces are documented. You might be needing to set up another similar wiki for a different project. You might even be wanting to set up a similar wiki for yourself. You have arrived at the right place. This is it. I will do my best to cover how it works.

The ikiwiki is a web site compiler. It consumes plain text files in one of the configured markup formats, we have been using markdown everywhere here, and produces a static html site ready for reading. It surrounds the pages with headers and footers include CSS and other necessary niceties. All without the writer of those pages needing to bother about those tedious details. Of course you knew all of that but wanted the details of this particular wiki installation. Let's start at the beginning.

The Beginning

Monday, May 6th, 2024, on the day after LibrePlanet 2024 I (Bob) was sitting at Ian's desk in the FSF office and we were discussing tasks. Ian spelled out needs for the newly formed FSF SysOps team. It included a documentation site. It was decided that it would be ikiwiki as that is well known and well used by many.

ikiwiki
Backend in git
Edit access controlled by a Savannah group
A Savannah group for the FSF SysOps team
Hosted on the Savannah web UI system

These all seem reasonable. Having decided upon the above the implementation falls out rather easily. Here are the points that might not be obvious.

DNS CNAME Configuration

Savannah's web UI frontend system these days is always running on either frontend1 or frontend2. One is running live production. The other is the standby system getting upgraded to the next OS version and getting debugged and setup. The next OS version includes the next PHP version with breaking changes that we need to find and fix. The next OS version includes the next python version and drops the last python version which breaks things which need to be worked through. And on and on with everything. We upgrade and develop on the standby system for new versions and new things, like this wiki. We then flip-flop the live production server from the one to the other. Then the previous machine becomes the standby machine to be upgraded to the next OS version. And the cycle repeats.

Therefore when Ian set up a DNS entry for sysops.fsf.org I asked for a CNAME record alias. And the obvious candidate was savannah.gnu.org which is the main Savannah web server. That DNS name is the one that flip-flops between frontend1 and frontend2. Being an alias CNAME it will follow the DNS changes to savannah.gnu.org and will arrive at the current live production web server.

Having DNS set to point to the frontend machine, whichever one, then we need to set up the web virtual host. Currently the Savannah web UI frontend system is running Apache. Apache is pretty heavy and I will eventually get the site converted over to Nginx. The rest of the Savannah infrastructure is running Nginx. But the web UI frontend is a little bit of a problem child and it is still running Apache.

Web Configuration

We want to have https certificates. We use EFF Let's Encrypt domain validation certificates. We can't create an https certificate until we have an http web site available to do the domain validation. (Though there are other EFF Let's Encrypt authentications but https works very well.) We bootstrap with an http configuration only for an initial setup enough to get a certificate.

Savannah's web UI throws various sites in /var/www and so made a directory there to hold this project.

root@frontend2:~# mkdir /var/www/sysops /var/www/sysops/html

Create an Apache virtual host configuration file with this content. This is a fairly standard basic configuration.

root@frontend2:~# cat /etc/apache2/sites-available/sysops.conf
<VirtualHost *:80>
        ServerName sysops.fsf.org
        ServerAdmin savannah-reports-private@gnu.org
        DocumentRoot /var/www/sysops/html
        LogLevel warn
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ServerSignature On
</VirtualHost>

Link it in to make it active.

root@frontend2:~# ln -s ../sites-available/sysops.conf /etc/apache2/sites-enabled/

Tell apache to reload files.

root@frontend2:~# apachectl graceful

Test that the domain validation path is working correctly such that Let's Encrypt can access the files and do domain validation. We are using the dehydrated client. I drop a foo.txt file containing the magic word "foo" in our configured sandbox.

root@frontend2:~# cat /var/local/dehydrated/www/.well-known/foo.txt
foo

With that in place and the web configuration active test that it works. Test from a machine not the frontend machine. Here I test from my laptop.

rwp@angst:~$ wget -O- -q http://sysops.fsf.org/.well-known/foo.txt
foo

rwp@angst:~$ curl http://sysops.fsf.org/.well-known/foo.txt
foo

That works. Time to get an https certificate.

HTTPS Certificates

Add this line to the dehydrated domains.txt file. Since dehydrated is already configured I am not going to describe setting up the dehydrated configuration itself but just adding a new domain name. There are not alternate names needed so it's just the one name but if there were they those would be added onto the same line. If there were multiple names then the primary name would be first on the line and additional alternate names would be listed out on the same line.

root@frontend2:~# grep sysops /etc/dehydrated/domains.txt
sysops.fsf.org

Run the daily renew script to have it run through the renewal process. This is best because then all of the owner:group permissions are maintained properly. It's very easy to accidentally create files as root otherwise and then things are broken for automated renewal later.

root@frontend2:~# /etc/cron.daily/renew-https-cert-local

That will log all output to the log file. Look for the most recent log file and review it.

root@frontend2:~# ll /var/log/dehydrated/
root@frontend2:~# less /var/log/dehydrated/dehydrated.log.20240515212507
...
Processing sysops.fsf.org
 + Creating new directory /var/local/dehydrated/certs/sysops.fsf.org ...
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for sysops.fsf.org
 + 1 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for sysops.fsf.org authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + Done!
...

Looks good! Let's stitch that into the Apache configuration. Modify the file to be like this. This follows the current Mozilla configuration guidelines for a site for use with most web browsers.

root@frontend2:~# cat /etc/apache2/sites-available/sysops.conf
<VirtualHost *:80>
        ServerName sysops.fsf.org
        ServerAdmin savannah-reports-private@gnu.org
        DocumentRoot /var/www/sysops/html
        RewriteEngine On
        RewriteCond "%{REQUEST_URI}" "!^/.well-known/"
        RewriteRule ^(.*)$ https://%{HTTP_HOST}/wiki$1 [R,L]
        LogLevel warn
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ServerSignature On
</VirtualHost>
<IfModule mod_ssl.c>
<VirtualHost *:443>
        SSLEngine on
        SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
        SSLHonorCipherOrder Off
        SSLSessionTickets Off
        SSLCertificateFile /var/local/dehydrated/certs/sysops.fsf.org/cert.pem
        SSLCertificateKeyFile /var/local/dehydrated/certs/sysops.fsf.org/privkey.pem
        SSLCertificateChainFile /var/local/dehydrated/certs/sysops.fsf.org/fullchain.pem
        SSLOpenSSLConfCmd DHParameters /etc/apache2/dhparam.pem
        ServerName sysops.fsf.org
        ServerAdmin savannah-reports-private@gnu.org
        DocumentRoot /var/www/sysops/html
        RedirectMatch ^/$ /wiki/
        LogLevel warn
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ServerSignature On
</VirtualHost>
</IfModule>

This site is configured to automatically redirect http to https. This means that of particular value is the rewrite rules to avoid doing that redirection for the .well-known path used for domain validation. Since we have modified the file test it again that it handles the domain validation path correctly.

root@frontend2:~# apachectl graceful

rwp@angst:~$ wget -O- -q http://sysops.fsf.org/.well-known/foo.txt
foo

rwp@angst:~$ curl http://sysops.fsf.org/.well-known/foo.txt
foo

How does that path work? How is it configured? That's already been configured to make dehydrated work with Apache. It's in a global Apache configuration file so applies to all of the VirtualHosts. This makes it available to all sites hosted there.

root@frontend2:~# cat /etc/apache2/conf-available/dehydrated-local.conf
Alias /.well-known /var/local/dehydrated/www/.well-known
<Directory "/var/local/dehydrated/www/">
  AllowOverride None
  Require all granted
</Directory>

lrwxrwxrwx 1 root root   39 Feb 10  2020 /etc/apache2/conf-enabled/dehydrated-local.conf -> ../conf-available/dehydrated-local.conf

Savannah Project Group

In order to support edit access controlled by a Savannah group we need to create a Savannah group. For that we registered a project on Savannah and submitted it. In this case I am also a Savannah admin and could have approved it myself but it is good form to have another admin review submissions. Corwin reviewed the project submission and approved it.

https://savannah.nongnu.org/projects/sysops/

Having that as a project allowed an easy well known web administration interface to add other members to the project. Adding members gives them commit access. Later on removing retired maintainers from the project removes commit access.

Backend Version Control

The ikiwiki web site compiler can use any one of several different version control backends. The most popular one is git and that was our choice for this wiki. The Savannah wiki uses Subversion however as an alternative example of a different version control system.

The Savannah Git page describes how to create auxiliary git repositories. Follow those instructions to create and configure the backend git repository.

https://savannah.nongnu.org/maintenance/Git/

In summary this is the process used for the SysOps backend git repository. The git backend lives on the Savannah git server, currently vcs2 but potentially a different updated system in the future. This step depends upon having the Savannah group "sysops" available.

mkdir /srv/git/sysops
chgrp sysops /srv/git/sysops
chmod g+s /srv/git/sysops
git init --shared=all --bare /srv/git/sysops/sysops.git

A straight line history being preferred by this group we enforce this policy against accidents with a pre-receive hook. Copy this from another project using it such as savane.

cp /srv/git/administration/savane.git/hooks/pre-receive /srv/git/sysops/sysops.git/hooks/

There are also instructions for setting up commit notifications. Be sure to use the multi-hook configuration because we will also need a post-receive for updating the wiki page after a push too. But this is a circular dependency so we come back later to install it.

Ikiwiki Sandbox Setup

On the frontend system make a source directory. Change directory to there. Checkout a git clone of the repository we just created. It will be empty at this moment. But it will point to the upstream repository and that is what matters. This stitches the frontend working copy sandbox for the wiki with the backend repository. Call it wiki since that is what people are expecting.

root@frontend2:~# mkdir /var/www/sysops/src
root@frontend2:~# cd /var/www/sysops/src
root@frontend2:/var/www/sysops/src# git clone https://git.savannah.gnu.org/git/sysops/sysops.git wiki

Note that this is the read-only anonymous clone address. The sandbox does not have any way to ssh into the git repository like a member account. Something could be set up but it is not needed. The anonymous clone is best.

We will fix ownership and permissions later. Still more setup is needed.

Ikiwiki Configuration Setup

The ikiwiki compiler reads its configuration from an ikiwiki.setup file. Create this file using ikiwiki. This only creates a template. After doing so then edit the file and configure the rest of what needs to be configured. Which I realize is a very vague hand-waving of a statement. It's somewhat "art" at this point. Compare existing files with the new file and converge them into something you want.

root@frontend2:/var/www/sysops/src# ikiwiki /var/www/sysops/src/wiki /var/www/sysops/html/wiki --url=http://sysops.fsf.org/wiki/ --dumpsetup ikiwiki.setup

root@frontend2:/var/www/sysops/src# $EDITOR ikiwiki.setup
...compare with /var/www/sviki/ikiwiki.setup for example...
...some highlights...

wikiname => 'FSF SysOps',
srcdir => '/var/www/sysops/src/wiki/',
destdir => '/var/www/sysops/html/wiki/',
url => 'http://sysops.fsf.org/wiki/',
rcs => 'git',

With the sandbox in place and the ikiwiki setup file in place we can now run the ikiwiki site compiler itself manually. This is to check for errors. This is where things will fail. Fix up what is needed to be fixed and repeat until it works.

root@frontend2:/var/www/sysops/src# ikiwiki --setup ikiwiki.setup

That will produce a bunch of output. If it looks okay then go look at the web site URL and verify that it looks like it worked there too.

https://sysops.fsf.org/

In terms of future-proofing the document structure the wiki is in the "wiki" directory. The top level is redirecting to this directory.

Automatic Updates After Git Push

At this point everything is working manually. The web site is configured for the static compiled site. But all of the files are owned by root. Change this so that the sandbox and the destination are owned and grouped by the www-data user and group.

root@frontend2:/var/www/sysops# chown -R www-data:www-data src html/wiki

After doing this ownership change never run ikiwiki as root again. Only ever run it as the www-data user. Use sudo for this.

root@frontend2:/var/www/sysops/src# sudo -u www-data ikiwiki --setup ikiwiki.setup

To set up automated updates on demand as changes are pushed we need to set up a communication path between the git repository via a hook and the frontend. The xinetd handles this nicely.

Install this update dispatcher script. The comments are longer than the script. This allows multiple wikis to operate on the same system.

root@frontend2:~# cat /usr/local/sbin/update-ikiwiki
#!/bin/sh

# Update the ikiwiki instance in response to a vcs commit.
#
# The sysops/sysops.git/hooks/post-receive-ikiwiki-update hook will
# send us the pwd such as /net/vcs/git/sysops/sysops.git and we will
# catch it and map that to the wiki to update.

# Copyright 2024 Bob Proulx <bob@proulx.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

if [ "$(id -un)" != "www-data" ]; then
    echo "Error: You are not www-data." 1>&2
    echo "Use: sudo -u www-data update-ikiwiki" 1>&2
    exit 1
fi

IFS= read -r line

case $line in
    */sysops/sysops.git*)
        /var/www/sysops/sysops-refresh
        ;;
esac

exit 0

Install the refresh script.

root@frontend2:~# cat /var/www/sysops/sysops-refresh
#!/bin/sh

## Run this as 'www-data' user:
##  sudo -u www-data ./sysops-refresh

# Copyright 2024 Bob Proulx <bob@proulx.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Based upon a previous subversion variation by Karl Berry.

test www-data = $(id -un) \
    || { echo "Error: please run as user 'www-data'" >&2 ; exit 1 ; }

# just in case, as 'ikiwiki' uses it
HOME=/var/www/sysops
export HOME

# checked-out workdirectory
cd /var/www/sysops/src || exit 1

# Default: refresh ikiwiki
method="--refresh"

if [ -d wiki ]; then
    ( cd wiki && git pull -q )
else
    git clone -q https://git.savannah.gnu.org/git/sysops/sysops.git wiki

    # force a rebuild for a new checkout
    method="--rebuild --verbose"
fi

ikiwiki --setup /var/www/sysops/src/ikiwiki.setup $method

Install the xinetd configuration file for this.

root@frontend2:~# cat /etc/xinetd.d/ikiwiki
service ikiwiki
{
        type                    = UNLISTED
        port                    = 9417
        socket_type             = stream
        wait                    = no
        user                    = www-data
        server                  = /usr/local/sbin/update-ikiwiki
}

Tell xinetd to reload its configuration files by sending it a SIGHUP Signal. This can be done by any of several methods. This is one way.

root@frontend2:~# kill -s HUP $(cat /run/xinetd.pid)

Open this port up through the firewall. On Shorewall systems like Savannah's frontend this is adding a like such as this to the Shorewall rules file.

root@frontend2:~# $EDITOR /etc/shorewall/rules
ACCEPT  net:209.51.188.0/24     fw      tcp     9417  # ikiwiki update
root@frontend2:~# shorewall safe-restart
root@frontend2:~# $EDITOR /etc/shorewall6/rules
ACCEPT  net:2001:470:142::/48   fw      tcp     9417  # ikiwiki update
root@frontend2:~# shorewall6 safe-restart

This sets up port 9417 (one down from the git port) as a port open to the subnet to catch notification events from the git hook script that a push has happened and therefore a pull to the sandbox and a refresh of the static site is needed.

On the backend git server add this hook to throw the notification. The comments exceed the script itself.

root@vcs2:/net/vcs/git/sysops/sysops.git/hooks# cat post-receive-ikiwiki-update
#!/bin/sh

# Send a message to the web server that it needs to update.

# Copyright 2024 Bob Proulx <bob@proulx.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script is run after receive-pack has accepted a pack and the
# repository has been updated.  It is passed arguments in through stdin
# in the form
#  <oldrev> <newrev> <refname>
# For example:
#  aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master
#
# In this case we don't have any need for this commit information.
# Instead we simply use this to update the checked out working copy
# whenever new files are pushed here.
#
# Discard git input as unused here.
cat >/dev/null

# Send our present working directory to the web server.  The catcher
# script listening on 9417 (git port minus one) there maps to the
# repositories using it.
#
# Example: /srv/git/sysops/sysops.git/hooks

pwd | nc -N sysops.fsf.org 9417

By sending the pwd to the web frontend the dispatcher there can map the wiki being updated to the sandbox to update and the wiki to compile. This supports multiple wikis.

The post-receive script with the multi-hook structure that calls it.

root@vcs2:/net/vcs/git/sysops/sysops.git/hooks# cat post-receive
#!/bin/sh

# Run all post receive hooks.

# Copyright 2024 Bob Proulx <bob@proulx.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script is run after receive-pack has accepted a pack and the
# repository has been updated.  It is passed arguments in through stdin
# in the form
#  <oldrev> <newrev> <refname>
# For example:
#  aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master

lines=$(cat)
printf "%s\n" "$lines" | ./hooks/post-receive-ikiwiki-update
printf "%s\n" "$lines" | ./hooks/post-receive-git_multimail-sysops

This also calls the post-receive-git_multimail-sysops hook which emails out commit notifications.

It's somewhat difficult to test this feature. Clone the repository. Make a commit update. Verify that it propagated. Did it work? If YES then move on. If NO then debug.

git clone ssh://git.savannah.gnu.org/srv/git/sysops/sysops.git

See also HowToEditThisWiki which is now germane at this point.

Remember that if setting up a brand new site initially that you are the only one who knows about it yet. At that time of creation and initial setup it is okay to rewind the repository and try the same thing again. It is not needed to make a long series of silly commits and pushes. That only pollutes the log history. And it looks silly. Simply reset the repository and then push again. This is a lot faster and allows more rapid development turn cycles. It is only later when the repository is known and in shared use by others that one should never rewind the public repository.

Dealing With Offline Servers

The backend git repository getting updated is on a different system than the frontend web server holding the static compiled site. It's possible for the update notification to be lost between them due to network outages or other events. The web frontend might not be online at the time that a git push is published. Also at this time there are two frontend systems. One is the live production server and the other is the standby development server for the next update. Only the production system will get the notification to update the sandbox and compile an updated static web site. How will this update robustly if the update notification is lost?

The usual method to deal with this is to update the site by cron on a periodic basis. Usually hourly is sufficient. This way even if the web host is offline or the network is broken the site will eventually sync up with the current published backend. This is a typical technique often used in such cases.

The Savannah wiki is updating at 23 minutes past the hour. I arbitrarily added the sysops wiki refresh at 21 minutes past the hour. It runs quickly. It should be done in the two minutes allocated.

In some circumstances cron jobs might get stuck and never complete. Such as when waiting for a network mounted file to be read or written. In those cases it is good to put in a semaphore such that only one cronjob of that type will ever run at a time. This prevents the system from getting into a state where every run adds another stuck process. This didn't seem necessary here but I wanted to mention it as a possible needed thing to do since it is always something to think about when adding a cronjob.

root@frontend2:/var/www/sysops# cat /etc/cron.d/savane
# minute (0-59),
#   hour (0-23),
#     day of the month (1-31),),
#       month of year (1-12),
#         day of the week (0-6 with 0=Sunday).
21 * * * * www-data /var/www/sysops/sysops-refresh
23 * * * * www-data /var/www/sviki/sviki-refresh

Sync the Frontend Systems

In the above description all of the work was done on frontend2 the live production server. In actuality I did all of the work on frontend1 the standby system used for testing the next OS. I was able to tell my web browser to go there by temporarily adding a /etc/hosts entry on my system pointing sysops.fsf.org to frontend1's IP address. And then when everything was working as I wanted used rsync to copy all of the files and directories to the other system. Here is a list of all of the files involved that we created or modified that need to be in sync between the two systems.

/etc/apache2/sites-available/sysops.conf
/etc/apache2/sites-enabled/sysops.conf
/etc/cron.d/savane
/etc/dehydrated/domains.txt
/etc/shorewall/rules
/etc/shorewall6/rules
/etc/xinetd.d/ikiwiki
/usr/local/sbin/update-ikiwiki
/var/www/sysops/

That last is a directory of files. Simplest to sync everything in it.