Debian Linux mit Firejail absichern!

Für Debian Linux gibt es hier im Blog bereits eine extrem interessanten Artikel zum Bau eines Debian basierten InHouse NAS mit verschlüsseltem Software-Raid (https://kmj.at/2018-06-14-inhouse-nas-mit-verschluesseltem-software-raid-teil-1). Teil 2 wird hier bald erscheinen.

Zusätzlich finden sich hier im Blog viele Artikel über den Einsatz von Linux und Open Source bei Unternehmen und Privatpersonen. Stöbern Sie einfach durch das https://kmj.at/inhaltsverzeichnis/ und lesen Sie weitere Artikel. Am besten starten Sie hier https://kmj.at/erfahrungsbericht-nach-vollst%C3%A4ndiger-umstellung-des-betriebes-auf-open-source-software/

Für spezielle Anwendungsfälle gibt es im https://digitalbooks24.com/ Buchshop verschiedene E-Books.

Sollte Ihnen das Blog gefallen, freue ich mich, wenn Sie für das EURAFRI Non-Profit Projekt eine kleine Spende erübrigen können. https://eurafri.com/contactanddonate/

Ein kritischer Punkt in der Absicherung von IT Systemen, insbesondere von Arbeitsplätzen auf denen Benutzer ohne IT Kenntnisse arbeiten, sind sogenannte Zero-Day Exploits (https://de.wikipedia.org/wiki/Exploit) und Software die offen, oder auch versteckt über Webseiten, E-Mail, Messenger und ähnliches an Benutzer übermittelt wird.

Es gibt einige wenige Exploits wo der Benutzer gar nichts machen muss und dieser Schadcode ausgeführt wird. In den meisten Fällen klickt aber der Benutzer in einer durch ein kommerzielles Betriebssystem sehr verbreiteten - Weiter, Weiter, Ja, Ja, OK - Mentalität auf Fragen und führt damit den Schadcode mit seinen Berechtigungen selbst aus. Dabei wurde niemand gehackt, sondern ein Benutzer hat einfach jegliche Sorgfaltspflicht ignoriert und das System und eventuell auch Daten am Server vorsätzlich vernichtet.

Die Aufgabe verantwortungsvolle Administratoren und Benutzer ist es nun die System bestmöglich gegen diese Benutzerfehler abzusichern, ohne die Möglichkeit zu arbeiten zu weit einzuschränken. Generell gilt aber, Sicherheit geht vor technisch unqualifizierten Aussagen und Wünschen der Anwender. Wir haben hier alle Arbeitsplätze auf Debian Linux umgestellt, aber Firejail sollte auch für jedes andere Linux gleich funktionieren.

Bei Debian is Firejail in den offiziellen Packages, d.h. ein

sudo apt-get install firejail

erledigt die Installation in kürzester Zeit.

Nun sollten wir alle Programme, welche Daten im Internet zeigen, bzw. austauschen so absichern, dass kein Zugriff auf interne Ordner und Dateien möglich ist. Nur ein minimaler Zugriff über Download Ordner solle gegeben sein. Zusätzlich sollen gewisse Bereiche nicht ausführbar markiert sein, damit das Programm in der Firejail Sandbox keinen Schaden anrichten kann.

Firejail hat eine riesige Liste an Standardprogrammen, für welche bereits sehr gute Profile mitgeliefert werden. Da es immer wieder lokale Abänderungen gibt, wurde folgende Art der Einrichtung gewählt:

# Einrichtung eines Firejail Config-Ordners im Home Verzeichnis
mkdir ~/.config/firejail/
cd ~/.config/firejail/
# Erstellen eines Benutzer Profiles, welche Standard Einstellungen beinhaltet
touch 0_KMJ_block.profile
# Als erstes geben wir Firefox in eine Firejail Sandbox
touch firefox.profile

Nun setzen wir im Benutzer spezifischen Konfigurationsdatei die gewünschten Einstellungen, die je nach Benutzer unterschiedlich sein können. Ich habe z.B. folgende Einstellungen gesetzt:

Dazu editieren wir: 0_KMJ_block.profile

blacklist /opt
blacklist ~/VirtualBox*
blacklist ~/OwnCloud*
blacklist ~/.Private
blacklist ~/Private
blacklist ~/.thunderbird
blacklist ~/Software
blacklist ~/.mozilla
blacklist ~/.ssh
blacklist ~/.putty
blacklist ~/.xca
blacklist ~/.vnc
blacklist ~/.remmina
blacklist ~/.thinclient*
blacklist ~/.android
blacklist ~/.joplin
blacklist ~/TEMP
blacklist ~/.scribus
blacklist ~/.FBReader

Diese Einstellungen sind aber abhängig von den Ordnern im Benutzer Home Verzeichnis.

Nun wollen wir den Firefox so einschränken, dass ein eventuell angerichteter Schaden möglichst gering ist. Dazu editieren wir: firefox.profile

## firefox Profile
## KMJ (https://kmj.at)
## 20200719
## Version 1.00

## you are able to test with:
## /usr/bin/firejail --profile=~/.config/firejail/firefox.profile  /bin/bash
## or start Firefox with (1 line!, Usinge Firefox ESR here)
## /usr/bin/firejail --profile=~/.config/firejail/firefox.profile  /usr/lib/firefox-esr/firefox-esr %u

###################################################################
## we include package firefox.profile and our standard config
## blacklist a additional folder too
###################################################################

include /etc/firejail/firefox.profile
include ~/.config/firejail/0_KMJ_block.profile
blacklist ~/Bilder*

Neue Konfigurationen sollten immer getestet werden. Dazu startet man firejail mit der bash:

/usr/bin/firejail --profile=~/.config/firejail/firefox.profile  /bin/bash
Reading profile /home/user/.config/firejail/firefox.profile
Reading profile /etc/firejail/firefox.profile
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/firefox-common.profile
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-devel.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/whitelist-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Warning: networking feature is disabled in Firejail configuration file
Reading profile /home/user/.config/firejail/0_KMJ_block.profile
Parent pid 13973, child pid 13974
Warning: An abstract unix socket for session D-BUS might still be available. Use --net or remove unix from --protocol set.
Post-exec seccomp protector enabled
Seccomp list in: !chroot, check list: @default-keep, prelist: unknown,
Child process initialized in 116.44 ms
bash: /usr/share/bash-completion/bash_completion: Datei oder Verzeichnis nicht gefunden
$

Nun ist der Zugriff auf Dateien massiv eingeschränkt. Das Firefox Standard Profil lässt nur den Download Ordner im Home Verzeichnis zu und schützt damit andere Dateien vor einem Befall durch Ransom, Trojaner und Viren. Weiters darf in dieser Sandbox im Download kein Programm ausgeführt werden. Das erhöht den Schutz zusätzlich.

Wenn dementsprechend getestet, dann stellen Sie alle Starter in der Leiste, bzw. am Desktop auf:

/usr/bin/firejail --profile=~/.config/firejail/firefox.profile  /usr/lib/firefox-esr/firefox-esr %u

um. Theoretisch können Sie auch im /usr/local/bin eine Datei firefox-esr (755) mit diesem Inhalt anlegen.

Das funktioniert aber nur, wenn sichergestellt ist, dass /usr/local/bin vor /usr/bin im Path zieht:

PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Sicherer ist es die Starter zu ändern und zu wissen, wo Firefox in der Sandbox gestartet wird. Andernfalls kann bei Änderung des Path die Firejail Wirkung ohne Warnhinweis verloren gehen.

Nach dem Start von Firefox gehen Sie unter Datei->Öffnen und prüfen im Datei Browser noch einmal, dass alles gesperrt ist. Damit haben wir die Sicherheit beim Browsen im Internet dramatisch erhöht und insbesondere die Gefahr eine Ransom Verschlüsselung des Dateisystems ist nun extrem reduziert.

Nachdem das mit Firefox so gut funktioniert, stellen wir weitere Programme in eine Sandbox. Insbesondere bei der Verwendung von AppImages empfiehlt sich die Verwendung von Firejail.

Deltachat Appimage

Dazu laden wir das AppImage, hier das Deltachat AppImage in ein Verzeichnis im

# Download nach:
/opt/Firejail/deltachat/Software/DeltaChat.AppImage
chown ihrbenutzer:ihregruppe /opt/Firejail/deltachat/Software/DeltaChat.AppImage
chmod 750 /opt/Firejail/deltachat/Software/DeltaChat.AppImage

Danach erstellen wir das Profil:

Hilfe findet man hier:

https://man7.org/linux/man-pages/man5/firejail-profile.5.html

oder in der Shell mit dem

man firejail-profile

Kommando.

Erstellen der Datei: deltachat.profile

## Deltachat Profile
## KMJ (https://kmj.at)
## 20200719
## Version 1.00

## you are able to test with:
## /usr/bin/firejail --profile=~/.config/firejail/deltachat.profile  /bin/bash
## or start with (1Line!)
## /usr/bin/firejail --profile=~/.config/firejail/deltachat.profile  --appimage /opt/Firejail/deltachat/Software/DeltaChat.AppImage

###################################################################
## NOTES:
## you must d/l the appimage to /opt/Firejail/deltachat/Software
###################################################################

noblacklist ${HOME}/.config/DeltaChat
noblacklist ${HOME}/.cache

noblacklist /opt/Firejail/deltachat
blacklist   /opt

## Disable access
include /etc/firejail/disable-common.inc
include /etc/firejail/disable-programs.inc
include /etc/firejail/disable-passwdmgr.inc
include /etc/firejail/disable-devel.inc
include /etc/firejail/disable-exec.inc
include /etc/firejail/disable-interpreters.inc
include /etc/firejail/disable-programs.inc

## Security filters
caps.drop all
netfilter
nodvd
nonewprivs
noroot
protocol unix,inet,inet6
nogroups
shell none

## Filesystem
disable-mnt
private-cache
private-tmp
private-dev
private-etc resolv.conf

private-bin bash,sh,grep,tail,env,gpg,id,readlink,dirname,test,mkdir,ln,sed,cp,rm,getconf

private /opt/Firejail/deltachat

# Blacklist
blacklist /boot
blacklist /mnt
blacklist /media
blacklist /root
blacklist /srv


RIOT.im / Element.io / Matrix.org

Danach mit der bash wieder testen und dann den Starter von Deltachat ändern. Natürlich sollen auch Electrum, Linphone, Midori, Thunderbird, vlc und andere in eine Sandbox.

Da immer mehr Unternehmen und Benutzer auf Element/Riot umsteigen und proprietäre Datensammel-Messenger verbieten will ich hier aber noch zeigen, wie man den von Millionen benutzten, extrem beliebten, sicheren, kostenlosen, Open Source Messenger Riot.im (seit July 2020 jetzt Element.io) in eine Sandbox gibt.

Der Element Matrix Client installiert sich unter /opt/Element. Für unsere Profile benötigen wir noch:

mkdir -p /opt/Firejail/element/.cache
mkdir -p /opt/Firejail/element/.config
mkdir -p /opt/Firejail/element/.local
mkdir -p /opt/Firejail/element/Transfer

Wir reichten es so ein, dass der Messenger nur auf das Transfer Verzeichnis zugreifen darf. Bilder die hochgeladen werden müssen vorher dort hinein kopiert werden. Speichern geht nur in dieses Verzeichnis. Danach weg kopieren, wo auch immer der endgültige Speicherort ist.

HINWEIS: Sollte der Element Matrix Client bereit installiert sein, einfach stoppen und die Verzeichnisse ~/.config/Element* nach /opt/Firejail/element/.config verschieben.

Danach das element.profil anlegen:

## Element Profile
## KMJ (https://kmj.at)
## 20200719
## Version 1.00

## you are able to test with:
## /usr/bin/firejail --profile=~/.config/firejail/element.profile  /bin/bash
## or start with
## /usr/bin/firejail --profile=~/.config/firejail/element.profile  /opt/Element/element-desktop --profile=MultiProfileName
##  --profile=MultiProfileName is optional

###################################################################
## NOTES:
## home is /opt/Firejail/element
###################################################################

noblacklist ${HOME}/.cache/element
noblacklist ${HOME}/.config/element
noblacklist ${HOME}/.local/share/element

noblacklist /opt/Firejail/element
blacklist /opt/Firejail

## Disable access
include /etc/firejail/disable-common.inc
include /etc/firejail/disable-programs.inc
include /etc/firejail/disable-passwdmgr.inc
include /etc/firejail/disable-devel.inc
include /etc/firejail/disable-exec.inc
include /etc/firejail/disable-interpreters.inc
include /etc/firejail/disable-programs.inc

## Security filters
caps.drop all
netfilter
nodvd
nonewprivs
noroot
protocol unix,inet,inet6
nogroups
shell none

## Filesystem
disable-mnt
private-cache
#otherwise we have no icon in taskbar
#private-tmp
private-dev
private-etc resolv.conf,localtime

private-bin bash,sh,grep,tail,env,gpg,id,readlink,dirname,test,mkdir,ln,sed,cp,rm,getconf

private /opt/Firejail/element

# Blacklist
blacklist /boot
blacklist /mnt
blacklist /media
blacklist /root
blacklist /srv

Und auch hier wieder testen, testen, testen und danach erst den Starter ändern.

Dieser Beitrag zeigt, dass es mit sehr geringem Aufwand möglich ist die Sicherheit massiv zu erhöhen. Bei einem Rollout für verschieden Systeme und mehrere Benutzer bietet sich eine Verteilung über NFS oder SMB basierte Home-Verzeichnisse, bzw. eine Verteilung über Owncloud (https://kmj.at/professionelle-zusammenarbeit-einfachstes-handling-und-gesch%C3%BCtzter-austausch-von-dateien-mit-owncloud/) an.

Damit ist es Administratoren möglich die Verwaltung zentral durchzuführen und den Aufwand zu reduzieren.

Natürlich ist mehr Sicherheit im Vorfeld etwas mehr Aufwand und das Warum sollte auch den Benutzern in einer Schulung erklärt werden. Wenn Benutzer das “Warum” verstehen wird mit der sicheren Lösung gerne gearbeitet.

Und ein Befall mit Verschlüsselungs- / Crypto- / Ransom-Software erzeugt einen Aufwand der brutal höher ist.

Sind Sie sicher, dass Sie ein ausgelagertes Backup haben, mit welchen Sie vom Stand 0 Ihre IT wieder herstellen können?

Sie sehen, der geringe Aufwand für mehr Sicherheit steht in keiner Relation zu einem Befall.


Infos & Support

Riot.im Räume:

Ich habe in der Matrix (kostenlos, Open Source) einen Raum eingerichtet in dem ich mich mit Freunden und netten Menschen austausche. Freundliche Menschen mit netten Umgangsformen, deutsch oder englisch sprechend, sind dort gerne willkommen.

  • Public room KMJ & friends (EN/DE) in der Matrix: #kmj:matrix.ctseuro.com
  • Eintreten durch Eingabe von /join #kmj:matrix.ctseuro.com in Ihrem Element Matrix Client (Riot).

Element.io / Riot.IM Info

INFO Im July 2020 wurde Riot.im in Element https://element.io/ umbenannt. Es handelt sich um eine reine Namensänderung bei gleichbleibenden Funktionen des Matrix Clients.

Anleitung:

https://kmj.at/riot-im-messenger-kurzanleitung-fuer-benutze-updated/

Profi Support

  • Professionelle Hilfe bei Open-Source Lösungen bietet seit 1985 die CTS GMBH https://cts-solutions.at (professionelle IT Lösungen seit 1985!)

How to Setup your Matrix Synapse Messaging Homeserver (Apr 2020/English)

Feel free to link to this page if you like this blog post, but do not copy/paste the text somewhere else without asking. If you want to post the text in full or partial on your site or blog you must add in bold inside a h1 tag:

This is the server side part, find client intro here:

This is the server side part. If you need a tutorial for the client side using Riot messenger I wrote an very detailed english version for my https://EURAFRI.com Business Networking Project! If you need more details than shown here please visit https://eurafri.com/info-whatsinside/ and choose the Riot area. In German you can find infos here: https://kmj.at/riot-im-messenger-kurzanleitung-fuer-benutze-updated/

FreeBSD PostgresQL Update 2019-10-30:

Be careful with updating, see updating section below. FreeBSD pkg wants PostgreSQL 11.5 now for py36-psycopg2 and turnserver. You can end up with deinstalled PostgreSQL 9.5 Server and create a problem. Read section below carefully!

Security Fix 2020-04-03:

The Twisted fix was already mentioned in the 1.12.0 release note, but due to a sqlite module load error, even one does not use it, updates was delayed till 1.12.1 comes out. Please upgrade Twisted to >=20.3.0 while env is activated using:

pip install 'Twisted>=20.3.0'

Cross Signing Update 2020-05-06


Just to be clear

This is how I have done it. Whatever you do, you do it for yourself. I am in no way responsible for anything. Everything you do is on your own risk!

Public Matrix Room

There is a public room #kmj:matrix.ctseuro.com where I answer questions from time to time in English or German. Please join if you have questions.

A guide to setup a Matrix Synapse Homeserver

End-to-end encrypted messenger and collaboration server using Python 3.7. We will add Turnserver for voice based communicatioa, add a firewall and more. If we are done you run your own homeserver which is able to communicate with all other matrix servers on the Internet via federation. Prefered Client on Linux, Mac, Windows, Android and IOS is https://riot.im .

At the time of writing this age we use FreeBSD 12.1. Newer FreeBSD Versions are very similar to setup.

prepare everything

For bare bone setup write the ISO to some USB stick, for virtual setup using Proxmox or VMWare, place the ISO on your host and create a virtual machine booting with the ISO file.

The required resources are:

  • RAM 4-6 GB for up to 100 users
  • 2-3 Cores
  • 80GB Harddisk
  • network card
  • static IP address with DNS entry pointing to it

Assume we create the homeserver matrix.ourdomain.com on IP 1.2.3.4. Each Matrix Synapse server is called homeserver. We have pointed an DNS A record matrix.ourdomain.com to IP 1.2.3.4. Check with nslookup matrix.ourdomain.com, or ping matrix.ourdomain.com that the name resolves.

NOTE: you must change ourdomain.com to your domain name and 1.2.3.4 to your IP!

Setup FreeBSD

Boot the ISO file and do a minimum setup of FreeBSD as described on FreeBSD.org. Enable sshd to be able to login. This sets the hostname, creates your user account, formats the harddisk and creates the partitions. Do not install ports or other stuff, except sshd. Finally reboot and login with the user you created. Detailed FreeBSD setup instructions is out of the scope of this document.

Setup additional software

use su to become root and enter y if asked to fetch and install pkg system.

  • pkg install ca_root_nss gcc mc bash curl wget
  • pkg install libffi libxslt jpeg
  • pkg install py37-virtualenv
  • pkg install py37-psycopg2
  • pkg clean -a

Create Environment

  • mkdir -p /home/synapse/env
  • virtualenv-3.7 -p python3.7 /home/synapse/env

Always use sh for working with virtualenv.

sh
. /home/synapse/env/bin/activate
(env) #

The env # must be shown and do not forget to add the . at the beginning of the line. If it is not shown you have made some error! Go back and recheck if this happens!

Setup basic Synapse inside virtualenv

While still showing (env) # do:

- pip install --upgrade pip
- pip install --upgrade setuptools

This should show:

 . /home/synapse/env/bin/activate
(env) # pip install --upgrade pip
Requirement already up-to-date: pip in /usr/home/synapse/env/lib/python3.7/site-packages (19.1.1)
(env) # pip install --upgrade setuptools
Requirement already up-to-date: setuptools in /usr/home/synapse/env/lib/python3.7/site-packages (41.0.1)

If this is fine, continue with:

  • pip install matrix-synapse[all]

This will take some time, download different things and should finally end without error.

Setup PostgreSQL Database

changed from 9.5 to 11.5 on 20191030

  • leave virtualenv with typing: deactivate
  • you should only see # without env now
  • pkg install postgresql11-server
  • sysrc postgresql_enable=yes
  • service postgresql initdb
  • service postgresql start

now we have a running PostgreSQL database engine.

Setup the Firewall, DNS, NTP and Mail.

Before we continue we should setup our firewall to avoid attacks while runnign our setup.

Create /etc/rc.firewall.KMJ with the following content:

#!/bin/sh -
#
# Setup system for ipfw(4) firewall service.
#

# Suck in the configuration variables.
if [ -z "${source_rc_confs_defined}" ]; then
        if [ -r /etc/defaults/rc.conf ]; then
                . /etc/defaults/rc.conf
                source_rc_confs
        elif [ -r /etc/rc.conf ]; then
                . /etc/rc.conf
        fi
fi

############

setup_loopback() {
        ############
        # Only in rare cases do you want to change these rules
        #
        ${fwcmd} add 100 pass all from any to any via lo0
        ${fwcmd} add 200 deny all from any to 127.0.0.0/8
        ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any
        ${fwcmd} add 400 deny all from any to ::1
        ${fwcmd} add 500 deny all from ::1 to any
}

. /etc/rc.subr
. /etc/network.subr

############
# Set quiet mode if requested
#
case ${firewall_quiet} in
[Yy][Ee][Ss])
        fwcmd="/sbin/ipfw -q"
        ;;
*)
        fwcmd="/sbin/ipfw"
        ;;
esac

############
# Flush out the list before we begin.
#
${fwcmd} -f flush
setup_loopback

############
#
# We do not allow IPv6 Traffic
${fwcmd} add deny  ip6 from any to any

# Allow TCP through if setup succeeded
${fwcmd} add pass tcp from any to any established

# Allow IP fragments to pass through
${fwcmd} add pass all from any to any frag

# Allow setup of incoming connections
         
${fwcmd} add pass tcp from ${MGMTIP} to me 22 setup     # SSH

${fwcmd} add pass tcp from any to me 8448 setup         # MATRIX
${fwcmd} add pass tcp from any to me 3478 setup         # TURN
${fwcmd} add pass udp from any to me 3478 keep-state    # TURN
${fwcmd} add pass tcp from any to me 5349 setup         # TURN TLS
${fwcmd} add pass udp from any to me 5349 keep-state    # TURN TLS

${fwcmd} add pass tcp from any to me 80 setup           # letsencrypt only
${fwcmd} add pass tcp from any to me 443 setup          # https rev proxy fuer matrix

# Allow setup of outgoing TCP connections only
${fwcmd} add pass tcp from me to any setup

# Disallow setup of all other TCP connections
${fwcmd} add deny tcp from any to any setup

# Allow DNS queries out in the world
${fwcmd} add pass udp from me to any 53 keep-state

# Allow NTP queries out in the world
${fwcmd} add pass udp from me to any 123 keep-state

# Deny the Rest
${fwcmd} add deny all from any to any

Assuming you have an IPv4 connectivity only we block all IPv6 traffic. Furthermore it is not recommended to open SSH to the public. I assume you are a good administrator and restrict ssh access to the static IP of your notebook or desktop system. Otherwise you should setup OpenVPN to access your server. Do not use any as MGMTIP. If you open critial services to the public you will regret it later. They will brute force you and one day somebody will break into your system. Be smart and work like a professional!

Now add or change the settings in /etc/rc.conf. We already add settings we need later. You network configuration is not shown here and should not be touched. Do not change

  • hostname=
  • ifconfig_
  • defaultrouter=

parameters.

keymap="de.kbd" # do not change your keymap. we use german
clear_tmp_enable="YES"
syslogd_flags="-ss"
dumpdev="NO"

MGMTIP="5.6.8.9" # change to static IP of your notebook or desktop system
 
firewall_enable="YES"           # Set to YES to enable firewall functionality
firewall_script="/etc/rc.firewall.KMJ" # Which script to run to set up the firewall
firewall_type="KMJ"         # Firewall type (see /etc/rc.firewall)
firewall_quiet="NO"             # Set to YES to suppress rule display
firewall_logging="YES"          # Set to YES to enable events logging
firewall_logif="NO"             # Set to YES to create logging-pseudo interface

#######################
 
local_unbound_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"

apache24_enable="YES"
postgresql_enable="YES"

synapse_enable="YES"
turnserver_enable="YES"

###############################

now change /etc/resolv.conf so it reads:

search ourdomain.com
nameserver 127.0.0.1
options edns0

The final setting requires that our server is able to send out e-mails.

  • cd /etc/mail
  • make
  • make install
  • edit /etc/mail/aliases and uncomment the root: entry.
  • change it to root: you@yourmaildomain.com
  • run newaliases

Using this settings we run a local cache DNS and resolve locally. Furthermore we run a time syncing via NTP daemon. To make sure everything is done OK, we should reboot now.

  • shutdown -r now

Check our Setup

  • login after reboot
  • do a ps -xaj and check that local-ubound, syslogd, ntpd and postgresql deamons are running
  • do a mail -s “test” root, enter some text+ and press CTRL+d to send. You should receive your root mail. Check /var/log/maillog if not,

Setup Turnserver

To handle voice calls behind NAT you should setup Turnserver as kind of proxy. Setup is pretty easy.

  • pkg install turnserver
  • create /usr/local/etc/turnserver.conf like below, but change your IP, domain and secret!
realm=matrix.ourdomain.com
listening-ip=1.2.3.4
#lt-cred-mech
use-auth-secret
static-auth-secret=1cd6c96c1f8d4f1008042d76d646691e3b0512e830d7f486d6ad6b702bfc234e
no-tcp-relay
user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user.
total-quota=1200

cert=/usr/local/etc/matrix-synapse/certs/matrix.ourdomain.com/cert.pem
pkey=/usr/local/etc/matrix-synapse/certs/matrix.ourdomain.com/privkey.pem

do not start turnserver now!

Setup Apache Reversed-Proxy and SSL Cert

We want Apache to handle the connections to https Port 443 from the clients. Federation is done via port 8448. So we need to setup an Apache Reversed-Proxy and get a free SSL Cert from Letsencrypt.

  • pkg install apache24 cronolog
  • we already added apache24_enable=“YES” to /etc/rc.conf earlier
  • save /usr/local/etc/apache24/httpd.conf and replace its content with
  • replace ourdomain.com with your domain
ServerRoot "/usr/local"

ServerSignature Off

Listen 80

LoadModule authn_file_module libexec/apache24/mod_authn_file.so
LoadModule authn_core_module libexec/apache24/mod_authn_core.so
LoadModule authz_host_module libexec/apache24/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache24/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache24/mod_authz_user.so
LoadModule authz_core_module libexec/apache24/mod_authz_core.so
LoadModule access_compat_module libexec/apache24/mod_access_compat.so
LoadModule auth_basic_module libexec/apache24/mod_auth_basic.so
LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so
LoadModule reqtimeout_module libexec/apache24/mod_reqtimeout.so
LoadModule filter_module libexec/apache24/mod_filter.so
LoadModule mime_module libexec/apache24/mod_mime.so
LoadModule log_config_module libexec/apache24/mod_log_config.so
LoadModule env_module libexec/apache24/mod_env.so
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule setenvif_module libexec/apache24/mod_setenvif.so
LoadModule version_module libexec/apache24/mod_version.so
LoadModule ssl_module libexec/apache24/mod_ssl.so
LoadModule mpm_prefork_module libexec/apache24/mod_mpm_prefork.so
LoadModule unixd_module libexec/apache24/mod_unixd.so
LoadModule autoindex_module libexec/apache24/mod_autoindex.so
<IfModule !mpm_prefork_module>
        #LoadModule cgid_module libexec/apache24/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
        #LoadModule cgi_module libexec/apache24/mod_cgi.so
</IfModule>
LoadModule dir_module libexec/apache24/mod_dir.so
LoadModule userdir_module libexec/apache24/mod_userdir.so
LoadModule alias_module libexec/apache24/mod_alias.so
LoadModule rewrite_module libexec/apache24/mod_rewrite.so

LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_connect_module libexec/apache24/mod_proxy_connect.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
IncludeOptional etc/apache24/modules.d/[0-9][0-9][0-9]_*.conf
<IfModule unixd_module>
    User www
    Group www
</IfModule>

ServerAdmin hostmaster@ourdomain.com
ServerName matrix.ourdomain.com:80

DirectoryIndex index.html

<Directory />
    AllowOverride none
    Require all denied
</Directory>

DocumentRoot "/usr/local/www/apache24/data"
<Directory "/usr/local/www/apache24/data">
    Options -Indexes +FollowSymLinks -ExecCGI -MultiViews
    AllowOverride None
    Require all granted
</Directory>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog  "|/usr/local/sbin/cronolog /var/log/www/%Y/%m/%d/errors.log"
LogLevel warn

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i port:%p\"" combined
CustomLog "|/usr/local/sbin/cronolog /var/log/www/%Y/%m/%d/access.log" combined

<IfModule mime_module>
    #
    TypesConfig etc/apache24/mime.types
    #AddType application/x-gzip .tgz
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
</IfModule>

Include etc/apache24/extra/httpd-ssl.conf

SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

Include etc/apache24/Includes/*.conf

then save /usr/local/etc/apache24/extra/httpd-ssl.conf and replace it with the following content. Again replace IP and ourdomain.com with yours.

lISTEN 443

SSLCipherSuite HIGH:MEDIUM:!SSLv3:!kRSA
SSLProxyCipherSuite HIGH:MEDIUM:!SSLv3:!kRSA

SSLHonorCipherOrder on

SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3

SSLPassPhraseDialog  builtin

SSLSessionCache        "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

<VirtualHost _default_:443>

ServerName matrix.ourdomain.com:443
ServerAdmin hostmaster@ourdomain.com

DocumentRoot "/usr/local/www/apache24/data-ssl"
SSLEngine on

SSLCertificateFile /usr/home/letsencrypt/cts/conf-ip1/certs/matrix.ourdomain.com/cert.pem
SSLCertificateKeyFile /usr/home/letsencrypt/cts/conf-ip1/certs/matrix.ourdomain.com/privkey.pem
SSLCertificateChainFile //usr/home/letsencrypt/cts/conf-ip1/certs/matrix.ourdomain.com/fullchain.pem

UserDir disabled
DirectoryIndex index.html index.htm


<Directory /usr/local/www/apache24/data-ssl>
    Options -Indexes -FollowSymLinks -ExecCGI -MultiViews
    AllowOverride None
    require all granted
</Directory>

################################################################################
# Matrix Synapse
################################################################################
# do NOT change 127.0.0.1 to your IP !!!!!!
ProxyPass /_matrix http://127.0.0.1:8008/_matrix
ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix

################################################################################

BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i port:%p\"" combined
CustomLog "|/usr/local/sbin/cronolog /var/log/www/%Y/%m/%d/access.log" combined

</VirtualHost>

Now create the SSL directory:

  • mkdir /usr/local/www/apache24/data-ssl
  • touch /usr/local/www/apache24/data-ssl/index.html
  • echo “Server is offline” > /usr/local/www/apache24/data-ssl/index.html

NOTE

To be able to receive our cert from Letsenrypt, temporary change

  • Include etc/apache24/extra/httpd-ssl.conf

to

  • #Include etc/apache24/extra/httpd-ssl.conf

in httpd.conf. This must be changed back after we received the initial cert.

Now create the Letsencrypt stuff

  • mkdir -p /home/letsencrypt/cts
  • mkdir -p /usr/local/etc/matrix-synapse/certs
  • create /home/letsencrypt/cts/Run_Certs.sh with the following content
#!/bin/sh

cd /home/letsencrypt/cts
chmod 750 dehydrated
/home/letsencrypt/cts/dehydrated  -c  --config /home/letsencrypt/cts/conf-ip1/config.sh \
                                      --hook /home/letsencrypt/cts/conf-ip1/hook.sh

/usr/sbin/service apache24 restart

########################################################################

HOST=`hostname`
echo "copy cert for synapse / matrix server on ${HOST}"

/bin/cp -r /usr/home/letsencrypt/cts/conf-ip1/certs/${HOST} /usr/local/etc/matrix-synapse/certs
/usr/sbin/chown -R root:nobody /usr/local/etc/matrix-synapse/certs
/bin/chmod -R 755 /usr/local/etc/matrix-synapse/certs

/usr/sbin/service synapse restart

/bin/ps -xaj | /usr/bin/grep synapse
########################################################################
  • chown root:wheel /home/letsencrypt/cts/Run_Certs.sh
  • chmod 750 /home/letsencrypt/cts/Run_Certs.sh

  • create /home/letsencrypt/cts/dehydrated with the following commands:

  • cd /home/letsencrypt/cts/

  • wget https://github.com/lukas2511/dehydrated/raw/master/dehydrated

  • chown root:wheel /home/letsencrypt/cts/dehydrated

  • chmod 750 /home/letsencrypt/cts/dehydrated

We now need to create some additional stuff for our Letsencrypt script. Remember to change IP and ourdomain.com to yours!

  • mkdir -p /home/letsencrypt/cts/conf-ip1
  • mkdir -p /usr/home/letsencrypt/cts/www/wellknown/acme-challenge
  • touch /usr/home/letsencrypt/cts/www/wellknown/acme-challenge/index.html
  • ln -s /usr/home/letsencrypt/cts/www/wellknown /usr/local/www/apache24/data/.well-known
  • echo “matrix.ourdomain.com” > /home/letsencrypt/cts/conf-ip1/domains.txt

  • create /home/letsencrypt/cts/conf-ip1/config.sh with the following content:

#!/usr/bin/env /usr/local/bin/bash
# FreeBSD special:
alias openssl='/usr/bin/openssl'
# Path to license agreement (default: https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf)
#LICENSE="https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf"
# Which challenge should be used? Currently http-01 and dns-01 are supported
CHALLENGETYPE="http-01"
#WELLKNOWN="${BASEDIR}/.acme-challenges"
WELLKNOWN="/home/letsencrypt/cts/www/wellknown/acme-challenge"
  • chown root:wheel /home/letsencrypt/cts/conf-ip1/config.sh
  • chmod 750 /home/letsencrypt/cts/conf-ip1/config.sh

  • create /home/letsencrypt/cts/conf-ip1/hook.sh with the following command:

cd /home/letsencrypt/cts/conf-ip1/
wget https://github.com/lukas2511/dehydrated/raw/master/docs/examples/hook.sh
  • chown root:wheel /home/letsencrypt/cts/conf-ip1/hook.sh
  • chmod 750 /home/letsencrypt/cts/conf-ip1/hook.sh

to receive our first cert we should start Apache and accept the terms in advance.

  • service apache24 start

  • cd /home/letsencrypt/cts

  • chmod 750 dehydrated

  • /home/letsencrypt/cts/dehydrated –register –accept-terms –config /home/letsencrypt/cts/conf-ip1/config.sh

  • ./Run_Certs.sh

if all goes well you should have placed your certs in

  • /home/letsencrypt/cts/conf-ip1/certs

and

  • /usr/local/etc/matrix-synapse/certs

Add the following to /etc/crontab:

#######################################################################
# Run letsencrypt renewals every thuesday @ 9:01
#######################################################################

5       9       *       *       2       root    /home/letsencrypt/cts/Run_Certs.sh

Now where we have the correct cert, change back

  • #Include etc/apache24/extra/httpd-ssl.conf

to

  • Include etc/apache24/extra/httpd-ssl.conf

in /usr/local/etc/apache24/httpd.conf and

  • service apache24 restart

Point your browser to https://matrix.ourdomain.com and check if you get a secure connection. Logs are in /var/log/www.

We have done most of our way. Lets finalize everything.

Finish PostgreSQL Setup

su to user postgres and create synape user and database.su - postgres

su - postgres
createdb postgres
createuser --pwprompt synapse_user
# check with
psql
\l
\du
\?
CREATE DATABASE synapse ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C'  template=template0  OWNER synapse_user;
\l
\q
exit

Finish Matrix Synapse setup

Add a synapse user, so we do not need to run the homeserver as root.

  • adduser synapse
adduser synapse
Username: synapse
Full name: synapse
Uid (Leave empty for default): 
Login group [synapse]: 
Login group is synapse. Invite synapse into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh bash rbash nologin) [sh]: nologin
Home directory [/home/synapse]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username   : synapse
Password   : <random>
Full Name  : synapse
Uid        : 1002
Class      : 
Groups     : synapse 
Home       : /nonexistent
Home Mode  : 
Shell      : /usr/sbin/nologin
Locked     : no
OK? (yes/no): yes

Now lets create a base config for our homeserver. Again, do not forget to change ourdomain.com to yours!

- sh
- . /home/synapse/env/bin/activate
- cd /usr/local/etc/matrix-synapse
- python -m synapse.app.homeserver --server-name matrix.ourdomain.com --config-path homeserver.yaml --generate-config --report-stats=no

We need to load a template which is not in the standard setup:

tls_certificate_path: "/usr/local/etc/matrix-synapse/certs/matrix.ourdomain.com/cert.pem"
tls_certificate_path: "/usr/local/etc/matrix-synapse/certs/matrix.ourdomain.com/fullchain.pem"
tls_private_key_path: "/usr/local/etc/matrix-synapse/certs/matrix.ourdomain.com/privkey.pem"

# PEM dh parameters for ephemeral keys
tls_dh_params_path: "/usr/local/etc/matrix-synapse/tls.dh"

# Don't bind to the https port
no_tls: False

tls_fingerprints: []

## Server ##
server_name: "matrix.ourdomain.com"
pid_file: /var/run/matrix-synapse/homeserver.pid
# We don't want the web client
web_client: False

public_baseurl: "https://matrix.ourdomain.com:8448/"

soft_file_limit: 0

listeners:
  -
    port: 8448

    bind_addresses:
      - '1.2.3.4'

    type: http

    tls: true
    x_forwarded: false

    resources:
      -
        names:
          - client     # The client-server APIs, both v1 and v2

        compress: true

      - names: [federation]  # Federation APIs
        compress: false

  - port: 8008
    tls: false
    bind_addresses: ['127.0.0.1']
    type: http

    x_forwarded: false

    resources:
      - names: [client]
        compress: true
      - names: [federation]
        compress: false

# Database configuration
database:
  name: "psycopg2"
  args:
    database: "synapse"
    user: "synapse_user"
    password: "yourpassword"
    cp_min: 5
    cp_max: 10

event_cache_size: "10K"

verbose: 0

#log_file: "/var/log/matrix-synapse/homeserver.log"
log_config: "/usr/local/etc/matrix-synapse/matrix.ourdomain.com.log.config"

rc_messages_per_second: 0.2
rc_message_burst_count: 10.0
federation_rc_window_size: 1000
federation_rc_sleep_limit: 10
federation_rc_sleep_delay: 500
federation_rc_reject_limit: 50
federation_rc_concurrent: 3

media_store_path: "/var/db/matrix-synapse/media_store"
uploads_path: "/var/db/matrix-synapse/uploads"
max_upload_size: "1M"
max_image_pixels: "32M"
dynamic_thumbnails: false

# List of thumbnail to precalculate when an image is uploaded.
thumbnail_sizes:
- width: 32
  height: 32
  method: crop
- width: 96
  height: 96
  method: crop
- width: 320
  height: 240
  method: scale
- width: 640
  height: 480
  method: scale
- width: 800
  height: 600
  method: scale

url_preview_enabled: False

max_spider_size: "1M"
recaptcha_public_key: "DISABLED"
recaptcha_private_key: "DISABLED"
enable_registration_captcha: False

recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"

## Turn ##
## CHANGE AS USED above in turnserver.conf!!!!!!!!!

# The public URIs of the TURN server to give to clients
turn_uris: ["turn:matrix.ourdomain.com:3478?transport=udp", "turn:matrix.ourdomain.com:3478?transport=tcp"]
turn_shared_secret: "1cd6c96c1f8d4f1008042d76d646691e3b0512e830d7f486d6ad6b702bfc234e"
turn_user_lifetime: "1h"
turn_allow_guests: False


## Registration ##
enable_registration: False
registration_shared_secret: "jkdshkdjshdfzv9d87v89df77v897df987v89df7v87df89vu89df7v89df7v7df897vsdfuvhfdjkhvdf89"
bcrypt_rounds: 12
allow_guest_access: False

trusted_third_party_id_servers:
    - matrix.org
    - vector.im


enable_metrics: False
report_stats: False

room_invite_state_types:
    - "m.room.join_rules"
    - "m.room.canonical_alias"
    - "m.room.avatar"
    - "m.room.name"


app_service_config_files: []
macaroon_secret_key: "hfdjkfhdjkshjbncnsuivhsdf8sduv8duv8dfuvufd8uv8fdu89"
expire_access_token: False

## Signing Keys ##

# Path to the signing key to sign messages with
signing_key_path: "/usr/local/etc/matrix-synapse/matrix.ourdomain.com.signing.key"

old_signing_keys: {}
key_refresh_interval: "1d" # 1 Day.

# The trusted servers to download signing keys from.
perspectives:
  servers:
    "matrix.org":
      verify_keys:
        "ed25519:auto":
          key: "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"

# Enable password for login.
password_config:
   enabled: true
   # Uncomment and change to a secret random string for extra security.
   # DO NOT CHANGE THIS AFTER INITIAL SETUP!
   #pepper: ""

email:
   enable_notifs: false
   smtp_host: "localhost"
   smtp_port: 25
   require_transport_security: False
   notif_from: "Ourdomain MATRIX <noreply@ourdomain.com>"
   app_name: Matrix
   template_dir:  /usr/local/etc/matrix-synapse/res/templates
   riot_base_url: "https://matrix.ourdomain.com/riot"


#server_notices:
#   system_mxid_localpart: serveralarm
#   system_mxid_display_name: "Server Infos and Alerts"
#   system_mxid_avatar_url: "https://matrix.ourdomain.com/_matrix/media/v1/thumbnail/matrix.ourdomain.com/fcpNAbnjbfjjHIegqEMoPFew"
#   room_name: "Server Infos and Alerts"

  • mkdir -p /var/run/matrix-synapse
  • chown synapse /var/run/matrix-synapse
  • mkdir -p /var/log/matrix-synapse/
  • chown synapse /var/log/matrix-synapse/
  • change log file in matrix.ourdomain.com.log.config to filename: /var/log/matrix-synapse/homeserver.log
  • mkdir -p /var/db/matrix-synapse/media_store
  • mkdir -p /var/db/matrix-synapse/uploads
  • chown -R synapse:synapse /var/db/matrix-synapse

  • create tls.h file in /usr/local/etc/matrix-synapse

  • cd /usr/local/etc/matrix-synapse/

  • openssl dhparam -out tls.dh 2048

Finally we need a startup script for Matrix Synapse. Lets create /usr/local/etc/rc.d/synapse:

#!/bin/sh
#
# Created by: Karl M. Joch <k.joch@cts.at>
#

# PROVIDE: synapse
# REQUIRE: LOGIN postgresql
# KEYWORD: shutdown

# synapse_enable="YES"

. /etc/rc.subr

#----------------------
# we run in virtual env
#----------------------
VIRTUAL_ENV="/home/synapse/env"
export VIRTUAL_ENV
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# to check run here: env

name=synapse

rcvar=synapse_enable
load_rc_config ${name}

: ${synapse_enable:=NO}
: ${synapse_user:=synapse}
: ${synapse_conf:=/usr/local/etc/matrix-synapse/homeserver.yaml}
: ${synapse_dbdir:=/var/db/matrix-synapse}
: ${synapse_logdir:=/var/log/matrix-synapse}
: ${synapse_pidfile:=/var/run/matrix-synapse/homeserver.pid}

pidfile="${synapse_pidfile}"
procname=/usr/home/synapse/env/bin/python3.7
command=/usr/home/synapse/env/bin/python3.7
command_args="-m synapse.app.homeserver --daemonize -c ${synapse_conf}"
start_precmd=start_precmd

start_precmd()
{
        if [ ! -d ${synapse_pidfile%/*} ] ; then
                install -d -o synapse -g wheel ${synapse_pidfile%/*};
        fi

        if [ ! -d ${synapse_dbdir} ] ; then
                install -d -o synapse -g wheel ${synapse_dbdir};
        fi

        if [ ! -d ${synapse_logdir} ] ; then
                install -d -o synapse -g wheel ${synapse_logdir};
        fi

        if $(grep -q CHANGEME ${synapse_conf}) ; then
                echo "Error: Default secret values in config."
                echo "Please replace the CHANGEME values in ${synapse_conf}"
                exit 1
        fi
}

run_rc_command "$1"
  • chown root:wheel /usr/local/etc/rc.d/synapse
  • chmod 750 /usr/local/etc/rc.d/synapse

Start Matrix Synapse the first time

  • service synapse start

will hopefully end in something like this:

2019-05-11 18:02:13,846 - synapse.server - 222 - INFO - None - Setting up.
2019-05-11 18:02:13,884 - synapse.storage.event_push_actions - 471 - INFO - None - Searching for stream ordering 1 month ago
2019-05-11 18:02:13,885 - synapse.storage.event_push_actions - 477 - INFO - None - Found stream ordering 1 month ago: it's 0
2019-05-11 18:02:13,885 - synapse.storage.event_push_actions - 479 - INFO - None - Searching for stream ordering 1 day ago
2019-05-11 18:02:13,885 - synapse.storage.event_push_actions - 485 - INFO - None - Found stream ordering
2019-05-11 18:02:13,887 - synapse.server - 226 - INFO - None - Finished setting up.

You can check the running synapse with ps -xaj and netstat -an should show you listeners on 8008 and 8448. If yes, you can open the first beer. Your log is in /var/log/matrix-synapse.

Adding the admin user

Again, change ourdomain.com to your domain!!!!!

  • sh
  • . /home/synapse/env/bin/activate
  • cd /usr/local/etc/matrix-synapse
  • register_new_matrix_user -c homeserver.yaml https://matrix.ourdomain.com:8448
  • make this user the admin for you!

  • reboot your server, to restart everything

  • shutdown -r now

  • download the https://riot.im messenger for your platform and login

  • test everything

if everything works as expected:

  • create all other users the same way, but without being admin!

UPDATING

Updating FreeBSD and Packages

  • freebsd-update fetch
  • freebsd-update install
  • pkg update
  • pkg upgrade

Updating Matrix Synapse

  • service synapse stop
  • sh
  • . /home/synapse/env/bin/activate
  • pip install -U matrix-synapse[all]
  • service synapse start
  • if massive update, simply reboot the server

Upgrade notes PostgreSQL 9.5 to 11.5

READ BELOW FOR POSTGRES UPDATE 9.5 to 11.5 if you are below 11.5!!

While upgrading Matrix/Synapse to 1.5.0 a problem with the pkg system occured. The depencies for py37-psycopg2 and turnserver changed to postgresql11-client-11.5 and it ended up with deinstalled postgresql95-server, which was not nice. Thanks to the Proxmox virtualization infrastructure I was able to move back to the snapshot before the update, skipped the pkg upgrade section and put the server online again. So far so good, but now there is need to upgrade to postgresql11-server-11.5 to be on track again.

As written in the documentation (https://www.postgresql.org/docs/11/upgrading.html): “The traditional method for moving data to a new major version is to dump and reload the database, though this can be slow. A faster method is pg_upgrade.” I already had the dump/restore idea because the FreeBSD 11.5 port also changes the location of the servers data.

The steps done to make it work (read 18.6.1 of PostgreSQL docu in advance):

  • deactivate some services on boot in /etc/rc.conf

    apache24_enable=“NO” synapse_enable=“NO” turnserver_enable=“NO”

  • full server shutdown and snapshot creation of Proxmox VM

  • as alternate if bare installed do a full server backup

  • now we need to create a full database dump of the 9.5 server

su
# create temp migration dir
mkdir /var/pgupdate
chmod 777 /var/pgupdate
# change to user pgsql
su - pgsql
cd /var/pgupdate
# backup config changes
cp /usr/local/pgsql/data/pg_hba.conf .
cp /usr/local/pgsql/data/postgresql.conf .
# dump database
pg_dumpall > database_dump_95.sql
# be patient, depending on server this can take time 
exit
# we are root again
service postgresql stop
# rename old data dir
mv /usr/local/pgsql/data /usr/local/pgsql/data-9.5
# delete your postgresql packages / use pkg info first
pkg delete postgresql95-client-9.5.19 postgresql95-server-9.5.19
# this will delete "Installed packages to be REMOVED:"
# postgresql95-client-9.5.19 postgresql95-server-9.5.19
# py36-psycopg2-2.8.3 turnserver-4.5.1.1
# which is ok for us. we reinstall last 2 later
# install server 11 which includes install of client
pkg install postgresql11-server
# initialize the database, run
/usr/local/etc/rc.d/postgresql initdb
# check for changes in pg_hba.conf and postgresql.conf
# best using 
diff -u  /var/pgupdate/pg_hba.conf /var/db/postgres/data11/pg_hba.conf | more
diff -u  /var/pgupdate/postgresql.conf /var/db/postgres/data11/postgresql.conf | more
# reinstall py36-psycopg2 turnserver
pkg install py36-psycopg2 turnserver
# start the postgres server
service postgresql start
# if start runs well, reload the data
su - postgres
psql -d postgres -f /var/pgupdate/database_dump_95.sql
# be patient again!
# if all goes well exit to be root again
exit
# finally upgrade all packages and clean out stuff
pkg update
pkg upgrade
pkg clean -a
# delete old psql data dir and user if all is finished
rm -r /usr/local/pgsql/data
rmuser pgsql
rm -r /var/pgupdate
  • reactivate some services on boot in /etc/rc.conf

    apache24_enable=“YES” synapse_enable=“YES” turnserver_enable=“YES”

# as root start the services and check if all is fine
service apache24 start
service turnserver start
service synapse start

have fun with Riot/Matrix/Synapse again.

NOTE Older versions used user pgsql for PostgreSQL, now user is named postgres and /var/db/postgres is owned in that way!

NOTE from Package The port is set up to use autovacuum for new databases, but you might also want to vacuum and perhaps backup your database regularly. There is a periodic script, /usr/local/etc/periodic/daily/502.pgsql, that you may find useful. You can use it to backup and perform vacuum on all databases nightly. Per default, it performs `vacuum analyze’. See the script for instructions. For autovacuum settings, please review ~pgsql/data/postgresql.conf.

Join my public room

If you like this blog entry feel free to join my public room by entering

/join #kmj:matrix.ctseuro.com

anywhere in the box to send a message! Feedback welcome!

Enterprise Support