502 Bad Gateway after reboot

Denis_O 2021-07-16 16:04:27 UTC #1

Hello dear community.

I will write in English so that it has the best chance to help other people as well. I am a professional Network/Security Engineer, but I am still taking my first steps into Linux & docker currently. This is my first egroupware install, I got it up& running perfectly yesterday but after a reboot, I get the message “502 Bad Gateway” when trying to access the webGUI.

Being a big supporter of RTFM, so I did my homework and looked at various posts that had similar issues. In particular these ones:

However I am still stuck, and am therefore hoping for your help. Please note that this is NOT a production environment (yet) so this is definitely not the kind of “I messed up our mission-critical production system, I know it’s 17H45 on a Friday afternoon and I don’t have paid support but my boss is looking at me and I need it to be solved before 18:00 sharp, so help me pleeeeeease” request! In fact, I’d like to understanding the reason of what’s happening as much as getting the issue fixed.

This is the environment:
Server (VM)
Operating System: Ubuntu 20.04.2 LTS
Kernel: Linux 5.4.0-77-generic
Architecture: x86-64
Egroupware:
version is 21.1

Both the VM and egroupware were installed from scratch, it is not an upgrade from a previous version (I read in one of the posts mentioned above that this does make a difference)

Troubleshooting steps already performed (according to the posts mentioned above). All these outputs have been gathered after the reboot, when I am experiencing the “Bad gateway” message on te browser. However, as this is a VM and I do have a snapshot from just before the reboot when everything worked, I can gather logs from before if needed.

admin@egw:~$ sudo docker ps -a
CONTAINER ID   IMAGE                                     COMMAND                  CREATED      STATUS         PORTS                      NAMES
32917ae6da71   nginx:stable-alpine                       "/docker-entrypoint.…"   2 days ago   Up 5 hours     127.0.0.1:8080->80/tcp     egroupware-nginx
06bb91fcd480   phpswoole/swoole:4.6-php7.4-alpine        "docker-php-entrypoi…"   2 days ago   Up 5 hours                                egroupware-push
51ea288519f5   containrrr/watchtower:latest              "/watchtower --sched…"   2 days ago   Up 5 hours     8080/tcp                   egroupware-watchtower
bad0b0857ab0   egroupware/egroupware:21.1                "/entrypoint.sh php-…"   2 days ago   Up 9 seconds   9000/tcp                   egroupware
69cadb8902d9   mariadb:10.4                              "docker-entrypoint.s…"   2 days ago   Up 5 hours     3306/tcp                   egroupware-db
24f18aeb4b5f   quay.io/egroupware/collabora-key:stable   "/bin/sh -c 'bash st…"   2 days ago   Up 5 hours     127.0.0.1:9980->9980/tcp   collabora-key

=> all containers seem OK (UP)

admin@egw:/etc/egroupware-docker$ systemctl status docker.service
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─egroupware.conf
     Active: active (running) since Fri 2021-07-16 09:32:39 UTC; 5h 2min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 969 (dockerd)
      Tasks: 46
     Memory: 156.3M
     CGroup: /system.slice/docker.service
             ├─ 969 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
             ├─1330 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 9980 -container-ip 172.18.0.2 -container-port 9980
             └─1358 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 8080 -container-ip 172.19.0.5 -container-port 80



admin@egw:/etc/egroupware-docker$ docker-compose logs -f egoupware
ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

If it’s at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

=> I don’t think this is the output expected. But this somehow contradicts the docker ps -a output, doesn’t it? if the DOCKER_HOST variable should be set manually, in which file do I have to modify it? (I am asking because some files, for instance docker-compose.yml shall not be modified, but docker-compose.override.yml instead).

sudo docker-compose up
[...]
egroupware    | /usr/bin/php7.4 -d memory_limit=-1 /usr/share/egroupware/setup/setup-cli.php --update 'all,admin,xxxxxxxxxxxx'
egroupware    | EGroupware API version 21.1 found.
egroupware    | EGroupware configuration file (header.inc.php) version 1.29 exists and is up to date
egroupware    | An error happened: Call to a member function read() on bool
egroupware    | Installing of EGroupware failed!
[...]

…and here’s where I get lost. The installation of egroupware failed even though we saw earlier that all containers were running… Could you help me investigating from there?

Thanks&cheers
Denis

StefanU 2021-07-16 18:00:57 UTC #2

Hi Denis.

Typo:
docker-compose logs -f egoupware

docker-compose logs -f egroupware

sudo ?

Stefan

Denis_O 2021-07-16 18:49:06 UTC #3

Hi Stefan,

yes of course that was a typo, I got mixed up in my notepad. The correct output of docker-compose logs -f egroupware (as sudo) is basically the following output over and over again in a loop :

egroupware    | /usr/bin/php7.4 -d memory_limit=-1 /usr/share/egroupware/setup/setup-cli.php --update 'all,admin,xxxxxxxx
egroupware    | EGroupware API version 21.1 found.
egroupware    | EGroupware configuration file (header.inc.php) version 1.29 exists and is up to date
egroupware    | An error happened: Call to a member function read() on bool
egroupware    | Retrying EGroupware installation in 3 seconds ...
egroupware    | EGroupware\Api\Cache::get_provider(Instance) no provider found (error instanciating provider EGroupware\Api\Cache\Files: EGroupware\Api\Cache\Files::__construct() can't create basepath !)!EGroupware\Api\Cache::get_provider:700 / EGroupware\Api\Cache::get_provider:664 / EGroupware\Api\Cache::getCache:298 / EGroupware\Api\Cache::getTree:425 / EGroupware\Api\Translation::init:179 / EGroupware\Api\Translation::translate:223 / try_lang:35(An error happened) / _egw_log_exception:66 / egw_exception_handler:100

cheers,
Denis

StefanU 2021-07-17 07:52:52 UTC #4

I have just installed an Ubuntu 20.04.2 and the following EGroupware for testing.
Afterwards I have restarted several times.
I can NOT reproduce your problem.

Please describe in detail how you installed EGroupware.

What is your IP network range?

Stefan

StefanU 2021-07-17 08:20:59 UTC #5

Enough hard disk space?
df

Stefan

NK1 2021-07-17 09:35:15 UTC #6

I have had this ‘Bad Gateway’ result inconsistently when installing the Docker versions on VMs without enough RAM. With 4GB of RAM the Docker install seems reliable. Less than 4GB and it seems the install process can timeout and fail resulting in ‘Bad Gateway’.

Denis_O 2021-07-17 09:35:24 UTC #7

I followed the exact installation step described here (“For a new 21.1 installation”):

…here (Step by step installation - for Community Edition)…

…here for adding the Ubuntu 20.04-specific repo:
https://software.opensuse.org/download.html?project=server%3AeGroupWare&package=egroupware-epl

…and here:

a few things about the server and its environment
the egw server has IPv4 172.16.15.15/24, GW=.1
AD Domain Ctrl (Univention): 172.16.10.10
Mail server: 172.16.15.11

The egw VM has the following config:
4vCPU
6GB RAM
40GB HD (during the install, I used the default setings “use the entire disk”)

admin@egw:/etc/egroupware-docker$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               1.9G     0  1.9G   0% /dev
tmpfs                              394M  1.4M  393M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   20G   11G  8.3G  56% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda2                          976M  107M  803M  12% /boot
/dev/loop0                          56M   56M     0 100% /snap/core18/1944
/dev/loop1                          56M   56M     0 100% /snap/core18/2074
/dev/loop2                          70M   70M     0 100% /snap/lxd/19188
/dev/loop3                          68M   68M     0 100% /snap/lxd/20326
/dev/loop4                          32M   32M     0 100% /snap/snapd/10707
/dev/loop5                          33M   33M     0 100% /snap/snapd/12398
tmpfs                              394M     0  394M   0% /run/user/1000
admin@egw:/etc/egroupware-docker$

Cheers

Denis

Denis_O 2021-07-17 09:38:26 UTC #8

@NK1: yeah that was my first thought too, so I tested with even 8GB RAM but that didn’t make any difference. During the initial VM setup, I had 4GB Ram, now I have added 2 GB for a total of 6GB, but it barely uses 4GB according to the Vcenter Dashboard.

StefanU 2021-07-17 10:45:01 UTC #9

That is more than enough. Very generous…

That should also fit:

And the installation of Ubuntu?
Have you installed anything else besides the basic system with ssh (database, docker, …)?

Stefan

Denis_O 2021-07-18 01:03:47 UTC #10

for the Ubuntu install, I installed the basic server OS and enabled SSH during the install process.
After the OS installation, I followed these instructions:

(section Ubuntu 20.04)

Denis_O 2021-07-18 12:27:31 UTC #11

Quoting:

Just adding my 2 cts here for what it’s worth… I think this is a very, very bad idea to use internal docker networks within the 172.16.0.0/12 range by default, as most enterprise networks will use some addresses or subnets in that range. IMHO it would be much better to use a subnet somewhere in the higher 192.168.0.0/16 range, e.g. 192.168.245.0/24, as these networks are are barely used both in enterprise networks or well designed geek’s home networks. Only basic home users will only use the default 192.168.0.0/24 or 192.168.1.0/24, which are the most common ranges preconfigured in consumer-grade toy-routers. And those people will certainly not have a egw instance running at home! Therefore, in 99,9% of the user cases, docker networks in the higher 192.168.0.0/16 range will never interfere with existing networks.

Cheers

Denis

StefanU 2021-07-18 12:56:34 UTC #12

I understand you. But:
It is not our idea to use this network area. This is Docker standard.

We always use standards whenever possible. That’s why we also described how to change the Docker network.

I have only come across two networks so far where this network range is used. And I have seen many…

But that doesn’t seem to be the problem with you either.
Everyone else can change the network and adapt it to their requirements.

Now I don’t know what to do about your problem either. It could still be that the database is not accessible(?)…

Could you install a new VM with Debian 10 server with EGroupware in parallel and test it (reboot)?
Takes me 8 minutes with Debian 10 template
I had documented this once here:

Stefan

Denis_O 2021-07-18 13:09:28 UTC #13

Thanks for the clarification! It makes perfect sense to leave the default docker network it as it is then.

I am downloading the Debian 10 Image as we speak and will do the installation according to the procedure you provided right after (note that Debian 11 will be released in 2 weeks)

Denis_O 2021-07-18 17:41:52 UTC #14

hum. with the Debian 10VM I get a 404 not found following the 1st reboot. The SSL part did not work at all (I got tons of errors), but http did work after egw install up until I rebooted the system.

docker is running fine:
root@egw02:/home/admax# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
046b54954e89 quay.io/egroupware/collabora-key:stable “/bin/sh -c 'bash st…” 33 minutes ago Up 7 minutes 127.0.0.1:9980->9980/tcp collabora-key
c2d522bc68f9 nginx:stable-alpine “/docker-entrypoint.…” About an hour ago Up 7 minutes 127.0.0.1:8080->80/tcp egroupware-nginx
147d04de0f08 phpswoole/swoole:4.6-php7.4-alpine “docker-php-entrypoi…” About an hour ago Up 7 minutes egroupware-push
4f5c10ebbad2 containrrr/watchtower:latest “/watchtower --sched…” About an hour ago Up 7 minutes 8080/tcp egroupware-watchtower
c598b8024da5 egroupware/egroupware:21.1 “/entrypoint.sh php-…” About an hour ago Up 7 minutes 9000/tcp egroupware
804c92f7aa89 mariadb:10.4 “docker-entrypoint.s…” About an hour ago Up 7 minutes 3306/tcp egroupware-db
root@egw02:/home/admax#

StefanU 2021-07-18 19:08:17 UTC #15

The cause must lie somewhere in your environment. This is definitely not a general problem.

Have you checked that too?:

Do you have applications running on the UCS that are installed as Docker containers? The UCS also uses the standard Docker network (I think).
On the UCS it is also sometimes necessary to change the Docker network.

Are you virtualising (the EGroupware-VM) on your UCS?

Something like this is difficult to solve within the context of community support.
We also offer support for community users:

In your case, this is certainly more effective, quicker and costs less nerves.

Stefan

Denis_O 2021-07-18 19:53:39 UTC #16

thanks a lot for the links. I will go through them tonight. I suspect the issue to be between the screen and the chair!

I will keep the support as a last option, because this is a purely private deployment, and for private use only. In that sense, 240 EUR/hr is quite some money for a private household. But again, as a last option, I will have to go that road, but I first want to rule out everything else.

Denis_O 2021-07-19 15:15:50 UTC #17

update: On the Debian reinstall, I don’t have the issue as long as I don’t configure ssl. With simple http, I can reboot the server and the webGUI is still accessible. I will dig in that direction.

Denis_O 2021-07-19 15:23:37 UTC #18

@Stefan: in your well documented egw install on Debian 10, you specify in the SSL/letsencrype section that the procedure described is not compatible with egw 20.1. I guess that this is also true for 21.1? In that case, that might be the problem on my Debian test install.

StefanU 2021-07-19 16:36:19 UTC #19

Yes.

I created the description when we still had an Apache reverse proxy installed by default. In the meantime, we use nginx in a standard installation. So also in the 21.1.

But you can orientate yourself on documentations from the net (for Lets encrypt-SSL with nginx).

Stefan

StefanU 2021-07-19 16:36:57 UTC #20

How do you install/configure SSL?

Stefan