Collabora Docker container log file, growing uncontrollably

e_l_e_s 2024-11-29 10:32:25 UTC #1

Hi,
one of my /var/lib/docker/containers/[container_id]/[container_id]-json.log keeps growing.

I’ve done what was recommended in Log rotation Docker in /etc/egroupware-docker/docker-compose.override.yml and latest-docker-compose.override.yml - but the log file keeps growing, by about 16,389KB per minute…

(I did not enable compression btw., the #compress is still commented in the override files)

Is the other way I found on forum Docker log files a good way to limit the log size (like, with the logrotate system service)?

In the said log i see it’s mostly taken by TRC lines of coolwsd… iow Collabora? Upon seeing that, I descended into other /etc/egroupware- directories, like egroupware-collabora, and added the same log-rotating section there - but it did not help)

I’m still hunting, I’ll update when I get somewhere but I’ll appreciate any pointers.

(my Egroupware is 21.1.20230210, I should probably update that… but the problem only manifested itself over last month or so when this log file ate up all my disk space dedicated for this server…)
Kind Regards

EDIT 1: so this logfile contains mostly “TRC Checking pid for jail 1”, followed by “TRC Skipping cleaning jails directory for running coolwsd with pid 1”, ending with “Failed to initialize COOLWSD: Operation not permitted\n”,“stream”:“stderr”,“time” " line and a backtrace.

StefanU 2024-11-29 11:06:57 UTC #2

Hi Lukasz.

That seems COOL. You should check this via the container_id.

Log-Rotate would then be used for COOL in
/etc/egroupware-collabora-key/docker-compose.override.yml
for COOL.

This works for me.

I will here:

I will add a note that for other services such as COOL, this must of course also be entered in the corresponding yml.

In principle, however, the cause of so many log entries should always be eliminated.

Are you up to date with COOL?

Stefan

e_l_e_s 2024-11-29 11:18:07 UTC #3

Hi Stefan,
thank you for your reply,
when I added the said logrotating lines to /etc/egroupware-collabora-key/docker-compose.yml, and ran the docker-compose restart command, it returned a fail :

/etc/egroupware-collabora-key# docker-compose restart
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.collabora-key.logging value Additional properties are not allowed ('max-file', 'max-size' were unexpected)
services.collabora-key.logging.options contains an invalid type, it should be an object

so I had to remove them again.

Well given what my Egroupware auto update stopped on,

and I have not manually ran updates since I probably … am not?

Thank you for replying,
Kind Regards
Lukasz

e_l_e_s 2024-11-29 13:28:19 UTC #4

Btw there is no docker-compose-override.yml in that /etc/egroupware-collabora directory - would that be because of how it is too old?

StefanU 2024-11-29 17:49:53 UTC #5

Yes, that’s right. We haven’t created them automatically yet.

You should be able to simply copy the
docker-compose.yml
easily. Then comment everything out and enter the settings for the log rotate there.

Basically, you should only make your own settings in .override.yml. The docker-compose.yml will otherwise be overwritten at some point by a package update.

Maybe you have a syntax error there. yml configuration files are very sensitive!

This is what it should look like:

...
restart: always
container_name: collabora-key
# set the ip-address of your docker host AND your official DNS name so Collabora
# can access EGroupware without the need to go over your firewall
extra_hosts:
- "egw.xxx.de:172.17.0.1"
logging:
  options:
    max-size: "1m"
    max-file: "5"

  # watchtower is disabled now in favor of the one included in egroupware-docker updating all container
  # see https://containrrr.github.io/watchtower for more information
...

I meant your Collabora Online version

Stefan

e_l_e_s 2024-11-30 12:05:24 UTC #6

I never touched Collabora beyond what Egroupware commanded by itself… I don’t think I have been configuring it to run on this server.

sudo apt list egroupware-collabora-key
Listing... Done
egroupware-collabora-key/unknown,now 22.05.20220815-1 amd64 [installed,automatic]

e_l_e_s 2024-12-02 11:48:16 UTC #7

StefanU:

> This is what it should look like:
> 
> ...
> restart: always
> container_name: collabora-key
> # set the ip-address of your docker host AND your official DNS name so Collabora
> # can access EGroupware without the need to go over your firewall
> extra_hosts:
> - "egw.xxx.de:172.17.0.1"
> logging:
>   options:
>     max-size: "1m"
>     max-file: "5"
> 
>   # watchtower is disabled now in favor of the one included in egroupware-docker updating all container
>   # see https://containrrr.github.io/watchtower for more information
> ...

this is what it looks like now :

# run with: docker-compose -f /etc/egroupware-collabora-key/docker-compose up -d
version: '2'
services:
  collabora-key:
    #image: "quay.io/egroupware/collabora-key:stable"
    #image: "collabora/code:latest"
    #volumes:
    #- /var/lib/egroupware/default/loolwsd:/etc/loolwsd
    # support for Collabora/CODE 21.11+
    #- /var/lib/egroupware/default/loolwsd:/etc/coolwsd
    # dont try to regenerate the (not used certificate) as volume is readonly
    #environment:
    #- DONT_GEN_SSL_CERT=1
    #ports:
    #- "127.0.0.1:9980:9980"
    #restart: always
    #container_name: collabora-key
    # set the ip-address of your docker host AND your official DNS name so Collabora
    # can access EGroupware without the need to go over your firewall
    #extra_hosts:
    #- "my.host.name:172.17.0.1"
    logging:
      options:
        max-size: "1m"
        max-file: "5"

[some watchtower settings below but the docker-compose.yml had it commented out anyway]

[i noticed why the error might ahve been… the file is really whitespace sensitive, i had not enough spaces in front of the options]

[ i also went to the egroupware-docker directory and amended the docker-compose.override and latest-docker-compose.override to the same effect - but the log file from collabora is still growing.]

result of docker-compose restart is success, but df -h still shows the file not deleted. although it has the log timestamp of the restart. the log ends with

{"log":"wsd-00001-00001 2024-12-02 11:46:45.128340 +0000 [ coolwsd ] DBG  New SocketPoll [websrv_poll] owned by 0x7f620effb840| net/Socket.cpp:242\n","stream":"stderr","time":"2024-12-02T11:46:45.128529885Z"}
{"log":"Failed to initialize COOLWSD: Operation not permitted\n","stream":"stderr","time":"2024-12-02T11:46:45.164330839Z"}
{"log":"wsd-00001-00001 2024-12-02 11:46:45.164191 +0000 [ coolwsd ] FTL  Failed to initialize COOLWSD: Operation not permitted| wsd/COOLWSD.hpp:530\n","stream":"stderr","time":"2024-12-02T11:46:45.164350129Z"}
{"log":"Operation not permitted\n","stream":"stderr","time":"2024-12-02T11:46:45.164564541Z"}
{"log":"\u003cshutdown\u003e-00001 2024-12-02 11:46:45.236379 +0000 [ coolwsd ] SIG   Fatal signal received: SIGABRT code: 18446744073709551610 for address: 0x6400000001\n","stream":"stderr","time":"2024-12-02T11:46:45.253659315Z"}
{"log":"Recent activity:\n","stream":"stderr","time":"2024-12-02T11:46:45.253696363Z"}
{"log":"\n","stream":"stderr","time":"2024-12-02T11:46:45.253705021Z"}
{"log":"Backtrace 1 - wsd 24.04.5.4 8679835:\n","stream":"stderr","time":"2024-12-02T11:46:45.260089939Z"}
{"log":"/usr/bin/coolwsd(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x7b3b83]\n","stream":"stderr","time":"2024-12-02T11:46:45.29202075Z"}
{"log":"/usr/bin/coolwsd[0x7b3fa5]\n","stream":"stderr","time":"2024-12-02T11:46:45.29228807Z"}
{"log":"/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f620f077050]\n","stream":"stderr","time":"2024-12-02T11:46:45.292303573Z"}
{"log":"/lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c)[0x7f620f0c5e2c]\n","stream":"stderr","time":"2024-12-02T11:46:45.292423833Z"}
{"log":"/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f620f076fb2]\n","stream":"stderr","time":"2024-12-02T11:46:45.29243973Z"}
{"log":"/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f620f061472]\n","stream":"stderr","time":"2024-12-02T11:46:45.292449786Z"}
{"log":"/usr/bin/coolwsd[0x5ace25]\n","stream":"stderr","time":"2024-12-02T11:46:45.292688321Z"}
{"log":"/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f620f077050]\n","stream":"stderr","time":"2024-12-02T11:46:45.292703028Z"}
{"log":"[0x1d5c0c0]\n","stream":"stderr","time":"2024-12-02T11:46:45.292796014Z"}

hope this helps,
kind regards
Lukasz

StefanU 2024-12-02 18:30:37 UTC #8

You can install Collabora Online on your own. Then you will also get a watchtower to install updates automatically. The configuration in this file is then required/used.

Correct.

I can’t say anything about the content.
It also makes no sense if the entire installation is not up to date.

The log is not deleted immediately with the settings. At a defined size, the file is renamed and a new file is written. Only when max-files are reached is the oldest file deleted.

Do you use Collabora Online at all?

Stefan

e_l_e_s 2024-12-03 10:07:24 UTC #9

Unless it is used in Egroupware for anything internal, I do not.

Hope this helps,

Kind Regards
Lukasz

e_l_e_s 2024-12-05 22:44:11 UTC #10

So i ended up stopping the collabora-key container with appropriate docker command;

Docker ps -a has this about it :

 quay.io/egroupware/collabora-key:stable   "/start-collabora-on…"   5 weeks ago <<< created timestamp

I, again, do not remember installing collabora … yes there were all the EGW announcements about it but I didn’t have the time to set up a trial image to try it out - does this mean the egroupware package pulled it from dependencies and I was supposed to run its setup to completion? (I’m still running debian 10.13 & egroupware-docker from its repositories only…)

RalfBecker 2024-12-06 07:13:18 UTC #11

Yes, the package egroupware-collabora-key is installed as a dependency and it installs /etc/egroupware-collabora-key/docker-compose.yml for the Collabora conainer. That’s also mentioned in our docu incl. how to avoid that, if you really want to …

Ralf