CardDAV sync with large addressbooks

[mlangermann]

We have about 4500 entries in one addressbook. When trying to synchronize this to e.g. Thunderbird, Thunderbird make a multiget request with all entries. The respone coming back from EGroupware is this:

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">

and that’s it. No entries are being returned. Shorter addressbooks, like the user accounts, do work correctly.
Egroupware is a docker isntallation with version 23.1.20230620

Maybe there is some size or response time limit that kills the response generation. There are no errors in the logs, only a message from nginx, that the client request is buffered on disk, because of its size.

Does anyone have an idea how to debug or fix this?

[StefanU]

Hi Malte.

What does the log say?:

Access to EGroupware via WebDAV (CalDAV/CardDAV)

---

Actually, 4500 data sets should not be a problem…

What TB-Version?
Native CardDAV implementation?

Stefan

[mlangermann]

The Log only contains those two lines regarding this issue:

2023/07/13 11:15:03 [warn] 22#22: *698 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000009, client: 172.20.0.1, server: _, request: "REPORT /egroupware/groupdav.php/XXXX/addressbook-Alle/ HTTP/1.1", host: "egroupware.XXXX"
2023/07/13 11:15:03 [error] 22#22: *698 FastCGI sent in stderr: "PHP message: ### 4730 resources requested in multiget REPORT --> turning logging off to allow streaming of the response" while reading response header from upstream, client: 172.20.0.1, server: _, request: "REPORT /egroupware/groupdav.php/XXXX/addressbook-Alle/ HTTP/1.1", upstream: "fastcgi://172.20.0.2:9000", host: "egroupware.XXXX"

I tested with TB 102 and 115 with the native CardDAV implementation.

[StefanU]

What does the log say?:

The CardDAV log…

Stefan

[mlangermann]

Where can I get the CardDAV log?

Malte

[StefanU]

See above:

What does the log say?:

Access to EGroupware via WebDAV (CalDAV/CardDAV)

Stefan

[mlangermann]

Sorry for that and thank you very much for the patience

With logging is enabled it works, no matter if I let EGroupware log into a file or the webservers log. As soon as I disable logging it fails again.

Malte

[StefanU]

With logging is enabled it works

Hmmm. Very curious.

@RalfBecker, Do you have any ideas?

Stefan

[RalfBecker]

The new TB addressbook is good news, in terms of CardDAV addressbooks no longer require an extra plugin

Unfortunate they picked the most dump approach to implement that for big addressbooks:

• they could have done a single PROPFIND request with calendar-data like it was done in the old days, and we would stream the megabytes of data in one request
• type could also - like before - do a PROPFIND request to get all URIs and then request them in chunks with a multiget-REPORT, like most clients do, which do not require any advanced CalDAV/CardDAV
• but they do a PROFPIND and then request all of them in a single multiget-REPORT, for which it seems we only enable streaming, when the logging is enabled
• they could also use a more advanced sync-collection REPORT request with sync-tokens and limiting results to chunks of N contacts

I’ll need to either always enable streaming for multiget-REPORTS independent if logging is enabled or not, or enable it for a huge number of entries requested.

Ralf

[mlangermann]

I would say that enabling streaming for a large number of items is a sensible solution.

[RalfBecker]

I commited a fix to master:

* Adressbook/CardDAV: unconditionally start output-buffering to fix problems with huge multiget reports from TB110 AB

Please patch (instructions) it in your container and let me know if it fixes your issue.

Ralf

[mlangermann]

With this patch and disabled logging the synchronisation works.