diff options
Diffstat (limited to 'doc/benchmark.md')
| -rw-r--r-- | doc/benchmark.md | 279 |
1 files changed, 279 insertions, 0 deletions
diff --git a/doc/benchmark.md b/doc/benchmark.md new file mode 100644 index 0000000..72f51a4 --- /dev/null +++ b/doc/benchmark.md @@ -0,0 +1,279 @@ +% InterIMAP benchmark metrics and comparison +% [Guilhem Moulin](mailto:guilhem@fripost.org) + +The [IMAP `QRESYNC` extension][RFC 7162] allows efficient mailbox +synchronization, in terms of I/O as well as CPU usage. In this document +we give some benchmark metrics to compare [InterIMAP]'s network usage with +so-called full synchronization solutions such as [OfflineIMAP]. The +timings are to be taken with a grain of salt, though: they likely won't +reflect real-world situations as the emails are stored in RAM for this +benchmark, and all network access is on the loopback interface. (Moreover +neither SSL/TLS nor STARTTLS are being used in the below. They would add +another 2-3 round-trips per connection.) + +These metrics show how [InterIMAP] scales linearly with the number of +*mailboxes* — pretty much regardless of how many messages they contain (at +least as long as the server can cope with large mailboxes) — while +[OfflineIMAP] scales with the number of *messages* on active mailboxes. + +While [InterIMAP] performs significantly better (especially given that it +can be relied upon to synchronize flag changes, unlike [OfflineIMAP]'s +“quick” mode), it should be noted that efficiency comes at the expense of +flexibility. In particular it's not possible to exclude old messages from +synchronization (mailboxes can be excluded but finer granularity is not +possible). And of course not all IMAP servers support [`QRESYNC`][RFC 7162] +and other extensions [InterIMAP] requires. Furthermore [InterIMAP] is +single threaded and doesn't use pipelining at the moment. (Concurrency +opens a can of worms, and given the below metrics it simply doesn't seem +worth the trouble ☺) + +----------------------------------------------------------------------- + +The script used to compute these metrics can be found [there][benchmark-script]. +We use [Dovecot] as IMAP server; the “remote” mailbox store is in +[multi-dbox][dbox] format (initially populated with random messages of average +size ~4kiB, and randomly pruned to avoid having only contiguous UIDs) while +[maildir] is used “locally”. The configuration files were not tuned for +performance (however [InterIMAP] takes advantage of Dovecot's support of the +[IMAP `COMPRESS` extension][RFC 4978] as it is its default behavior). + +The *user* (resp. *system*) column denotes the number of CPU-seconds +used by the process in user (resp. kernel) mode. The *real* column is +the elapsed real (wall clock) time. Network measurements are obtained +by placing packet counters on the interface. + +[RFC 4978]: https://tools.ietf.org/html/rfc4978 +[RFC 7162]: https://tools.ietf.org/html/rfc7162 +[InterIMAP]: interimap.1.html +[OfflineIMAP]: https://www.offlineimap.org/ +[benchmark-script]: https://git.guilhem.org/interimap/plain/benchmark/run +[Dovecot]: https://dovecot.org +[dbox]: https://wiki.dovecot.org/MailboxFormat/dbox +[maildir]: https://wiki.dovecot.org/MailboxFormat/Maildir + +----------------------------------------------------------------------- + +Single mailbox {#single-mailbox} +============== + +We create a mailbox on the remote server, populate it with a number of +messages, and synchronize it locally. We then collect metrics for no-op +synchronization (i.e., of mailboxes that are already in sync), and +reconciliation after receiving a *single* message on the remote server. + +[OfflineIMAP]'s network usage remains low in “quick” mode for large +mailboxes that are already in sync, but as soon as a mail arrives the +performance degrades by *several orders of magnitude*. On the other +hand [InterIMAP] has very little overhead on large mailboxes (also +memory-wise), and when a message is delivered there is barely more +traffic than what's required for the transfer of said message. + +100 messages +------------ + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.05s 0.01s 0.07s 85% 21368k 1439B / 1017B 13 / 15 +offlineimap -q 0.04s 0.01s 0.27s 23% 19748k 2497B / 1236B 16 / 20 +offlineimap 0.05s 0.01s 0.32s 22% 19268k 10kiB / 1456B 21 / 23 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.08s 83% 21116k 4516B / 1412B 17 / 19 +offlineimap -q 0.06s 0.00s 0.32s 22% 19968k 15kiB / 1670B 23 / 26 +offlineimap 0.06s 0.00s 0.32s 22% 18616k 14kiB / 1284B 25 / 19 + +1000 messages +------------- + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.05s 0.01s 0.07s 84% 21204k 1449B / 965B 13 / 14 +offlineimap -q 0.06s 0.01s 0.33s 24% 19068k 2664B / 1236B 19 / 20 +offlineimap 0.09s 0.02s 0.37s 30% 19868k 75kiB / 1508B 26 / 24 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.08s 78% 21212k 4524B / 1333B 17 / 16 +offlineimap -q 0.08s 0.03s 0.33s 37% 22284k 80kiB / 1775B 29 / 28 +offlineimap 0.10s 0.01s 0.32s 36% 20116k 80kiB / 1597B 24 / 25 + +10000 messages +-------------- + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.09s 75% 20980k 1449B / 965B 13 / 14 +offlineimap -q 0.10s 0.03s 0.37s 37% 36708k 2719B / 1184B 20 / 19 +offlineimap 0.50s 0.09s 0.78s 75% 45424k 746kiB / 2080B 37 / 35 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.12s 54% 21136k 4530B / 1205B 17 / 16 +offlineimap -q 0.51s 0.08s 0.76s 77% 42860k 751kiB / 2608B 43 / 44 +offlineimap 0.62s 0.16s 0.88s 89% 47996k 750kiB / 2222B 38 / 37 + +100000 messages +--------------- + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.16s 38% 21080k 1441B / 1017B 13 / 15 +offlineimap -q 1.06s 0.10s 1.40s 83% 201376k 2722B / 1236B 20 / 20 +offlineimap 4.88s 0.83s 5.23s 109% 280716k 7626kiB / 5564B 138 / 102 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.48s 15% 22876k 4532B / 1362B 17 / 19 +offlineimap -q 5.09s 0.75s 5.38s 108% 277336k 7637kiB / 9941B 261 / 185 +offlineimap 4.92s 0.76s 5.22s 108% 279592k 7631kiB / 5603B 144 / 102 + +----------------------------------------------------------------------- + +75 mailboxes {#multi-mailbox} +============ + +We create 75 mailboxes on the remote server, populate them with an equal +number of messages, and synchronize them locally. We then collect +metrics for no-op synchronization (i.e., of mailboxes that are already +in sync), and reconciliation after the following changes are being +applied to the remote server: + + - 3 *new* messages (two on mailbox #2, one on mailbox #3); and + - 5 existing messages *EXPUNGEd* (two on mailboxes #3 and #4, one on + mailbox #5). + +The results are not surprising given the metrics from the [above +section](#single-mailbox). In “quick” mode [OfflineIMAP] still performs +reasonably well when the mailboxes are in sync (even though it iterates +through each mailbox and the extra roundtrips increase network traffic +compared to the single mailbox case), but performance decrease +significantly when a message is delivered to a large mailbox. Once +again [InterIMAP] has very little network overhead regardless of mailbox +size; it does take longer on very large mailboxes, but the bottleneck is +the IMAP server ([InterIMAP] is just rolling thumbs waiting for Dovecot +to compute `STATUS` responses). + +100 messages per mailbox +------------------------ + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.06s 0.00s 0.12s 55% 21712k 1949B / 898B 11 / 13 +offlineimap -q 0.32s 0.08s 0.43s 92% 22400k 36kiB / 7260B 93 / 99 +offlineimap 0.97s 0.32s 1.32s 98% 22648k 606kiB / 19kiB 243 / 251 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.07s 0.00s 0.15s 53% 21860k 10kiB / 1634B 19 / 19 +offlineimap -q 0.34s 0.11s 0.59s 77% 21248k 81kiB / 8697B 109 / 117 +offlineimap 0.93s 0.35s 1.30s 98% 22804k 620kiB / 20kiB 252 / 253 + +1000 messages per mailbox +------------------------- + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.05s 0.01s 0.31s 22% 22028k 1944B / 898B 11 / 13 +offlineimap -q 0.97s 0.22s 1.22s 97% 23920k 36kiB / 7000B 90 / 94 +offlineimap 4.87s 1.54s 5.01s 127% 25040k 5507kiB / 26kiB 393 / 388 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.08s 0.00s 0.29s 28% 22132k 10kiB / 1931B 20 / 19 +offlineimap -q 1.25s 0.32s 1.45s 108% 27276k 344kiB / 9038B 119 / 123 +offlineimap 4.72s 1.70s 5.05s 127% 26464k 5521kiB / 27kiB 399 / 392 + +10000 messages per mailbox +-------------------------- + +### No-op (in sync) ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.07s 0.00s 1.57s 4% 21896k 1942B / 898B 11 / 13 +offlineimap -q 12.10s 3.98s 11.67s 137% 58624k 37kiB / 10kiB 94 / 168 +offlineimap 55.49s 23.68s 51.50s 153% 70652k 54MiB / 57kiB 1072 / 996 + +### Reconciliation ### + + user system real CPU max RSS traffic (in/out) packets (in/out) +-------------- ------ ------- ------ ---- -------- ------------------ ------------------ + interimap 0.08s 0.00s 1.73s 5% 23108k 10kiB / 1624B 20 / 23 +offlineimap -q 14.60s 5.22s 14.00s 141% 64988k 3028kiB / 15kiB 203 / 263 +offlineimap 57.24s 25.92s 53.72s 154% 76560k 54MiB / 89kiB 1981 / 1625 + +----------------------------------------------------------------------- + +Live synchronization {#live-sync} +==================== + +97 mailboxes, 500000 messages in total: + + - 2 with 100000 messages; + - 10 with 10000 messages; + - 20 with 5000 messages; + - 45 with 2000 messages; and + - 20 with 500 messages. + +The two local mail stores (respectively for [InterIMAP] and +[OfflineIMAP]) are initially in sync with the remote server, and we keep +long-running “autorefresh” synchronization processes alive for 6h, with +updates being regularly applied to the remote server: every 5 seconds, + + - a new message is delivered to a random mailbox with 5% probability + (once every 100s on average); + - a random message is EXPUNGEd with 5% probability (once every 100s on + average); and + - a random message is marked as seen with 10% probability (once every + 50s on average). + +`interimap` is configured to sync every *30s*. `offlineimap` is +configured to quick sync very *30s*, with a regular sync every *1h*. + + user system max RSS traffic (in/out) packets (in/out) +----------- -------- -------- -------- ------------------ ------------------ + interimap 12.95s 0.26s 24276k 743kiB / 257kiB 2207 / 4143 +offlineimap 5327.79s 1495.78s 394044k 942MiB / 7840kiB 87k / 126k + +Long-lived synchronization for large and busy mail stores is where +[InterIMAP] truly shines, in terms of CPU as well as network usage. +(The amount of CPU time spent in kernel mode is so low because the +process spends most of its time sleeping or in blocking calls waiting +for the server to compute `STATUS` responses. Smart servers like +Dovecot should cache states though, hence are able to serve these +responses quickly.) Thanks to the [`QRESYNC`][RFC 7162]-based +synchronization there is no need for complex client-side computation, +nor for sending vast amount of data over the network. (To be fair, +while the amount of CPU time spent in user mode remains low, the local +IMAP server might do a bit of extra work which is not counted here. But +here again caching helps avoid expensive directory traversal.) The +performance gain is most appreciated for battery-powered devices, as +well as devices behind slow and/or high-latency network connections ☺. +Moreover [InterIMAP] *does* synchronize flag updates at every step, while +[OfflineIMAP] normally skips these in “quick” mode so might *delay* flag +updates for up to one hour. |