aboutsummaryrefslogtreecommitdiffstats
path: root/doc/benchmark.md
diff options
context:
space:
mode:
authorGuilhem Moulin <guilhem@fripost.org>2019-11-19 20:20:41 +0100
committerGuilhem Moulin <guilhem@fripost.org>2019-11-19 20:22:09 +0100
commit13337de269b207136e2462c3f1f7fbd842522a7c (patch)
tree5953bf9fee2804584b58913066b80edf9d9f2ffb /doc/benchmark.md
parent27535a7849fc937284681d2dc49c49370b15c281 (diff)
Add benchmark metrics, with timings and network + memory usage.
Along with a comparison with OfflineIMAP.
Diffstat (limited to 'doc/benchmark.md')
-rw-r--r--doc/benchmark.md279
1 files changed, 279 insertions, 0 deletions
diff --git a/doc/benchmark.md b/doc/benchmark.md
new file mode 100644
index 0000000..72f51a4
--- /dev/null
+++ b/doc/benchmark.md
@@ -0,0 +1,279 @@
+% InterIMAP benchmark metrics and comparison
+% [Guilhem Moulin](mailto:guilhem@fripost.org)
+
+The [IMAP `QRESYNC` extension][RFC 7162] allows efficient mailbox
+synchronization, in terms of I/O as well as CPU usage. In this document
+we give some benchmark metrics to compare [InterIMAP]'s network usage with
+so-called full synchronization solutions such as [OfflineIMAP]. The
+timings are to be taken with a grain of salt, though: they likely won't
+reflect real-world situations as the emails are stored in RAM for this
+benchmark, and all network access is on the loopback interface. (Moreover
+neither SSL/TLS nor STARTTLS are being used in the below. They would add
+another 2-3 round-trips per connection.)
+
+These metrics show how [InterIMAP] scales linearly with the number of
+*mailboxes* — pretty much regardless of how many messages they contain (at
+least as long as the server can cope with large mailboxes) — while
+[OfflineIMAP] scales with the number of *messages* on active mailboxes.
+
+While [InterIMAP] performs significantly better (especially given that it
+can be relied upon to synchronize flag changes, unlike [OfflineIMAP]'s
+“quick” mode), it should be noted that efficiency comes at the expense of
+flexibility. In particular it's not possible to exclude old messages from
+synchronization (mailboxes can be excluded but finer granularity is not
+possible). And of course not all IMAP servers support [`QRESYNC`][RFC 7162]
+and other extensions [InterIMAP] requires. Furthermore [InterIMAP] is
+single threaded and doesn't use pipelining at the moment. (Concurrency
+opens a can of worms, and given the below metrics it simply doesn't seem
+worth the trouble ☺)
+
+-----------------------------------------------------------------------
+
+The script used to compute these metrics can be found [there][benchmark-script].
+We use [Dovecot] as IMAP server; the “remote” mailbox store is in
+[multi-dbox][dbox] format (initially populated with random messages of average
+size ~4kiB, and randomly pruned to avoid having only contiguous UIDs) while
+[maildir] is used “locally”. The configuration files were not tuned for
+performance (however [InterIMAP] takes advantage of Dovecot's support of the
+[IMAP `COMPRESS` extension][RFC 4978] as it is its default behavior).
+
+The *user* (resp. *system*) column denotes the number of CPU-seconds
+used by the process in user (resp. kernel) mode. The *real* column is
+the elapsed real (wall clock) time. Network measurements are obtained
+by placing packet counters on the interface.
+
+[RFC 4978]: https://tools.ietf.org/html/rfc4978
+[RFC 7162]: https://tools.ietf.org/html/rfc7162
+[InterIMAP]: interimap.1.html
+[OfflineIMAP]: https://www.offlineimap.org/
+[benchmark-script]: https://git.guilhem.org/interimap/plain/benchmark/run
+[Dovecot]: https://dovecot.org
+[dbox]: https://wiki.dovecot.org/MailboxFormat/dbox
+[maildir]: https://wiki.dovecot.org/MailboxFormat/Maildir
+
+-----------------------------------------------------------------------
+
+Single mailbox {#single-mailbox}
+==============
+
+We create a mailbox on the remote server, populate it with a number of
+messages, and synchronize it locally. We then collect metrics for no-op
+synchronization (i.e., of mailboxes that are already in sync), and
+reconciliation after receiving a *single* message on the remote server.
+
+[OfflineIMAP]'s network usage remains low in “quick” mode for large
+mailboxes that are already in sync, but as soon as a mail arrives the
+performance degrades by *several orders of magnitude*. On the other
+hand [InterIMAP] has very little overhead on large mailboxes (also
+memory-wise), and when a message is delivered there is barely more
+traffic than what's required for the transfer of said message.
+
+100 messages
+------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.05s 0.01s 0.07s 85% 21368k 1439B / 1017B 13 / 15
+offlineimap -q 0.04s 0.01s 0.27s 23% 19748k 2497B / 1236B 16 / 20
+offlineimap 0.05s 0.01s 0.32s 22% 19268k 10kiB / 1456B 21 / 23
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.08s 83% 21116k 4516B / 1412B 17 / 19
+offlineimap -q 0.06s 0.00s 0.32s 22% 19968k 15kiB / 1670B 23 / 26
+offlineimap 0.06s 0.00s 0.32s 22% 18616k 14kiB / 1284B 25 / 19
+
+1000 messages
+-------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.05s 0.01s 0.07s 84% 21204k 1449B / 965B 13 / 14
+offlineimap -q 0.06s 0.01s 0.33s 24% 19068k 2664B / 1236B 19 / 20
+offlineimap 0.09s 0.02s 0.37s 30% 19868k 75kiB / 1508B 26 / 24
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.08s 78% 21212k 4524B / 1333B 17 / 16
+offlineimap -q 0.08s 0.03s 0.33s 37% 22284k 80kiB / 1775B 29 / 28
+offlineimap 0.10s 0.01s 0.32s 36% 20116k 80kiB / 1597B 24 / 25
+
+10000 messages
+--------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.09s 75% 20980k 1449B / 965B 13 / 14
+offlineimap -q 0.10s 0.03s 0.37s 37% 36708k 2719B / 1184B 20 / 19
+offlineimap 0.50s 0.09s 0.78s 75% 45424k 746kiB / 2080B 37 / 35
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.12s 54% 21136k 4530B / 1205B 17 / 16
+offlineimap -q 0.51s 0.08s 0.76s 77% 42860k 751kiB / 2608B 43 / 44
+offlineimap 0.62s 0.16s 0.88s 89% 47996k 750kiB / 2222B 38 / 37
+
+100000 messages
+---------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.16s 38% 21080k 1441B / 1017B 13 / 15
+offlineimap -q 1.06s 0.10s 1.40s 83% 201376k 2722B / 1236B 20 / 20
+offlineimap 4.88s 0.83s 5.23s 109% 280716k 7626kiB / 5564B 138 / 102
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.48s 15% 22876k 4532B / 1362B 17 / 19
+offlineimap -q 5.09s 0.75s 5.38s 108% 277336k 7637kiB / 9941B 261 / 185
+offlineimap 4.92s 0.76s 5.22s 108% 279592k 7631kiB / 5603B 144 / 102
+
+-----------------------------------------------------------------------
+
+75 mailboxes {#multi-mailbox}
+============
+
+We create 75 mailboxes on the remote server, populate them with an equal
+number of messages, and synchronize them locally. We then collect
+metrics for no-op synchronization (i.e., of mailboxes that are already
+in sync), and reconciliation after the following changes are being
+applied to the remote server:
+
+ - 3 *new* messages (two on mailbox #2, one on mailbox #3); and
+ - 5 existing messages *EXPUNGEd* (two on mailboxes #3 and #4, one on
+ mailbox #5).
+
+The results are not surprising given the metrics from the [above
+section](#single-mailbox). In “quick” mode [OfflineIMAP] still performs
+reasonably well when the mailboxes are in sync (even though it iterates
+through each mailbox and the extra roundtrips increase network traffic
+compared to the single mailbox case), but performance decrease
+significantly when a message is delivered to a large mailbox. Once
+again [InterIMAP] has very little network overhead regardless of mailbox
+size; it does take longer on very large mailboxes, but the bottleneck is
+the IMAP server ([InterIMAP] is just rolling thumbs waiting for Dovecot
+to compute `STATUS` responses).
+
+100 messages per mailbox
+------------------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.06s 0.00s 0.12s 55% 21712k 1949B / 898B 11 / 13
+offlineimap -q 0.32s 0.08s 0.43s 92% 22400k 36kiB / 7260B 93 / 99
+offlineimap 0.97s 0.32s 1.32s 98% 22648k 606kiB / 19kiB 243 / 251
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.07s 0.00s 0.15s 53% 21860k 10kiB / 1634B 19 / 19
+offlineimap -q 0.34s 0.11s 0.59s 77% 21248k 81kiB / 8697B 109 / 117
+offlineimap 0.93s 0.35s 1.30s 98% 22804k 620kiB / 20kiB 252 / 253
+
+1000 messages per mailbox
+-------------------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.05s 0.01s 0.31s 22% 22028k 1944B / 898B 11 / 13
+offlineimap -q 0.97s 0.22s 1.22s 97% 23920k 36kiB / 7000B 90 / 94
+offlineimap 4.87s 1.54s 5.01s 127% 25040k 5507kiB / 26kiB 393 / 388
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.08s 0.00s 0.29s 28% 22132k 10kiB / 1931B 20 / 19
+offlineimap -q 1.25s 0.32s 1.45s 108% 27276k 344kiB / 9038B 119 / 123
+offlineimap 4.72s 1.70s 5.05s 127% 26464k 5521kiB / 27kiB 399 / 392
+
+10000 messages per mailbox
+--------------------------
+
+### No-op (in sync) ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.07s 0.00s 1.57s 4% 21896k 1942B / 898B 11 / 13
+offlineimap -q 12.10s 3.98s 11.67s 137% 58624k 37kiB / 10kiB 94 / 168
+offlineimap 55.49s 23.68s 51.50s 153% 70652k 54MiB / 57kiB 1072 / 996
+
+### Reconciliation ###
+
+ user system real CPU max RSS traffic (in/out) packets (in/out)
+-------------- ------ ------- ------ ---- -------- ------------------ ------------------
+ interimap 0.08s 0.00s 1.73s 5% 23108k 10kiB / 1624B 20 / 23
+offlineimap -q 14.60s 5.22s 14.00s 141% 64988k 3028kiB / 15kiB 203 / 263
+offlineimap 57.24s 25.92s 53.72s 154% 76560k 54MiB / 89kiB 1981 / 1625
+
+-----------------------------------------------------------------------
+
+Live synchronization {#live-sync}
+====================
+
+97 mailboxes, 500000 messages in total:
+
+ - 2 with 100000 messages;
+ - 10 with 10000 messages;
+ - 20 with 5000 messages;
+ - 45 with 2000 messages; and
+ - 20 with 500 messages.
+
+The two local mail stores (respectively for [InterIMAP] and
+[OfflineIMAP]) are initially in sync with the remote server, and we keep
+long-running “autorefresh” synchronization processes alive for 6h, with
+updates being regularly applied to the remote server: every 5 seconds,
+
+ - a new message is delivered to a random mailbox with 5% probability
+ (once every 100s on average);
+ - a random message is EXPUNGEd with 5% probability (once every 100s on
+ average); and
+ - a random message is marked as seen with 10% probability (once every
+ 50s on average).
+
+`interimap` is configured to sync every *30s*. `offlineimap` is
+configured to quick sync very *30s*, with a regular sync every *1h*.
+
+ user system max RSS traffic (in/out) packets (in/out)
+----------- -------- -------- -------- ------------------ ------------------
+ interimap 12.95s 0.26s 24276k 743kiB / 257kiB 2207 / 4143
+offlineimap 5327.79s 1495.78s 394044k 942MiB / 7840kiB 87k / 126k
+
+Long-lived synchronization for large and busy mail stores is where
+[InterIMAP] truly shines, in terms of CPU as well as network usage.
+(The amount of CPU time spent in kernel mode is so low because the
+process spends most of its time sleeping or in blocking calls waiting
+for the server to compute `STATUS` responses. Smart servers like
+Dovecot should cache states though, hence are able to serve these
+responses quickly.) Thanks to the [`QRESYNC`][RFC 7162]-based
+synchronization there is no need for complex client-side computation,
+nor for sending vast amount of data over the network. (To be fair,
+while the amount of CPU time spent in user mode remains low, the local
+IMAP server might do a bit of extra work which is not counted here. But
+here again caching helps avoid expensive directory traversal.) The
+performance gain is most appreciated for battery-powered devices, as
+well as devices behind slow and/or high-latency network connections ☺.
+Moreover [InterIMAP] *does* synchronize flag updates at every step, while
+[OfflineIMAP] normally skips these in “quick” mode so might *delay* flag
+updates for up to one hour.