% InterIMAP benchmark metrics and comparison % [Guilhem Moulin](mailto:guilhem@fripost.org) The [IMAP `QRESYNC` extension][RFC 7162] allows efficient mailbox synchronization, in terms of I/O as well as CPU usage. In this document we give some benchmark metrics to compare [InterIMAP]'s network usage with so-called full synchronization solutions such as [OfflineIMAP]. The timings are to be taken with a grain of salt, though: they likely won't reflect real-world situations as the emails are stored in RAM for this benchmark, and all network access is on the loopback interface. (Moreover neither SSL/TLS nor STARTTLS are being used in the below. They would add another 2-3 round-trips per connection.) These metrics show how [InterIMAP] scales linearly with the number of *mailboxes* — pretty much regardless of how many messages they contain (at least as long as the server can cope with large mailboxes) — while [OfflineIMAP] scales with the number of *messages* on active mailboxes. While [InterIMAP] performs significantly better (especially given that it can be relied upon to synchronize flag changes, unlike [OfflineIMAP]'s “quick” mode), it should be noted that efficiency comes at the expense of flexibility. In particular it's not possible to exclude old messages from synchronization (mailboxes can be excluded but finer granularity is not possible). And of course not all IMAP servers support [`QRESYNC`][RFC 7162] and other extensions [InterIMAP] requires. Furthermore [InterIMAP] is single threaded and doesn't use pipelining at the moment. (Concurrency opens a can of worms, and given the below metrics it simply doesn't seem worth the trouble ☺) ----------------------------------------------------------------------- The script used to compute these metrics can be found [there][benchmark-script]. We use [Dovecot] as IMAP server; the “remote” mailbox store is in [multi-dbox][dbox] format (initially populated with random messages of average size ~4kiB, and randomly pruned to avoid having only contiguous UIDs) while [maildir] is used “locally”. The configuration files were not tuned for performance (however [InterIMAP] takes advantage of Dovecot's support of the [IMAP `COMPRESS` extension][RFC 4978] as it is its default behavior). The *user* (resp. *system*) column denotes the number of CPU-seconds used by the process in user (resp. kernel) mode. The *real* column is the elapsed real (wall clock) time. Network measurements are obtained by placing packet counters on the interface. [RFC 4978]: https://tools.ietf.org/html/rfc4978 [RFC 7162]: https://tools.ietf.org/html/rfc7162 [InterIMAP]: interimap.1.html [OfflineIMAP]: https://www.offlineimap.org/ [benchmark-script]: https://git.guilhem.org/interimap/plain/benchmark/run [Dovecot]: https://dovecot.org [dbox]: https://doc.dovecot.org/admin_manual/mailbox_formats/dbox/ [maildir]: https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/ ----------------------------------------------------------------------- Single mailbox {#single-mailbox} ============== We create a mailbox on the remote server, populate it with a number of messages, and synchronize it locally. We then collect metrics for no-op synchronization (i.e., of mailboxes that are already in sync), and reconciliation after receiving a *single* message on the remote server. [OfflineIMAP]'s network usage remains low in “quick” mode for large mailboxes that are already in sync, but as soon as a mail arrives the performance degrades by *several orders of magnitude*. On the other hand [InterIMAP] has very little overhead on large mailboxes (also memory-wise), and when a message is delivered there is barely more traffic than what's required for the transfer of said message. 100 messages ------------ ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.05s 0.01s 0.07s 85% 21368k 1439B / 1017B 13 / 15 offlineimap -q 0.04s 0.01s 0.27s 23% 19748k 2497B / 1236B 16 / 20 offlineimap 0.05s 0.01s 0.32s 22% 19268k 10kiB / 1456B 21 / 23 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.08s 83% 21116k 4516B / 1412B 17 / 19 offlineimap -q 0.06s 0.00s 0.32s 22% 19968k 15kiB / 1670B 23 / 26 offlineimap 0.06s 0.00s 0.32s 22% 18616k 14kiB / 1284B 25 / 19 1000 messages ------------- ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.05s 0.01s 0.07s 84% 21204k 1449B / 965B 13 / 14 offlineimap -q 0.06s 0.01s 0.33s 24% 19068k 2664B / 1236B 19 / 20 offlineimap 0.09s 0.02s 0.37s 30% 19868k 75kiB / 1508B 26 / 24 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.08s 78% 21212k 4524B / 1333B 17 / 16 offlineimap -q 0.08s 0.03s 0.33s 37% 22284k 80kiB / 1775B 29 / 28 offlineimap 0.10s 0.01s 0.32s 36% 20116k 80kiB / 1597B 24 / 25 10000 messages -------------- ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.09s 75% 20980k 1449B / 965B 13 / 14 offlineimap -q 0.10s 0.03s 0.37s 37% 36708k 2719B / 1184B 20 / 19 offlineimap 0.50s 0.09s 0.78s 75% 45424k 746kiB / 2080B 37 / 35 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.12s 54% 21136k 4530B / 1205B 17 / 16 offlineimap -q 0.51s 0.08s 0.76s 77% 42860k 751kiB / 2608B 43 / 44 offlineimap 0.62s 0.16s 0.88s 89% 47996k 750kiB / 2222B 38 / 37 100000 messages --------------- ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.16s 38% 21080k 1441B / 1017B 13 / 15 offlineimap -q 1.06s 0.10s 1.40s 83% 201376k 2722B / 1236B 20 / 20 offlineimap 4.88s 0.83s 5.23s 109% 280716k 7626kiB / 5564B 138 / 102 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.48s 15% 22876k 4532B / 1362B 17 / 19 offlineimap -q 5.09s 0.75s 5.38s 108% 277336k 7637kiB / 9941B 261 / 185 offlineimap 4.92s 0.76s 5.22s 108% 279592k 7631kiB / 5603B 144 / 102 ----------------------------------------------------------------------- 75 mailboxes {#multi-mailbox} ============ We create 75 mailboxes on the remote server, populate them with an equal number of messages, and synchronize them locally. We then collect metrics for no-op synchronization (i.e., of mailboxes that are already in sync), and reconciliation after the following changes are being applied to the remote server: - 3 *new* messages (two on mailbox #2, one on mailbox #3); and - 5 existing messages *EXPUNGEd* (two on mailboxes #3 and #4, one on mailbox #5). The results are not surprising given the metrics from the [above section](#single-mailbox). In “quick” mode [OfflineIMAP] still performs reasonably well when the mailboxes are in sync (even though it iterates through each mailbox and the extra roundtrips increase network traffic compared to the single mailbox case), but performance decrease significantly when a message is delivered to a large mailbox. Once again [InterIMAP] has very little network overhead regardless of mailbox size; it does take longer on very large mailboxes, but the bottleneck is the IMAP server ([InterIMAP] is just rolling thumbs waiting for Dovecot to compute `STATUS` responses). 100 messages per mailbox ------------------------ ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.06s 0.00s 0.12s 55% 21712k 1949B / 898B 11 / 13 offlineimap -q 0.32s 0.08s 0.43s 92% 22400k 36kiB / 7260B 93 / 99 offlineimap 0.97s 0.32s 1.32s 98% 22648k 606kiB / 19kiB 243 / 251 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.07s 0.00s 0.15s 53% 21860k 10kiB / 1634B 19 / 19 offlineimap -q 0.34s 0.11s 0.59s 77% 21248k 81kiB / 8697B 109 / 117 offlineimap 0.93s 0.35s 1.30s 98% 22804k 620kiB / 20kiB 252 / 253 1000 messages per mailbox ------------------------- ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.05s 0.01s 0.31s 22% 22028k 1944B / 898B 11 / 13 offlineimap -q 0.97s 0.22s 1.22s 97% 23920k 36kiB / 7000B 90 / 94 offlineimap 4.87s 1.54s 5.01s 127% 25040k 5507kiB / 26kiB 393 / 388 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.08s 0.00s 0.29s 28% 22132k 10kiB / 1931B 20 / 19 offlineimap -q 1.25s 0.32s 1.45s 108% 27276k 344kiB / 9038B 119 / 123 offlineimap 4.72s 1.70s 5.05s 127% 26464k 5521kiB / 27kiB 399 / 392 10000 messages per mailbox -------------------------- ### No-op (in sync) ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.07s 0.00s 1.57s 4% 21896k 1942B / 898B 11 / 13 offlineimap -q 12.10s 3.98s 11.67s 137% 58624k 37kiB / 10kiB 94 / 168 offlineimap 55.49s 23.68s 51.50s 153% 70652k 54MiB / 57kiB 1072 / 996 ### Reconciliation ### user system real CPU max RSS traffic (in/out) packets (in/out) -------------- ------ ------- ------ ---- -------- ------------------ ------------------ interimap 0.08s 0.00s 1.73s 5% 23108k 10kiB / 1624B 20 / 23 offlineimap -q 14.60s 5.22s 14.00s 141% 64988k 3028kiB / 15kiB 203 / 263 offlineimap 57.24s 25.92s 53.72s 154% 76560k 54MiB / 89kiB 1981 / 1625 ----------------------------------------------------------------------- Live synchronization {#live-sync} ==================== 97 mailboxes, 500000 messages in total: - 2 with 100000 messages; - 10 with 10000 messages; - 20 with 5000 messages; - 45 with 2000 messages; and - 20 with 500 messages. The two local mail stores (respectively for [InterIMAP] and [OfflineIMAP]) are initially in sync with the remote server, and we keep long-running “autorefresh” synchronization processes alive for 6h, with updates being regularly applied to the remote server: every 5 seconds, - a new message is delivered to a random mailbox with 5% probability (once every 100s on average); - a random message is EXPUNGEd with 5% probability (once every 100s on average); and - a random message is marked as seen with 10% probability (once every 50s on average). `interimap` is configured to sync every *30s*. `offlineimap` is configured to quick sync very *30s*, with a regular sync every *1h*. user system max RSS traffic (in/out) packets (in/out) ----------- -------- -------- -------- ------------------ ------------------ interimap 12.95s 0.26s 24276k 743kiB / 257kiB 2207 / 4143 offlineimap 5327.79s 1495.78s 394044k 942MiB / 7840kiB 87k / 126k Long-lived synchronization for large and busy mail stores is where [InterIMAP] truly shines, in terms of CPU as well as network usage. (The amount of CPU time spent in kernel mode is so low because the process spends most of its time sleeping or in blocking calls waiting for the server to compute `STATUS` responses. Smart servers like Dovecot should cache states though, hence are able to serve these responses quickly.) Thanks to the [`QRESYNC`][RFC 7162]-based synchronization there is no need for complex client-side computation, nor for sending vast amount of data over the network. (To be fair, while the amount of CPU time spent in user mode remains low, the local IMAP server might do a bit of extra work which is not counted here. But here again caching helps avoid expensive directory traversal.) The performance gain is most appreciated for battery-powered devices, as well as devices behind slow and/or high-latency network connections ☺. Moreover [InterIMAP] *does* synchronize flag updates at every step, while [OfflineIMAP] normally skips these in “quick” mode so might *delay* flag updates for up to one hour.