% interimap(1) % [Guilhem Moulin](mailto:guilhem@fripost.org) % July 2015 Name ==== InterIMAP - Fast bidirectional synchronization for QRESYNC-capable IMAP servers Synopsis ======== `interimap` [*OPTION* ...] [*COMMAND*] [*MAILBOX* ...] Description =========== `interimap` performs stateful synchronization between two IMAP4rev1 servers. Such synchronization is made possible by the [`QRESYNC` IMAP extension][RFC 7162]; for convenience reasons servers must also support the [`LIST-EXTENDED`][RFC 5258], [`LIST-STATUS`][RFC 5819] (or [`NOTIFY`][RFC 5465]) and [`UIDPLUS`][RFC 4315] IMAP extensions. See also the **[supported extensions](#supported-extensions)** section below. Stateful synchronization is only possible for mailboxes supporting persistent message Unique Identifiers (UID) and persistent storage of mod-sequences (MODSEQ); any non-compliant mailbox will cause `interimap` to abort. Furthermore, because UIDs are allocated not by the client but by the server, `interimap` needs to keep track of associations between local and remote UIDs for each mailbox. The synchronization state of a mailbox consists of its `UIDNEXT` and `HIGHESTMODSEQ` values on each server; it is then assumed that each message with UID smaller than `UIDNEXT` have been replicated to the other server, and that the metadata (such as flags) of each message with MODSEQ at most `HIGHESTMODSEQ` have been synchronized. Conceptually, the synchronization algorithm is derived from [RFC 4549] with the [RFC 7162] (sec. 6) amendments, and works as follows: 1. `SELECT` (on both servers) a mailbox the current `UIDNEXT` or `HIGHESTMODSEQ` values of which differ from the values found in the database (for either server). Use the `QRESYNC` `SELECT` parameter from [RFC 7162] to list changes (vanished messages and flag updates) since `HIGHESTMODSEQ` to messages with UID smaller than `UIDNEXT`. 2. Propagate these changes onto the other server: get the corresponding UIDs from the database, then: a. issue a `UID STORE` command, followed by `UID EXPUNGE`, to remove messages that have not already been deleted on both servers; and b. issue some `UID STORE` commands to propagate flag updates (send a single command for each flag list in order the reduce the number of round trips). (Conflicts may occur if the metadata of a message has been updated on both servers with different flag lists; in that case, `interimap` issues a warning and updates the message on each server with the union of both flag lists.) Repeat this step if the server sent some updates in the meantime. Otherwise, update the `HIGHESTMODSEQ` value in the database. 3. Process new messages (if the current `UIDNEXT` value of the mailbox differs from the one found in the database) by issuing a `UID FETCH` command; process each received message on-the-fly by issuing an `APPEND` command with the message's `RFC822` body, `FLAGS` and `INTERNALDATE`. Repeat this step if the server received new messages in the meantime. Otherwise, update the `UIDNEXT` value in the database. Go back to step 2 if the server sent some metadata (such as flag) updates in the meantime. 4. Go back to step 1 to proceed with the next unsynchronized mailbox. Commands ======== By default, `interimap` synchronizes each mailbox listed by the `LIST "" "*"` IMAP command; the *list-mailbox*, *list-select-opts* and *ignore-mailbox* options from the [configuration file](#configuration-file) can be used to shrink that list and save bandwidth. However if some extra argument are provided on the command line, `interimap` ignores these options and synchronizes the given *MAILBOX*es instead. Note that each *MAILBOX* is taken “as is”; in particular, it must be [UTF-7 encoded][RFC 2152], unquoted, and the list wildcards ‘\*’ and ‘%’ are passed verbatim to the IMAP server. If the local and remote hierarchy delimiter differ, then within the *MAILBOX* names the *local* delimiter should be used (it is transparently substituted for remote commands and responses). If the synchronization was interrupted during a previous run while some messages were being replicated (but before the `UIDNEXT` or `HIGHESTMODSEQ` values have been updated), `interimap` performs a “full synchronization” on theses messages: downloading the whole UID and flag lists on each servers allows `interimap` to detect messages that have been removed or for which their flags have changed in the meantime. Finally, after propagating the offline changes for these messages, `interimap` resumes the synchronization for the rest of the mailbox. Specifying one of the commands below makes `interimap` perform an action other than the default [`QRESYNC`][RFC 7162]-based synchronization. `--repair` [*MAILBOX* ...] : List the database anomalies and try to repair them. (Consider only the given *MAILBOX*es if non-optional arguments are provided.) This is done by performing a so-called “full synchronization”, namely: 1/ download all UIDs along with their flag list both from the local and remote servers; 2/ ensure that each entry in the database corresponds to an existing UID; and 3/ ensure that both flag lists match. Any message found on a server but not in the database is replicated on the other server (which in the worst case, might yield a message duplicate). Flag conflicts are solved by updating each message to the union of both lists. `--delete` *MAILBOX* [*MAILBOX* ...] : Delete the given *MAILBOX*es on each target (by default each server plus the database, unless `--target` specifies otherwise) where it exists. Note that per the [IMAP4rev1 standard][RFC 3501] deletion is not recursive. Thus *MAILBOX*'s children are not deleted. `--rename` *SOURCE* *DEST* : Rename the mailbox *SOURCE* to *DEST* on each target (by default each server plus the database, unless `--target` specifies otherwise) where it exists. `interimap` aborts if *DEST* already exists on either target. Note that per the [IMAP4rev1 standard][RFC 3501] renaming is recursive. Thus *SOURCE*'s children are moved to become *DEST*'s children instead. Options ======= `--config=`*FILE* : Specify an alternate [configuration file](#configuration-file). Relative paths start from *$XDG_CONFIG_HOME/interimap*, or *~/.config/interimap* if the `XDG_CONFIG_HOME` environment variable is unset. `--target={local,remote,database}` : Limit the scope of a `--delete` or `--rename` command to the given target. Can be repeated to act on multiple targets. By default all three targets are considered. `--watch`[`=`*seconds*] : Don't exit after a successful synchronization. Instead, keep synchronizing forever. Sleep for the given number of *seconds* (by default 1 minute if `--notify` is unset, and 15 minutes if `--notify` is set) between two synchronizations. Setting this options enables `SO_KEEPALIVE` on the socket for *type*s other than `tunnel`. `--notify` : Whether to use the [IMAP `NOTIFY` extension][RFC 5465] to instruct the server to automatically send updates to the client. (Both local and remote servers must support [RFC 5465] for this to work.) This greatly reduces IMAP traffic since `interimap` can rely on server notifications instead of manually polling for updates. If the connection remains idle for 15 minutes (configurable with `--watch`), then `interimap` sends a `NOOP` command to avoid being logged out for inactivity. `-q`, `--quiet` : Try to be quiet. `--debug` : Turn on debug mode. Debug messages, which includes all IMAP traffic besides literals, are written to the given *logfile*. The `LOGIN` and `AUTHENTICATE` commands are however redacted (in order to avoid disclosing authentication credentials) unless the `--debug` flag is set multiple times. `-h`, `--help` : Output a brief help and exit. `--version` : Show the version number and exit. Configuration file ================== Unless told otherwise by the `--config=FILE` command-line option, `interimap` reads its configuration from *$XDG_CONFIG_HOME/interimap/config* (or *~/.config/interimap/config* if the `XDG_CONFIG_HOME` environment variable is unset) as an [INI file]. The syntax of the configuration file is a series of `OPTION=VALUE` lines organized under some `[SECTION]`; lines starting with a ‘#’ or ‘;’ character are ignored as comments. The `[local]` and `[remote]` sections define the two IMAP servers to synchronize. Valid options are: *database* : SQLite version 3 database file to use to keep track of associations between local and remote UIDs, as well as the `UIDVALIDITY`, `UIDNEXT` and `HIGHESTMODSEQ` of each known mailbox on both servers. Relative paths start from *$XDG_DATA_HOME/interimap*, or *~/.local/share/interimap* if the `XDG_DATA_HOME` environment variable is unset. This option is only available in the default section. (Default: `HOST.db`, where *HOST* is taken from the `[remote]` or `[local]` sections, in that order.) *list-reference* : An optional “reference name” to use for the initial `LIST` command, indicating the context in which the *MAILBOX*es are interpreted. For instance, by specifying `list-reference=perso/` in the `[local]` section, *MAILBOX* names are interpreted relative to `perso/` on the local server; in other words the remote mailbox hierarchy is mapped to the `perso/` sub-hierarchy on the local server. This is useful for synchronizing multiple remote servers against different namespaces belonging to the same local IMAP server (using a different `interimap` instance for each local namespace ↔ remote synchronization). (Note that if the reference name is not a level of mailbox hierarchy and/or does not end with the hierarchy delimiter, by [RFC 3501] its interpretation by the IMAP server is implementation-dependent.) *list-mailbox* : A space separated list of mailbox patterns to use when issuing the initial `LIST` command (overridden by the *MAILBOX*es given as command-line arguments). Names containing special characters such as spaces or brackets need to be enclosed in double quotes. Within double quotes C-style backslash escape sequences can be used (‘\\t’ for an horizontal tab, ‘\\n’ for a new line, ‘\\\\’ for a backslash, etc.), as well as hexadecimal escape sequences ‘\\xHH’. Furthermore, non-ASCII names must be [UTF-7 encoded][RFC 2152]. Two wildcards are available, and passed verbatim to the IMAP server: a ‘\*’ character matches zero or more characters, while a ‘%’ character matches zero or more characters up to the hierarchy delimiter. Hardcoding the hierarchy delimiter in this setting is not advised because the server might silently change it at some point. A null character should be used instead. For instance, if *list-mailbox* is set `"foo\x00bar"` then, assuming the hierarchy delimiter is ‘/’, only the mailbox named `foo/bar` is considered for synchronization. This option is only available in the default section. (The default pattern, `*`, matches all visible mailboxes on the server.) *list-select-opts* : An optional space separated list of selectors for the initial `LIST` command. (Requires a server supporting the [`LIST-EXTENDED` IMAP extension][RFC 5258].) Useful values are `SUBSCRIBED` (to list only subscribed mailboxes), `REMOTE` (to also list remote mailboxes on a server supporting mailbox referrals), and `RECURSIVEMATCH` (to list parent mailboxes with children matching one of the above *list-mailbox* patterns). This option is only available in the default section. *ignore-mailbox* : An optional Perl Compatible Regular Expressions ([PCRE]) covering mailboxes to exclude: any ([UTF-7 encoded][RFC 2152] and unquoted) mailbox listed in the initial `LIST` responses is ignored if it matches the given expression after trimming the reference names and substituting the hierarchy delimiter with the null character. For instance, specifying `^virtual(?:\x00|$)` excludes the mailbox named “virtual” as well as its descendants. Note that the *MAILBOX*es given as command-line arguments bypass the check and are always considered for synchronization. This option is only available in the default section. *logfile* : A file name to use to log debug and informational messages. (By default these messages are written to the error output.) This option is only available in the default section. *log-prefix* : A `printf`(3)-like format string to use as prefix for each log message. Interpreted sequences are `%n` and `%m`, expanding respectively to the component name (*local*/*remote*) and to the name of the mailbox relevant for the log entry. Conditions on a specifier `%X` can be obtained with `%?X?then?` or `%?X?then&else?`, which expands to *then* if the `%X` specifier expands to a non-empty string, and to *else* (or the empty string if there is no else condition) if it doesn't. Literal `%` characters need to be escaped as `%%`, while `&`, `?` and `\` characters need to be `\`-escaped. (Default: `%?n?%?m?%n(%m)&%n?: ?`.) *type* : One of `imap`, `imaps` or `tunnel`. `type=imap` and `type=imaps` are respectively used for IMAP and IMAP over SSL/TLS connections over an INET socket. `type=tunnel` causes `interimap` to create an unnamed pair of connected sockets for interprocess communication with a *command* instead of opening a network socket. Note that specifying `type=tunnel` in the `[remote]` section makes the default *database* to be `localhost.db`. (Default: `imaps`.) *host* : Server hostname, for `type=imap` and `type=imaps`. (Default: `localhost`.) *port* : Server port. (Default: `143` for `type=imap`, `993` for `type=imaps`.) *proxy* : An optional SOCKS proxy to use for TCP connections to the IMAP server (`type=imap` and `type=imaps` only), formatted as `PROTOCOL://[USER:PASSWORD@]PROXYHOST[:PROXYPORT]`. If `PROXYPORT` is omitted, it is assumed at port 1080. Only [SOCKSv5][RFC 1928] is supported (with optional [username/password authentication][RFC 1929]), in two flavors: `socks5://` to resolve *hostname* locally, and `socks5h://` to let the proxy resolve *hostname*. *command* : Command to use for `type=tunnel`. Must speak the [IMAP4rev1 protocol][RFC 3501] on its standard output, and understand it on its standard input. The value is passed to `` `/bin/sh -c` `` if it contains shell metacharacters; otherwise it is split into words and the resulting list is passed to `execvp`(3). *STARTTLS* : Whether to use the [`STARTTLS`][RFC 2595] directive to upgrade to a secure connection. Setting this to `YES` for a server not advertising the `STARTTLS` capability causes `interimap` to immediately abort the connection. (Ignored for *type*s other than `imap`. Default: `YES`.) *auth* : Space-separated list of preferred authentication mechanisms. `interimap` uses the first mechanism in that list that is also advertised (prefixed with `AUTH=`) in the server's capability list. Supported authentication mechanisms are `PLAIN` and `LOGIN`. (Default: `PLAIN LOGIN`.) *username*, *password* : Username and password to authenticate with. Can be required for non pre-authenticated connections, depending on the chosen authentication mechanism. *compress* : Whether to use the [`IMAP COMPRESS` extension][RFC 4978] for servers advertising it. (Default: `NO` for the `[local]` section, `YES` for the `[remote]` section.) *null-stderr* : Whether to redirect *command*'s standard error to `/dev/null` for `type=tunnel`. (Default: `NO`.) *SSL_protocols* : A space-separated list of SSL protocols to enable or disable (if prefixed with an exclamation mark `!`. Known protocols are `SSLv2`, `SSLv3`, `TLSv1`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3`. Enabling a protocol is a short-hand for disabling all other protocols. (Default: `!SSLv2 !SSLv3 !TLSv1 !TLSv1.1`, i.e., only enable TLSv1.2 and above.) *SSL_cipher_list* : The cipher list to send to the server. Although the server determines which cipher suite is used, it should take the first supported cipher in the list sent by the client. See [`ciphers`(1ssl)] for more information. *SSL_fingerprint* : Fingerprint of the server certificate's Subject Public Key Info, in the form `[ALGO$]DIGEST_HEX` where `ALGO` is the used algorithm (by default `sha256`). Attempting to connect to a server with a non-matching certificate SPKI fingerprint causes `interimap` to abort the connection during the SSL/TLS handshake. The following command can be used to compute the SHA-256 digest of a certificate's Subject Public Key Info: openssl x509 -in /path/to/server/certificate.pem -pubkey \ | openssl pkey -pubin -outform DER \ | openssl dgst -sha256 *SSL_verify* : Whether to verify the server certificate chain. Note that using *SSL_fingerprint* to specify the fingerprint of the server certificate is an orthogonal authentication measure as it ignores the CA chain. (Default: `YES`.) *SSL_CApath* : Directory to use for server certificate verification if `SSL_verify=YES`. This directory must be in “hash format”, see [`verify`(1ssl)] for more information. *SSL_CAfile* : File containing trusted certificates to use during server certificate authentication if `SSL_verify=YES`. Supported extensions ==================== `interimap` takes advantage of servers supporting the following extensions to the [IMAP4rev1 protocol][RFC 3501] (those marked as “recommended” give the most significant performance gain): * `LITERAL+` ([RFC 2088], recommended); * `MULTIAPPEND` ([RFC 3502], recommended); * `COMPRESS=DEFLATE` ([RFC 4978], recommended); * `NOTIFY` ([RFC 5465], recommended); * `SASL-IR` ([RFC 4959]); and * `UNSELECT` ([RFC 3691]). Known bugs and limitations ========================== * Using `interimap` on two identical servers with a non-existent or empty *database* will duplicate each message due to the absence of local ↔ remote UID association. (Should they arise, an external tool such as [`doveadm-deduplicate`(1)] can be used to weed them out.) Hence one needs to manually empty the mail store on one end when migrating to `interimap` from another synchronization solution. * `interimap` is single threaded and doesn't use IMAP command pipelining. Synchronization could be boosted up by sending independent commands (such as the initial `LIST` and `STATUS` commands) to both servers in parallel, and for a given server, by sending independent commands (such as flag updates) in a pipeline. * Because the [IMAP protocol][RFC 3501] doesn't have a specific response code for when a message is moved to another mailbox (either using the `MOVE` command from [RFC 6851], or via `COPY` + `STORE` + `EXPUNGE`), moving a message causes `interimap` to believe that it was deleted while another one (which is replicated again) was added to the other mailbox in the meantime. * Because the [IMAP protocol][RFC 3501] doesn't provide a way for clients to determine whether a disappeared mailbox was deleted or renamed, `interimap` aborts when a known mailbox disappeared from one server but not the other. The `--delete` (resp. `rename`) command should be used instead to delete (resp. rename) the mailbox on both servers as well as within `interimap`'s internal database. * `PLAIN` and `LOGIN` are the only authentication mechanisms currently supported. * `interimap` will probably not work with non [RFC][RFC 3501]-compliant servers. In particular, no work-around is currently implemented beside the tunables in the [configuration file](#configuration-file). Moreover, few IMAP servers have been tested so far. Standards ========= * M. Leech, M. Ganis, Y. Lee, R. Kuris, D. Koblas and L. Jones, _SOCKS Protocol Version 5_, [RFC 1928], March 1996. * M. Leech, _Username/Password Authentication for SOCKS V5_, [RFC 1929], March 1996. * J. Myers, _IMAP4 non-synchronizing literals_, [RFC 2088], January 1997. * D. Goldsmith and M. Davis, _A Mail-Safe Transformation Format of Unicode_, [RFC 2152], May 1997. * C. Newman, _Using TLS with IMAP, POP3 and ACAP_, [RFC 2595], June 1999. * M. Crispin, _Internet Message Access Protocol - Version 4rev1_, [RFC 3501], March 2003. * M. Crispin, _Internet Message Access Protocol (IMAP) - `MULTIAPPEND` Extension_, [RFC 3502], March 2003. * A. Melnikov, _Internet Message Access Protocol (IMAP) `UNSELECT` command_, [RFC 3691], February 2004. * M. Crispin, _Internet Message Access Protocol (IMAP) - `UIDPLUS` extension_, [RFC 4315], December 2005. * A. Melnikov, _Synchronization Operations for Disconnected IMAP4 Clients_, [RFC 4549], June 2006. * A. Gulbrandsen, _The IMAP `COMPRESS` Extension_, [RFC 4978], August 2007. * R. Siemborski and A. Gulbrandsen, _IMAP Extension for Simple Authentication and Security Layer (SASL) Initial Client Response_, [RFC 4959], September 2007. * A. Gulbrandsen and A. Melnikov, _The IMAP `ENABLE` Extension_, [RFC 5161], March 2008. * B. Leiba and A. Melnikov, _Internet Message Access Protocol version 4 - `LIST` Command Extensions_, [RFC 5258], June 2008. * A. Gulbrandsen, C. King and A. Melnikov, _The IMAP `NOTIFY` Extension_, [RFC 5465], February 2009. * A. Melnikov and T. Sirainen, _IMAP4 Extension for Returning `STATUS` Information in Extended LIST_, [RFC 5819], March 2010. * A. Gulbrandsen and N. Freed, _Internet Message Access Protocol (IMAP) - `MOVE` Extension_, [RFC 6851], January 2013. * A. Melnikov and D. Cridland, _IMAP Extensions: Quick Flag Changes Resynchronization (`CONDSTORE`) and Quick Mailbox Resynchronization (`QRESYNC`)_, [RFC 7162], May 2014. [RFC 7162]: https://tools.ietf.org/html/rfc7162 [RFC 5258]: https://tools.ietf.org/html/rfc5258 [RFC 5819]: https://tools.ietf.org/html/rfc5819 [RFC 4315]: https://tools.ietf.org/html/rfc4315 [RFC 4549]: https://tools.ietf.org/html/rfc4549 [RFC 2152]: https://tools.ietf.org/html/rfc2152 [RFC 3501]: https://tools.ietf.org/html/rfc3501 [RFC 1928]: https://tools.ietf.org/html/rfc1928 [RFC 1929]: https://tools.ietf.org/html/rfc1929 [RFC 2595]: https://tools.ietf.org/html/rfc2595 [RFC 4978]: https://tools.ietf.org/html/rfc4978 [RFC 2088]: https://tools.ietf.org/html/rfc2088 [RFC 3502]: https://tools.ietf.org/html/rfc3502 [RFC 4959]: https://tools.ietf.org/html/rfc4959 [RFC 3691]: https://tools.ietf.org/html/rfc3691 [RFC 6851]: https://tools.ietf.org/html/rfc6851 [RFC 5161]: https://tools.ietf.org/html/rfc5161 [RFC 5465]: https://tools.ietf.org/html/rfc5465 [INI file]: https://en.wikipedia.org/wiki/INI_file [PCRE]: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions [`ciphers`(1ssl)]: https://www.openssl.org/docs/manmaster/apps/ciphers.html [`verify`(1ssl)]: https://www.openssl.org/docs/manmaster/apps/verify.html [`doveadm-deduplicate`(1)]: https://wiki.dovecot.org/Tools/Doveadm/Deduplicate