Refactor documentation.

In particular, move manpages to the 'doc' directory, and generate HTML
documentation  with `make html`.

1 files changed, 526 insertions, 0 deletions

diff --git a/doc/interimap.1.md b/doc/interimap.1.md
new file mode 100644
index 0000000..387850a
--- /dev/null
+++ b/doc/interimap.1.md
@@ -0,0 +1,526 @@
+% interimap(1)
+% [Guilhem Moulin](mailto:guilhem@fripost.org)
+% July 2015
+
+Name
+====
+
+InterIMAP - Fast bidirectional synchronization for QRESYNC-capable IMAP servers
+
+Synopsis
+========
+
+`interimap` [*OPTION* ...] [*COMMAND*] [*MAILBOX* ...]
+
+Description
+===========
+
+`interimap` performs stateful synchronization between two IMAP4rev1
+servers.
+Such synchronization is made possible by the [`QRESYNC` IMAP
+extension][RFC 7162]; for convenience reasons servers must also support
+the [`LIST-EXTENDED`][RFC 5258], [`LIST-STATUS`][RFC 5819] (or
+[`NOTIFY`][RFC 5465]) and [`UIDPLUS`][RFC 4315] IMAP extensions.
+See also the **[supported extensions](#supported-extensions)** section
+below.
+
+Stateful synchronization is only possible for mailboxes supporting
+persistent message Unique Identifiers (UID) and persistent storage of
+mod-sequences (MODSEQ); any non-compliant mailbox will cause `interimap`
+to abort.
+Furthermore, because UIDs are allocated not by the client but by the
+server, `interimap` needs to keep track of associations between local
+and remote UIDs for each mailbox.
+The synchronization state of a mailbox consists of its `UIDNEXT` and
+`HIGHESTMODSEQ` values on each server; it is then assumed that each
+message with UID smaller than `UIDNEXT` have been replicated to the
+other server, and that the metadata (such as flags) of each message with
+MODSEQ at most `HIGHESTMODSEQ` have been synchronized.
+Conceptually, the synchronization algorithm is derived from [RFC 4549]
+with the [RFC 7162] (sec. 6) amendments, and works as follows:
+
+ 1. `SELECT` (on both servers) a mailbox the current `UIDNEXT` or `HIGHESTMODSEQ`
+    values of which differ from the values found in the database (for
+    either server).  Use the `QRESYNC` `SELECT` parameter from [RFC
+    7162] to list changes (vanished messages and flag updates) since
+    `HIGHESTMODSEQ` to messages with UID smaller than `UIDNEXT`.
+
+ 2. Propagate these changes onto the other server: get the corresponding
+    UIDs from the database, then:
+     a. issue a `UID STORE` command, followed by `UID EXPUNGE`, to
+        remove messages that have not already been deleted on both
+        servers; and
+     b. issue some `UID STORE` commands to propagate flag updates (send
+        a single command for each flag list in order the reduce the
+        number of round trips).
+
+    (Conflicts may occur if the metadata of a message has been updated
+    on both servers with different flag lists; in that case, `interimap`
+    issues a warning and updates the message on each server with the
+    union of both flag lists.)
+    Repeat this step if the server sent some updates in the meantime.
+    Otherwise, update the `HIGHESTMODSEQ` value in the database.
+
+ 3. Process new messages (if the current `UIDNEXT` value of the mailbox
+    differs from the one found in the database) by issuing a `UID FETCH`
+    command; process each received message on-the-fly by issuing an
+    `APPEND` command with the message's `RFC822` body, `FLAGS` and
+    `INTERNALDATE`.
+    Repeat this step if the server received new messages in the
+    meantime.  Otherwise, update the `UIDNEXT` value in the database.
+    Go back to step 2 if the server sent some metadata (such as flag)
+    updates in the meantime.
+
+ 4. Go back to step 1 to proceed with the next unsynchronized mailbox.
+
+Commands
+========
+
+By default, `interimap` synchronizes each mailbox listed by the `LIST ""
+"*"` IMAP command;
+the *list-mailbox*, *list-select-opts* and *ignore-mailbox* options from
+the [configuration file](#configuration-file) can be used to shrink that
+list and save bandwidth.
+However if some extra argument are provided on the command line,
+`interimap` ignores these options and synchronizes the given
+*MAILBOX*es instead.  Note that each *MAILBOX* is taken “as is”; in
+particular, it must be [UTF-7 encoded][RFC 2152], unquoted, and the list
+wildcards ‘\*’ and ‘%’ are passed verbatim to the IMAP server.
+
+If the synchronization was interrupted during a previous run while some
+messages were being replicated (but before the `UIDNEXT` or
+`HIGHESTMODSEQ` values have been updated), `interimap` performs a “full
+synchronization” on theses messages: downloading the whole UID and flag
+lists on each servers allows `interimap` to detect messages that have
+been removed or for which their flags have changed in the meantime.
+Finally, after propagating the offline changes for these messages,
+`interimap` resumes the synchronization for the rest of the mailbox.
+
+Specifying one of the commands below makes `interimap` perform an action
+other than the default [`QRESYNC`][RFC 7162]-based synchronization.
+
+`--repair` [*MAILBOX* ...]
+
+:   List the database anomalies and try to repair them.  (Consider only
+    the given *MAILBOX*es if non-optional arguments are provided.)
+    This is done by performing a so-called “full synchronization”,
+    namely:
+      1/ download all UIDs along with their flag list both from the
+         local and remote servers;
+      2/ ensure that each entry in the database corresponds to an
+         existing UID; and
+      3/ ensure that both flag lists match.
+    Any message found on a server but not in the database is replicated
+    on the other server (which in the worst case, might yield a message
+    duplicate).
+    Flag conflicts are solved by updating each message to the union of
+    both lists.
+
+`--delete` *MAILBOX* [*MAILBOX* ...]
+
+:   Delete the given *MAILBOX*es on each target (by default each server
+    plus the database, unless `--target` specifies otherwise) where it
+    exists.
+    Note that per the [IMAP4rev1 standard][RFC 3501] deletion is not
+    recursive.  Thus *MAILBOX*'s children are not deleted.
+
+`--rename` *SOURCE* *DEST*
+
+:   Rename the mailbox *SOURCE* to *DEST* on each target (by default
+    each server plus the database, unless `--target` specifies
+    otherwise) where it exists.
+    `interimap` aborts if *DEST* already exists on either target.
+    Note that per the [IMAP4rev1 standard][RFC 3501] renaming is
+    recursive.  Thus *SOURCE*'s children are moved to become *DEST*'s
+    children instead.
+
+Options
+=======
+
+`--config=`*FILE*
+
+:   Specify an alternate [configuration file](#configuration-file).
+    Relative paths start from *$XDG_CONFIG_HOME/interimap*, or *~/.config/interimap*
+    if the `XDG_CONFIG_HOME` environment variable is unset.
+
+`--target={local,remote,database}`
+
+:   Limit the scope of a `--delete` or `--rename` command to the given
+    target.  Can be repeated to act on multiple targets.  By default all
+    three targets are considered.
+
+`--watch`[`=`*seconds*]
+
+:   Don't exit after a successful synchronization.  Instead, keep
+    synchronizing forever.  Sleep for the given number of *seconds* (by
+    default 1 minute if `--notify` is unset, and 15 minutes if
+    `--notify` is set) between two synchronizations.  Setting this
+    options enables `SO_KEEPALIVE` on the socket for *type*s other than
+    `tunnel`.
+
+`--notify`
+
+:   Whether to use the [IMAP `NOTIFY` extension][RFC 5465] to instruct
+    the server to automatically send updates to the client.  (Both local
+    and remote servers must support [RFC 5465] for this to work.)
+    This greatly reduces IMAP traffic since `interimap` can rely on
+    server notifications instead of manually polling for updates.
+    If the connection remains idle for 15 minutes (configurable with
+    `--watch`), then `interimap` sends a `NOOP` command to avoid being
+    logged out for inactivity.
+
+`-q`, `--quiet`
+
+:   Try to be quiet.
+
+`--debug`
+
+:   Turn on debug mode.  Debug messages are written to the given *logfile*.
+    Note that this include all IMAP traffic (except literals).
+    Depending on the chosen authentication mechanism, this might include
+    authentication credentials.
+
+`-h`, `--help`
+
+:   Output a brief help and exit.
+
+`--version`
+
+:   Show the version number and exit.
+
+Configuration file
+==================
+
+Unless told otherwise by the `--config=FILE` command-line option,
+`interimap` reads its configuration from *$XDG_CONFIG_HOME/interimap/config*
+(or *~/.config/interimap/config* if the `XDG_CONFIG_HOME` environment
+variable is unset) as an [INI file].
+The syntax of the configuration file is a series of `OPTION=VALUE`
+lines organized under some `[SECTION]`; lines starting with a ‘#’ or
+‘;’ character are ignored as comments.
+The `[local]` and `[remote]` sections define the two IMAP servers to
+synchronize.
+Valid options are:
+
+*database*
+
+:   SQLite version 3 database file to use to keep track of associations
+    between local and remote UIDs, as well as the `UIDVALIDITY`,
+    `UIDNEXT` and `HIGHESTMODSEQ` of each known mailbox on both servers.
+    Relative paths start from *$XDG_DATA_HOME/interimap*, or
+    *~/.local/share/interimap* if the `XDG_DATA_HOME` environment
+    variable is unset.  This option is only available in the default
+    section.
+    (Default: `HOST.db`, where *HOST* is taken from the `[remote]` or
+    `[local]` sections, in that order.)
+
+*list-reference*
+
+:   An optional “reference name” to use for the initial `LIST` command,
+    indicating the context in which the *MAILBOX*es are interpreted.
+    For instance, by specifying `list-reference=perso/` in the `[local]`
+    section, *MAILBOX* names are interpreted relative to `perso/` on the
+    local server; in other words the remote mailbox hierarchy is mapped
+    to the `perso/` sub-hierarchy on the local server.  This is useful
+    for synchronizing multiple remote servers against different
+    namespaces belonging to the same local IMAP server (using a
+    different `interimap` instance for each local namespace ↔ remote
+    synchronization).
+
+    (Note that if the reference name is not a level of mailbox hierarchy
+    and/or does not end with the hierarchy delimiter, by [RFC 3501] its
+    interpretation by the IMAP server is implementation-dependent.)
+
+*list-mailbox*
+
+:   A space separated list of mailbox patterns to use when issuing the
+    initial `LIST` command (overridden by the *MAILBOX*es given as
+    command-line arguments).
+    Names containing special characters such as spaces or brackets need
+    to be enclosed in double quotes.  Within double quotes C-style
+    backslash escape sequences can be used (‘\\t’ for an horizontal tab,
+    ‘\\n’ for a new line, ‘\\\\’ for a backslash, etc.), as well as
+    hexadecimal escape sequences ‘\\xHH’.
+    Furthermore, non-ASCII names must be [UTF-7 encoded][RFC 2152].
+    Two wildcards are available, and passed verbatim to the IMAP server:
+    a ‘\*’ character matches zero or more characters, while a ‘%’
+    character matches zero or more characters up to the hierarchy
+    delimiter.
+    This option is only available in the default section.
+    (The default pattern, `*`, matches all visible mailboxes on the
+    server.)
+
+*list-select-opts*
+
+:   An optional space separated list of selectors for the initial `LIST`
+    command.  (Requires a server supporting the [`LIST-EXTENDED` IMAP
+    extension][RFC 5258].)  Useful values are `SUBSCRIBED` (to list only
+    subscribed mailboxes), `REMOTE` (to also list remote mailboxes on a
+    server supporting mailbox referrals), and `RECURSIVEMATCH` (to
+    list parent mailboxes with children matching one of the above
+    *list-mailbox* patterns).  This option is only available in the
+    default section.
+
+*ignore-mailbox*
+
+:   An optional Perl Compatible Regular Expressions ([PCRE]) covering
+    mailboxes to exclude: any ([UTF-7 encoded][RFC 2152] and unquoted)
+    mailbox listed in the initial `LIST` responses is ignored if it
+    matches the given expression.
+    Note that the *MAILBOX*es given as command-line arguments bypass the
+    check and are always considered for synchronization.  This option is
+    only available in the default section.
+
+*logfile*
+
+:   A file name to use to log debug and informational messages.  (By
+    default these messages are written to the error output.)  This
+    option is only available in the default section.
+
+*type*
+
+:   One of `imap`, `imaps` or `tunnel`.
+    `type=imap` and `type=imaps` are respectively used for IMAP and IMAP
+    over SSL/TLS connections over an INET socket.
+    `type=tunnel` causes `interimap` to create an unnamed pair of
+    connected sockets for interprocess communication with a *command*
+    instead of opening a network socket.
+    Note that specifying `type=tunnel` in the `[remote]` section makes
+    the default *database* to be `localhost.db`.
+    (Default: `imaps`.)
+
+*host*
+
+:   Server hostname, for `type=imap` and `type=imaps`.
+    (Default: `localhost`.)
+
+*port*
+
+:   Server port.
+    (Default: `143` for `type=imap`, `993` for `type=imaps`.)
+
+*proxy*
+
+:   An optional SOCKS proxy to use for TCP connections to the IMAP
+    server (`type=imap` and `type=imaps` only), formatted as
+    `PROTOCOL://[USER:PASSWORD@]PROXYHOST[:PROXYPORT]`.
+    If `PROXYPORT` is omitted, it is assumed at port 1080.
+    Only [SOCKSv5][RFC 1928] is supported (with optional
+    [username/password authentication][RFC 1929]), in two flavors:
+    `socks5://` to resolve *hostname* locally, and `socks5h://` to let
+    the proxy resolve *hostname*.
+
+*command*
+
+:   Command to use for `type=tunnel`.  Must speak the [IMAP4rev1
+    protocol][RFC 3501] on its standard output, and understand it on its
+    standard input.  The value is passed to `` `/bin/sh -c` `` if it
+    contains shell metacharacters; otherwise it is split into words and
+    the resulting list is passed to `execvp`(3).
+
+*STARTTLS*
+
+:   Whether to use the [`STARTTLS`][RFC 2595] directive to upgrade to a
+    secure connection.  Setting this to `YES` for a server not
+    advertising the `STARTTLS` capability causes `interimap` to
+    immediately abort the connection.
+    (Ignored for *type*s other than `imap`.  Default: `YES`.)
+
+*auth*
+
+:   Space-separated list of preferred authentication mechanisms.
+    `interimap` uses the first mechanism in that list that is also
+    advertised (prefixed with `AUTH=`) in the server's capability list.
+    Supported authentication mechanisms are `PLAIN` and `LOGIN`.
+    (Default: `PLAIN LOGIN`.)
+
+*username*, *password*
+
+:   Username and password to authenticate with.  Can be required for non
+    pre-authenticated connections, depending on the chosen
+    authentication mechanism.
+
+*compress*
+
+:   Whether to use the [`IMAP COMPRESS` extension][RFC 4978] for servers
+    advertising it.
+    (Default: `NO` for the `[local]` section, `YES` for the `[remote]`
+    section.)
+
+*null-stderr*
+
+:   Whether to redirect *command*'s standard error to `/dev/null` for
+    `type=tunnel`.  (Default: `NO`.)
+
+*SSL_protocols*
+
+:   A space-separated list of SSL protocols to enable or disable (if
+    prefixed with an exclamation mark `!`.  Known protocols are `SSLv2`,
+    `SSLv3`, `TLSv1`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3`.  Enabling a
+    protocol is a short-hand for disabling all other protocols.
+    (Default: `!SSLv2 !SSLv3 !TLSv1 !TLSv1.1`, i.e., only enable TLSv1.2
+    and above.)
+
+*SSL_cipher_list*
+
+:   The cipher list to send to the server.  Although the server
+    determines which cipher suite is used, it should take the first
+    supported cipher in the list sent by the client.  See
+    [`ciphers`(1ssl)] for more information.
+
+*SSL_fingerprint*
+
+:   Fingerprint of the server certificate's Subject Public Key Info, in
+    the form `[ALGO$]DIGEST_HEX` where `ALGO` is the used algorithm (by
+    default `sha256`).
+    Attempting to connect to a server with a non-matching certificate
+    SPKI fingerprint causes `interimap` to abort the connection during
+    the SSL/TLS handshake.
+    The following command can be used to compute the SHA-256 digest of a
+    certificate's Subject Public Key Info:
+
+        openssl x509 -in /path/to/server/certificate.pem -pubkey \
+        | openssl pkey -pubin -outform DER \
+        | openssl dgst -sha256
+
+*SSL_verify*
+
+:   Whether to verify the server certificate chain.
+    Note that using *SSL_fingerprint* to specify the fingerprint of the
+    server certificate is an orthogonal authentication measure as it
+    ignores the CA chain.
+    (Default: `YES`.)
+
+*SSL_CApath*
+
+:   Directory to use for server certificate verification if
+    `SSL_verify=YES`.
+    This directory must be in “hash format”, see [`verify`(1ssl)] for
+    more information.
+
+*SSL_CAfile*
+
+:   File containing trusted certificates to use during server
+    certificate authentication if `SSL_verify=YES`.
+
+Supported extensions
+====================
+
+`interimap` takes advantage of servers supporting the following
+extensions to the [IMAP4rev1 protocol][RFC 3501] (those marked as
+“recommended” give the most significant performance gain):
+
+ * `LITERAL+` ([RFC 2088], recommended);
+ * `MULTIAPPEND` ([RFC 3502], recommended);
+ * `COMPRESS=DEFLATE` ([RFC 4978], recommended);
+ * `NOTIFY` ([RFC 5465], recommended);
+ * `SASL-IR` ([RFC 4959]); and
+ * `UNSELECT` ([RFC 3691]).
+
+Known bugs and limitations
+==========================
+
+ * Using `interimap` on two identical servers with a non-existent or
+   empty *database* will duplicate each message due to the absence of
+   local ↔ remote UID association.  Hence one needs to manually empty
+   the mail store on one end when migrating to `interimap` from another
+   synchronisation solution.
+
+ * `interimap` is single threaded and doesn't use IMAP command
+   pipelining.  Synchronization could be boosted up by sending
+   independent commands (such as the initial `LIST` and `STATUS`
+   commands) to both servers in parallel, and for a given server, by
+   sending independent commands (such as flag updates) in a pipeline.
+
+ * Because the [IMAP protocol][RFC 3501] doesn't have a specific
+   response code for when a message is moved to another mailbox (either
+   using the `MOVE` command from [RFC 6851], or via `COPY` + `STORE` +
+   `EXPUNGE`), moving a message causes `interimap` to believe that it
+   was deleted while another one (which is replicated again) was added
+   to the other mailbox in the meantime.
+
+ * `PLAIN` and `LOGIN` are the only authentication mechanisms currently
+   supported.
+
+ * `interimap` will probably not work with non [RFC][RFC 3501]-compliant
+   servers.  In particular, no work-around is currently implemented
+   beside the tunables in the [configuration file](#configuration-file).
+   Moreover, few IMAP servers have been tested so far.
+
+Standards
+=========
+
+ * M. Leech, M. Ganis, Y. Lee, R. Kuris, D. Koblas and L. Jones,
+   _SOCKS Protocol Version 5_,
+   [RFC 1928], March 1996.
+ * M. Leech, _Username/Password Authentication for SOCKS V5_,
+   [RFC 1929], March 1996.
+ * J. Myers, _IMAP4 non-synchronizing literals_,
+   [RFC 2088], January 1997.
+ * D. Goldsmith and M. Davis,
+   _A Mail-Safe Transformation Format of Unicode_,
+   [RFC 2152], May 1997.
+ * C. Newman, _Using TLS with IMAP, POP3 and ACAP_,
+   [RFC 2595], June 1999.
+ * M. Crispin, _Internet Message Access Protocol - Version 4rev1_,
+   [RFC 3501], March 2003.
+ * M. Crispin,
+   _Internet Message Access Protocol (IMAP) - `MULTIAPPEND` Extension_,
+   [RFC 3502], March 2003.
+ * A. Melnikov,
+   _Internet Message Access Protocol (IMAP) `UNSELECT` command_,
+   [RFC 3691], February 2004.
+ * M. Crispin,
+   _Internet Message Access Protocol (IMAP) - `UIDPLUS` extension_,
+   [RFC 4315], December 2005.
+ * A. Melnikov,
+   _Synchronization Operations for Disconnected IMAP4 Clients_,
+   [RFC 4549], June 2006.
+ * A. Gulbrandsen, _The IMAP `COMPRESS` Extension_,
+   [RFC 4978], August 2007.
+ * R. Siemborski and A. Gulbrandsen, _IMAP Extension for Simple
+   Authentication and Security Layer (SASL) Initial Client Response_,
+   [RFC 4959], September 2007.
+ * A. Gulbrandsen and A. Melnikov,
+   _The IMAP `ENABLE` Extension_,
+   [RFC 5161], March 2008.
+ * B. Leiba and A. Melnikov,
+   _Internet Message Access Protocol version 4 - `LIST` Command Extensions_,
+   [RFC 5258], June 2008.
+ * A. Gulbrandsen, C. King and A. Melnikov,
+   _The IMAP `NOTIFY` Extension_,
+   [RFC 5465], February 2009.
+ * A. Melnikov and T. Sirainen,
+   _IMAP4 Extension for Returning `STATUS` Information in Extended LIST_,
+   [RFC 5819], March 2010.
+ * A. Gulbrandsen and N. Freed,
+   _Internet Message Access Protocol (IMAP) - `MOVE` Extension_,
+   [RFC 6851], January 2013.
+ * A. Melnikov and D. Cridland,
+   _IMAP Extensions: Quick Flag Changes Resynchronization (`CONDSTORE`)
+   and Quick Mailbox Resynchronization (`QRESYNC`)_,
+   [RFC 7162], May 2014.
+
+[RFC 7162]: https://tools.ietf.org/html/rfc7162
+[RFC 5258]: https://tools.ietf.org/html/rfc5258
+[RFC 5819]: https://tools.ietf.org/html/rfc5819
+[RFC 4315]: https://tools.ietf.org/html/rfc4315
+[RFC 4549]: https://tools.ietf.org/html/rfc4549
+[RFC 2152]: https://tools.ietf.org/html/rfc2152
+[RFC 3501]: https://tools.ietf.org/html/rfc3501
+[RFC 1928]: https://tools.ietf.org/html/rfc1928
+[RFC 1929]: https://tools.ietf.org/html/rfc1929
+[RFC 2595]: https://tools.ietf.org/html/rfc2595
+[RFC 4978]: https://tools.ietf.org/html/rfc4978
+[RFC 2088]: https://tools.ietf.org/html/rfc2088
+[RFC 3502]: https://tools.ietf.org/html/rfc3502
+[RFC 4959]: https://tools.ietf.org/html/rfc4959
+[RFC 3691]: https://tools.ietf.org/html/rfc3691
+[RFC 6851]: https://tools.ietf.org/html/rfc6851
+[RFC 5161]: https://tools.ietf.org/html/rfc5161
+[RFC 5465]: https://tools.ietf.org/html/rfc5465
+
+[INI file]: https://en.wikipedia.org/wiki/INI_file
+[PCRE]: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions
+[`ciphers`(1ssl)]: https://www.openssl.org/docs/manmaster/apps/ciphers.html
+[`verify`(1ssl)]: https://www.openssl.org/docs/manmaster/apps/verify.html