From f1a6866c265bf63efefce68be7f6eae22d477309 Mon Sep 17 00:00:00 2001
From: Gustav Eek <gustav.eek@fripost.org>
Date: Fri, 5 Jul 2019 22:23:01 +0200
Subject: WIP: getting started document

---
 doc/baseline.mdwn      | 219 +++++++++++++++++++++++++++++++++++++++++++++++
 doc/development.md     |  13 +--
 doc/getting-started.md | 228 +++++++++++++++++++++++++++++++++++++++++++++++++
 doc/index.md           |  18 ++--
 doc/template.html      |   2 +-
 5 files changed, 461 insertions(+), 19 deletions(-)
 create mode 100644 doc/baseline.mdwn
 create mode 100644 doc/getting-started.md

(limited to 'doc')

diff --git a/doc/baseline.mdwn b/doc/baseline.mdwn
new file mode 100644
index 0000000..29b121b
--- /dev/null
+++ b/doc/baseline.mdwn
@@ -0,0 +1,219 @@
+---
+title: |
+ Case of setting up InterIMAP with Dovecot and Mutt on Debian
+...
+
+This case guide explains the setup of InterIMAP. The guide also covers
+the overview of configuration philosophy and strategy. Eventually how
+to configure the Mutt client is covered.
+
+# Overview
+
+A local IMAP server has a local mail store. InterIMAP always
+synchronises between two IMAP servers, *local* and *remote*. In this
+case the goal is to sync a remote IMAP server with a local one.
+
+The following software components are involved:
+
+ * [Dovecot](https://dovecot.org)
+ * [Mutt](http://mutt.org)
+ * [InterIMAP](https://git.guilhem.org/interimap/about/)
+
+The environment is assumed to be Debian Stable, Debian 9 Stretch. For
+InterIMAP, however the Debian testing repository needs to be available
+for the purpose of installation.
+
+The mail client (Mutt) is served via IMAP from a local instance of
+Dovecot. Local storage will be Maildir format. Using the IMAP server
+to serve content to the mail client is preferred, compared to using the
+client's internal Maildir support.
+
+The setup corresponds to multiple running instances of Dovecot
+processes: Dovecot is invoked with a separate instance for every
+client session, as well as for the instance of InterIMAP. This is of
+course no problem. All process connect the same mail storage,
+including caching data, etc.
+
+On configuration, the setup is straight forward. Also, all
+configuration and services are user local. no system wide
+configuration changes will be needed. The system wide dovecot service
+is even disabled.
+
+# Install software
+
+Make sure testing is in sources. Install (run as root):
+
+    apt-get install interimap dovecot-imapd
+
+<!--
+# Structure of Maildir locally
+
+*TODO Explain the local storage setup*
+
+-->
+
+What is the "fripost" abstraction?
+
+ - private
+ - work
+
+The use of "private" is for contrasting that to possible completing
+"work" accounts, or similar.  Replace "private" in all of continuation
+with what describes your account. E.g. authors would use "fripost".
+
+```
+ACCOUNT=private
+```
+
+Note on delimiters. Below is referred to "separator". That has to do
+with the server-client IMAP communication and should not be confused
+with the Maildir mail folder delimiter. Maildir hierarchy delimiter
+will always be "." (period).
+
+# Setup Dovecot
+
+Disable the system wide Dovecot service (run as sudo):
+
+```
+systemctl stop dovecot
+systemctl disable dovecot
+```
+
+Provide local configuration for Dovecot:
+
+```
+mkdir -pm 0700 "${XDG_CONFIG_HOME:-$HOME/.config}/dovecot"
+tee "${XDG_CONFIG_HOME:-$HOME/.config}/dovecot/$ACCOUNT.conf" <<-EOF
+namespace inbox {
+    location  = maildir:~/Maildir/$ACCOUNT
+    inbox     = yes
+    list      = yes
+    separator = /
+}
+EOF
+```
+
+Verify connectivity and setup with the following:
+
+    doveadm -c "${XDG_CONFIG_HOME:-$HOME/.config}/dovecot/$ACCOUNT.conf" exec imap
+
+you should see long output pattern `* PREAUTH ... Logged in as
+...`. The PREAUTH shows your authenticated at login. Type the command
+`a list "" "*"` to list your mailboxes. Logout with `b logout`.
+
+Create a wrapper for execution
+
+```
+install -m 0755 /dev/stdin ~/.local/bin/imap <<EOF
+#!/bin/sh
+exec doveadm -c "\${XDG_CONFIG_HOME:-\$HOME/.config}/dovecot/\$1.conf" exec imap
+EOF
+```
+
+# Configure email client (Mutt)
+
+Put in your mutt configuration:
+
+```
+set tunnel = "exec ~/.local/bin/imap private" # Put here what describes your account
+set folder = "imap://foo"
+set spoolfile = "imap://foo"
+```
+
+# Configure InterIMAP
+
+Prepare config.
+
+```
+mkdir -m 0700 -p "${XDG_CONFIG_HOME:-$HOME/.config}/interimap"
+install -m 0600 /usr/share/doc/interimap/interimap.sample "${XDG_CONFIG_HOME:-$HOME/.config}/interimap/$ACCOUNT"
+```
+
+Edit your InterIMAP configuration
+(e.g. *~/.config/interimap/private*):
+
+ * Remove *list-select-opts* option
+ * Edit *local* and *remote* sections
+
+```
+[local]
+type = tunnel
+command = exec ~/.local/bin/imap private
+null-stderr = YES
+
+[remote]
+#type = imaps
+host = imap.example.net
+#port = 993
+#proxy = socks5h://localhost:9050
+username = user
+password = xxxxxxxxxxxxxxxx
+#compress = YES
+```
+
+In accordance with the case above of the authors "imap.example.net"
+should be "imap.fripost.org", "user" e.g. "gustav.eek@fripost.org",
+"xxxxxx..." your password.
+
+# Systemd setup
+
+Here's a documentation bifurcation depending on Debian version: Debian
+10 Buster vs. Debian 11 Bullseye and later.
+
+## Bullseye
+
+The easier case is for *Bullseye*.
+
+Enable the service for your user.
+
+```
+systemctl --user start interimap@$ACCOUNT.service
+systemctl --user enable interimap@$ACCOUNT.service
+```
+
+In need of configuration change override the system wide configuration
+(in */usr/lib/systemd/user/interimap.service*), e.g:
+
+
+```
+mkdir -m 0755 -p "${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/interimap@.service.d"
+install -m 0644 /dev/stdin "${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/interimap@.service.d/override.conf" <<EOF
+[Service]
+ExecStart=
+ExecStart=/usr/bin/interimap --config=%i --watch=20
+EOF
+systemctl --user daemon-reload
+```
+
+This is standard Systemd configuration. See "systemd.unit(5)".
+
+## Buster
+
+For older Debian version, such as *Buster* or *Stretch* setup is more
+tedious.
+
+
+The easier case is for *Bullseye*.
+
+Enable the service for your user.
+
+```
+systemctl --user start interimap@$ACCOUNT.service
+systemctl --user enable interimap@$ACCOUNT.service
+```
+
+In need of configuration change override the system wide configuration
+(in */usr/lib/systemd/user/interimap.service*), e.g:
+
+
+```
+mkdir -m 0755 -p "${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/interimap@.service.d"
+install -m 0644 /dev/stdin "${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/interimap@.service.d/override.conf" <<EOF
+[Service]
+ExecStart=
+ExecStart=/usr/bin/interimap --config=%i --watch=20
+EOF
+systemctl --user daemon-reload
+```
+
+This is standard Systemd configuration. See "systemd.unit(5)".
diff --git a/doc/development.md b/doc/development.md
index 49e8d74..86cd4b0 100644
--- a/doc/development.md
+++ b/doc/development.md
@@ -11,9 +11,6 @@ a new temporary directory:
 
     $ BASEDIR="$(mktemp --tmpdir --directory)"
 
-(The leading `$ ` in this document are command-line prompt strings, and
-are not part of the command themselves.)
-
 
 Dovecot configuration
 =====================
@@ -26,7 +23,7 @@ will be skipped).
 		log_path      = "$BASEDIR/dovecot.log"
 		ssl           = no
 		mail_home     = "$BASEDIR/%u"
-		mail_location = maildir:~/mail
+		mail_location = maildir:~/Mail
 	EOF
 
 Here are some details on the above:
@@ -37,11 +34,6 @@ Here are some details on the above:
     It's annoying to clutter syslog with test entries, so instead we make it
     log to a file under `$BASEDIR`.
 
-`ssl`
-
-  : Not required, but turned off here so dumping the configuration with
-    `` `doveconf -c "$BASEDIR/dovecot.conf" -n` `` doesn't spew a warning.
-
 `mail_home`
 
   : Dovecot needs the name of the user to (pre-)authenticate.  It is shown
@@ -67,7 +59,7 @@ Here are some details on the above:
 `mail_location`
 
   : The user's mail storage resides — in [Maildir](https://wiki.dovecot.org/MailLocation/Maildir)
-    format — in a directory `mail` under their home directory.  This is
+    format — in a directory `Mail` under their home directory.  This is
     enough if you're fine with the default IMAP hierarchy delimiter
     (which depends on the mail format) is used, and if you need a single
     [IMAP namespace](https://tools.ietf.org/html/rfc2342).  For more
@@ -89,6 +81,7 @@ pre-authenticated [IMAP4rev1] in the test environment for username
     * LIST (\HasNoChildren) "." INBOX
     a OK List completed (0.002 + 0.000 + 0.001 secs).
     q LOGOUT
+    * BYE Logging out
     q OK Logout completed (0.001 + 0.000 secs).
 
 For mailbox (create, delete, rename) and message (add, flag update)
diff --git a/doc/getting-started.md b/doc/getting-started.md
new file mode 100644
index 0000000..5b59f92
--- /dev/null
+++ b/doc/getting-started.md
@@ -0,0 +1,228 @@
+% Getting started with InterIMAP
+% [Guilhem Moulin](mailto:guilhem@fripost.org);
+  [Gustav Eek](mailto:gustav.eek@fripost.org)
+
+This describes the setup of InterIMAP for a rather usual user-case, where
+messages on a remote IMAP server `imaps://imap.example.net` are
+synchronized locally.  Changes on either end is replicated on the other
+one.
+
+
+Local IMAP server
+=================
+
+Background and rationale
+------------------------
+
+On a workstation, one's mail storage is often found under `~/Maildir`
+(in [*Maildir* format](https://en.wikipedia.org/wiki/Maildir)) or in
+`/var/mail/$USER` (in [*mbox* format](https://en.wikipedia.org/wiki/Mbox)).
+Local mail clients typically access it directly, and maintain their own
+cache in order to speed up message header listing and searches.
+
+While bidirectional synchronisation software (such as [OfflineIMAP])
+is often able to handle a mail storage in Maildir format, *InterIMAP is
+not*.  Instead, InterIMAP needs an [IMAP4rev1] server on both ends to
+synchronize.  This may sound like a severe limitation at first, but by
+seeing both local and remote mail storages though the same “IMAP lens”,
+InterIMAP is able to take advantage of the abstraction layer and do
+significant optimizations, yielding much faster synchronization.
+(*TODO* link to benchmark)
+*Note*: InterIMAP uses the [Quick Mailbox Resynchronization][RFC 7162]
+extension for stateful synchronization, hence won't work on IMAP servers
+that don't advertize support for that extension.
+
+Installing an [IMAP4rev1] server on a single-user workstation may sound
+overkill, but we argue that Debian GNU/Linux systems generally come with a
+[Message Transfer Agent][MTA] installed, and not only on servers.  Just
+like one may use `/usr/sbin/sendmail` (or a compatible interface) in
+order to send mail out, we propose to use an `imap` binary to access
+them.
+
+In order to fully take advantage of the abstraction layer and
+InterIMAP's optimizations, one should *always* access the mail storage
+through the local [IMAP4rev1] server and *never directly*.  Otherwise
+the IMAP server will keep invalidating its cache when it notices
+inconsistencies, yielding a performance hit.  (*Or worse*: very likely
+many [IMAP4rev1] servers are not able to gracefuly deal with cache
+inconsistencies.)  As far as the mail client is concerned, the cost of
+abstraction seems to be negligible (*TODO* link to benchmark).
+Furthermore, we think that approach is in line with the [Unix
+philosophy]: the mail client only takes care of the rendering part,
+leaving the rest to the IMAP server (searches, sorting/threading, as
+well as storage and caching logic).
+
+Installation
+------------
+
+While this document focuses on [Dovecot](https://dovecot.org), a popular
+[IMAP4rev1] server, any other [`QRESYNC`][RFC 7162]-capable server
+should work equally well.
+
+On Debian GNU/Linux systems, run the following command (as root):
+
+    $ apt install dovecot-imapd
+
+(The leading `$ ` in this document are command-line prompt strings, and
+are not part of the command themselves.)
+
+Configuration
+-------------
+
+Our [`interimap`(1)] instance will use the `imap` command from Dovecot's
+`libexec_dir` in order to access the local mail storage.  We assume that
+the mail client can access it in the same fashion.  In other words, that
+it can spawn a command and use its standard input (resp. output) for
+[IMAP4rev1] commands (resp. responses).  An example of such mail client
+is [Mutt], for which we propose a configuration snippet [below][todo
+link].
+
+Since we don't need the Dovecot services nor master process here, we
+simply disable it.  Run the following as root:
+
+    $ systemctl mask --now dovecot.socket dovecot.service
+
+*Note*: If your mail client doesn't have this ability, and instead insists on
+connecting to a `INET` socket, then you may have to use `socat`(1) or
+similar to spawn the command; or alternatively, keep the Dovecot master
+process but edit the configuration so the IMAP daemon listens on the
+loopback interface.  This is however out of scope for the present
+document.  Consult the [Dovecot wiki](https://wiki.dovecot.org/) for
+details.
+
+We store the local Dovecot configuration under `$XDG_CONFIG_HOME/dovecot`:
+
+    $ mkdir -pvm 0700 ${XDG_CONFIG_HOME:-~/.config}/dovecot
+<!-- -->
+    $ cat >${XDG_CONFIG_HOME:-~/.config}/dovecot/dovecot.conf <<-EOF
+		ssl = no
+		mail_location = maildir:~/Mail
+		namespace {
+		    inbox     = yes
+		    list      = yes
+		    separator = /
+		}
+	EOF
+
+Some remarks on the above:
+
+  * SSL/TLS is explicitely turned off so dumping the configuration with
+    `` `doveconf -c ${XDG_CONFIG_HOME:-~/.config}/dovecot/dovecot.conf -n` ``
+    doesn't spew a warning.
+  * Messages will be stored in Maildir format under `~/Mail`.  Ensure
+    the directory is either *empty* or *doesn't exist* before
+    continuing!  You may want to choose a different [format](https://wiki.dovecot.org/MailboxFormat)
+    here, or simply append `:LAYOUT=fs` to the `mail_location` value in
+    order to use a nicer (File System like) Maildir layout.
+  * The `separator` setting defines the IMAP hierarchy delimiter.  This
+    is orthogonal to the Maildir layout delimiter, and you can safely
+    change it later (on an existing mail store).  Popular hierarchy
+    delimiters include `/` (slash) and `.` (period).
+
+Let's now test the configuration by starting a pre-authenticated
+[IMAP4rev1] session and issuing two command, first `` `LIST "" "*"` ``
+to recursively list all mailboxes (along with their hierarchy
+delimiter), then `` `LOGOUT` `` to… log out and exit.
+
+    $ doveadm -c ${XDG_CONFIG_HOME:-~/.config}/dovecot/dovecot.conf exec imap
+    * PREAUTH [CAPABILITY IMAP4rev1 …] Logged in as myuser
+    a LIST "" "*"
+    * LIST (\HasNoChildren) "/" INBOX
+    a OK List completed (0.001 + 0.000 secs).
+    q LOGOUT
+    * BYE Logging out
+    q OK Logout completed (0.001 + 0.000 secs).
+
+Create a wrapper under `~/.local/bin` in order to avoid hard-coding the
+Dovecot configuration path:
+
+    $ install -Dm 0755 /dev/stdin ~/.local/bin/dovecot-imap <<-EOF
+		#!/bin/sh
+		set -ue
+		export PATH="/usr/bin:/bin"
+		exec env -i PATH="\$PATH" HOME="\$HOME" USER="\$USER" \\
+		    doveadm -c "\${XDG_CONFIG_HOME:-\$HOME/.config}/dovecot/dovecot.conf" \\
+		        exec imap
+	EOF
+
+You can now start a pre-authenticated [IMAP4rev1] session to your local
+mail store by simply running `` `~/.local/bin/dovecot-imap` ``.
+
+
+InterIMAP
+========
+
+On Debian 9 (codename Stretch) and later, installing the packange is one
+command away.  Simply run the following (as root):
+
+    $ apt install interimap
+
+Create config file:
+
+    $ mkdir -pvm 0700 ${XDG_CONFIG_HOME:-~/.config}/interimap \
+        ${XDG_DATA_HOME:-~/.local/share}/interimap
+<!-- -->
+    $ install -m0600 /dev/stdin ${XDG_CONFIG_HOME:-~/.config}/interimap/config <<-EOF
+		# only consider subscribed mailboxes
+		list-select-opts = SUBSCRIBED
+		# ignore the mailbox named 'virtual' and its descendants
+		# WARN: for <=0.4 it should be: ^virtual(?:/|$)
+		ignore-mailbox = ^virtual(?:\x00|$)
+
+		[local]
+		type = tunnel
+		command = exec ~/.local/bin/dovecot-imap
+
+		[remote]
+		type = imaps
+		host = imap.example.net
+		username = myname
+		password = xxxxxxxx
+	EOF
+
+Create the database:
+
+    $ interimap
+    Creating new schema in database file …/interimap.db
+    database: Created mailbox INBOX
+    […]
+
+Override systemd unit if desired, for instance to reduce the interval
+between synchronization runs to 20s:
+
+    $ mkdir -p ${XDG_CONFIG_HOME:-~/.config}/systemd/user/interimap.service.d
+<!-- -->
+    $ install -m 0644 /dev/stdin ${XDG_CONFIG_HOME:-~/.config}/systemd/user/interimap.service.d/override.conf" <<-EOF
+		[Service]
+		ExecStart=
+		ExecStart=/usr/bin/interimap --watch=20
+	EOF
+<!-- -->
+    $ systemctl --user daemon-reload
+
+Enable unit and start it:
+
+    $ systemctl --user enable --now interimap
+
+
+Configure email client
+======================
+
+Put in your mutt configuration (XXX):
+
+    $ cat >/tmp/muttrc <<-EOF
+		set tunnel    = "exec ~/.local/bin/dovecot-imap"
+		set folder    = "imap://foo"
+		set spoolfile = "imap://foo"
+	EOF
+<!-- -->
+    $ mutt -n -F /tmp/muttrc
+
+
+[IMAP4rev1]: https://tools.ietf.org/html/rfc3501
+[`interimap`(1)]: interimap.1.html
+[OfflineIMAP]: https://www.offlineimap.org/
+[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent
+[Mutt]: http://mutt.org/
+[RFC 7162]: https://tools.ietf.org/html/rfc7162
+[Unix philosophy]: https://en.wikipedia.org/wiki/Unix_philosophy
diff --git a/doc/index.md b/doc/index.md
index a403a3e..9225e12 100644
--- a/doc/index.md
+++ b/doc/index.md
@@ -1,13 +1,18 @@
-% [`interimap`(1)] and [`pullimap`(1)]
+% InterIMAP & PullIMAP
 % [Guilhem Moulin](mailto:guilhem@fripost.org)
 
+General documentation
+---------------------
+
+ * [Getting started with InterIMAP](getting-started.html)
+
 Manuals (HTML versions)
 -----------------------
 
-  * [`interimap`(1)] — Fast bidirectional synchronization for
-    QRESYNC-capable IMAP servers
-  * [`pullimap`(1)] — Pull mails from an IMAP mailbox and deliver them
-    to an SMTP session
+  * [`interimap`(1)](interimap.1.html) — Fast bidirectional
+    synchronization for QRESYNC-capable IMAP servers
+  * [`pullimap`(1)](pullimap.1.html) — Pull mails from an IMAP mailbox
+    and deliver them to an SMTP session
 
 Resources for developers
 ------------------------
@@ -15,6 +20,3 @@ Resources for developers
   * [Source-code repository](https://git.guilhem.org/interimap)
   * [Build instructions](build.html)
   * [Test environment setup](development.html)
-
-[`interimap`(1)]: interimap.1.html
-[`pullimap`(1)]: pullimap.1.html
diff --git a/doc/template.html b/doc/template.html
index 2cd7cc9..dbcc0e6 100644
--- a/doc/template.html
+++ b/doc/template.html
@@ -68,7 +68,7 @@ $endif$
 $endif$
 
 $body$
-</div>
+  </div>
 
   <footer>
     <hr/>
-- 
cgit v1.2.3