Arnaud R - planet-debianhttps://arnaudr.io/2023-01-18T00:00:00+00:00Build container images in GitLab CI (iptables-legacy at the rescue)2023-01-18T00:00:00+00:002023-01-18T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2023-01-18:/2023/01/18/build-container-images-in-gitlab-ci-iptables-legacy-at-the-rescue/<p>It's 2023 and these days, building a container image in a CI pipeline should be
straightforward. So let's try.</p>
<p>It's 2023 and these days, building a container image in a CI pipeline should be
straightforward. So let's try.</p>
<p>For this blog post we'll focus on GitLab SaaS only, that is,
<a href="https://gitlab.com">gitlab.com</a>, as it's what I use for work and for personal
projects.</p>
<p>To get started, we just need two files in our Git repository:</p>
<ul>
<li>a <code>Containerfile</code> (or <code>Dockerfile</code> if you prefer to name it this way) that
defines how to build a container image.</li>
<li>a <code>.gitlab-ci.yml</code> file that defines what the CI should do. In the example
below, we want to build a container image and push it to the GitLab registry
associated with the GitLab repo.</li>
</ul>
<p>Here is our Git tree:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-A
Containerfile<span class="w"> </span>.git<span class="w"> </span>.gitlab-ci.yml
$<span class="w"> </span>cat<span class="w"> </span>Containerfile<span class="w"> </span>
FROM<span class="w"> </span>debian:stable
RUN<span class="w"> </span>apt-get<span class="w"> </span>update
CMD<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>hello<span class="w"> </span>world
$<span class="w"> </span>cat<span class="w"> </span>.gitlab-ci.yml<span class="w"> </span>
build-container-image:
<span class="w"> </span>stage:<span class="w"> </span>build
<span class="w"> </span>image:<span class="w"> </span>debian:testing
<span class="w"> </span>before_script:
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>update
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>buildah<span class="w"> </span>ca-certificates
<span class="w"> </span>script:
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>login<span class="w"> </span>-u<span class="w"> </span><span class="nv">$CI_REGISTRY_USER</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$CI_JOB_TOKEN</span><span class="w"> </span><span class="nv">$CI_REGISTRY</span>
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>push<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span>
</code></pre></div>
<p>A few remarks:</p>
<ul>
<li>We use <code>buildah</code>, but we could have used <code>podman</code>.</li>
<li>However we don't use <code>docker</code>, because its client/server design makes it
cumbersome to use in a CI environment: it requires a separate container to
run the Docker daemon, plus setting the <code>DOCKER_HOST</code> variable. Why bother?</li>
</ul>
<p>Now let's push that. Does the CI pass? No, of course, otherwise I wouldn't be
writing this blog post ;)</p>
<p>The CI fails at the <code>buildah build</code> command, with a rather cryptic error:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>--tag<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="o">[</span>...<span class="o">]</span>
STEP<span class="w"> </span><span class="m">2</span>/3:<span class="w"> </span>RUN<span class="w"> </span>apt-get<span class="w"> </span>update
error<span class="w"> </span>running<span class="w"> </span>container:<span class="w"> </span>did<span class="w"> </span>not<span class="w"> </span>get<span class="w"> </span>container<span class="w"> </span>start<span class="w"> </span>message<span class="w"> </span>from<span class="w"> </span>parent:<span class="w"> </span>EOF
Error:<span class="w"> </span>building<span class="w"> </span>at<span class="w"> </span>STEP<span class="w"> </span><span class="s2">"RUN apt-get update"</span>:<span class="w"> </span>netavark:<span class="w"> </span>code:<span class="w"> </span><span class="m">4</span>,<span class="w"> </span>msg:<span class="w"> </span>iptables<span class="w"> </span>v1.8.8<span class="w"> </span><span class="o">(</span>nf_tables<span class="o">)</span>:<span class="w"> </span>Could<span class="w"> </span>not<span class="w"> </span>fetch<span class="w"> </span>rule<span class="w"> </span><span class="nb">set</span><span class="w"> </span>generation<span class="w"> </span>id:<span class="w"> </span>Invalid<span class="w"> </span>argument
</code></pre></div>
<p>The hint here is <code>nf_tables</code>... Back in July 2021, GitLab did a major update of
their shared runners infrastructure, and broke nftables support in the process,
<a href="https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5184#note_637383773">as it seems</a>.
So we have to use iptables instead.</p>
<p>Let's fix our <code>.gitlab-ci.yml</code>, which now looks like that:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>.gitlab-ci.yml<span class="w"> </span>
build-container-image:
<span class="w"> </span>stage:<span class="w"> </span>build
<span class="w"> </span>image:<span class="w"> </span>debian:testing
<span class="w"> </span>before_script:
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>update
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>buildah<span class="w"> </span>ca-certificates
<span class="w"> </span>-<span class="w"> </span><span class="p">|</span>
<span class="w"> </span><span class="c1"># Switch to iptables legacy, as GitLab CI doesn't support nftables.</span>
<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>--no-install-recommends<span class="w"> </span>iptables
<span class="w"> </span>update-alternatives<span class="w"> </span>--set<span class="w"> </span>iptables<span class="w"> </span>/usr/sbin/iptables-legacy
<span class="w"> </span>script:
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>login<span class="w"> </span>-u<span class="w"> </span><span class="nv">$CI_REGISTRY_USER</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$CI_JOB_TOKEN</span><span class="w"> </span><span class="nv">$CI_REGISTRY</span>
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>push<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span>
</code></pre></div>
<p>And push again. Does that work? Yes!</p>
<p>If you're interested in this issue, feel free to fork
<a href="https://gitlab.com/arnaudr/gitlab-build-container-image">https://gitlab.com/arnaudr/gitlab-build-container-image</a> and try it by
yourself.</p>
<p>It's been more than a year since this change, and I'm surprised that I didn't
find much about it on the Internet, neither mentions of the issue, nor of a
workaround. Maybe nobody builds container images in GitLab CI, or maybe they do
it another way, I don't know. In any case, now it's documented in this blog,
hopefully some will find it useful.</p>
<p>Happy 2023!</p>Send emails from your terminal with msmtp2020-08-24T00:00:00+00:002020-08-24T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-24:/2020/08/24/send-emails-from-your-terminal-with-msmtp/<p>In this tutorial, we'll configure everything needed to send emails from the
terminal. We'll use <a href="https://marlam.de/msmtp/">msmtp</a>, a lightweight SMTP client. For the sake of
the example, we'll use a GMail account, but any other email provider can do.
Your OS is expected to be <a href="https://www.debian.org/">Debian</a>, as usual on this blog, although it
doesn't really matter. We will also see how to store the credentials for the
email account in the system keyring. And finally, we'll go the extra mile, and
see how to configure various command-line utilities so that they automatically
use msmtp to send emails. Even better, we'll make msmtp the default email
sender, to actually avoid configuring these utilities one by one.</p>
<p>In this tutorial, we'll configure everything needed to send emails from the
terminal. We'll use <a href="https://marlam.de/msmtp/">msmtp</a>, a lightweight SMTP client. For the sake of
the example, we'll use a GMail account, but any other email provider can do.
Your OS is expected to be <a href="https://www.debian.org/">Debian</a>, as usual on this blog, although it
doesn't really matter. We will also see how to store the credentials for the
email account in the system keyring. And finally, we'll go the extra mile, and
see how to configure various command-line utilities so that they automatically
use msmtp to send emails. Even better, we'll make msmtp the default email
sender, to actually avoid configuring these utilities one by one.</p>
<h2>Prerequisites</h2>
<p>Strong prerequisites (if you don't recognize yourself here, you probably landed
on the wrong page):</p>
<ul>
<li>You run Linux on your computer (let's assume a Debian-like distro).</li>
<li>You want to send emails from your terminal.</li>
</ul>
<p>Weak prerequisites (if your setup doesn't match those points exactly, that's
fine, you can still read on):</p>
<ul>
<li>Your email account is a GMail one.</li>
<li>Your desktop environment is GNOME.</li>
</ul>
<h2>GMail account setup</h2>
<p>For a GMail account, there's a bit of configuration to do. For other email
providers, I have no idea, maybe you can just skip this part, or maybe you will
have to go through a similar procedure.</p>
<p>If you want an external program (<code>msmtp</code> in this case) to talk to the GMail
servers on your behalf, and send emails, you can't just use your usual GMail
password. Instead, GMail requires you to generate so-called <em>app passwords</em>,
one for each application that needs to access your GMail account.</p>
<p>This approach has several advantages:</p>
<ul>
<li>it will basically work, GMail won't block you because it thinks that you're
trying to sign in from an unknown device, a weird location or whatever.</li>
<li>your main GMail password remains secret, you won't have to write it down
in any configuration file or anywhere else.</li>
<li>you can change your main GMail password, no breakage, apps will still work as
each of them use their own passwords.</li>
<li>you can revoke an app password anytime, without impacting anything else.</li>
</ul>
<p>So app passwords are a good idea, it just requires a bit of work to set it up.
Let's see what it takes.</p>
<p>First, <em>2-Step Verification</em> must be enabled on your GMail account. Visit
<a href="https://myaccount.google.com/security">https://myaccount.google.com/security</a>, and if that's not the case, enable it.
You'll need to authorize all of your devices (computer(s), phone(s) and so on),
and it can be a bit tedious, granted. But you only have to do it once in a
lifetime, and after it's done, you're left with a more secure account, so it's
not that bad, right?</p>
<p>Enabling the 2-Step Verification will unlock the feature we need: <em>App
passwords</em>. Visit <a href="https://myaccount.google.com/apppasswords">https://myaccount.google.com/apppasswords</a>, and under
"<em>Signing in to Google</em>", click "<em>App passwords</em>", and generate one. An app
password is a 16 characters string, something like <code>qwertyuiopqwerty</code>. It's
supposed to be used from only one place, ie. from ONE application that is
installed on ONE device. That's why it's common to give it a name of the form
<code>application@device</code>, so in our case it could be <code>msmtp@laptop</code>, but really
it's free form, choose whatever name suits you, as long as it makes sense to
you.</p>
<p>So let's give a name to this app password, write it down for now, and we're
done with the GMail config.</p>
<h2>Send your first email</h2>
<p>Time to get started with <code>msmtp</code>.</p>
<p>First thing first, installation, trivial:</p>
<div class="highlight"><pre><span></span><code>sudo apt install msmtp
</code></pre></div>
<p>Let's try to send an email. At this point, we did not create any configuration
file for msmtp yet, so we have to provide every details on the command line.</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="k">Write</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">dummy</span><span class="w"> </span><span class="n">email</span>
<span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span>
<span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">SOMEONE_ELSE</span><span class="nv">@SOMEWHERE_ELSE</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Cafe</span><span class="w"> </span><span class="n">Sua</span><span class="w"> </span><span class="n">Da</span>
<span class="n">Iced</span><span class="o">-</span><span class="n">coffee</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">condensed</span><span class="w"> </span><span class="n">milk</span>
<span class="n">EOF</span>
<span class="err">#</span><span class="w"> </span><span class="n">Send</span><span class="w"> </span><span class="n">it</span>
<span class="n">cat</span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">auth</span><span class="o">=</span><span class="k">on</span><span class="w"> </span><span class="o">--</span><span class="n">tls</span><span class="o">=</span><span class="k">on</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">port</span><span class="w"> </span><span class="mi">587</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">read</span><span class="o">-</span><span class="n">envelope</span><span class="o">-</span><span class="k">from</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">read</span><span class="o">-</span><span class="n">recipients</span>
<span class="err">#</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="n">prompts</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="nl">password</span><span class="p">:</span>
<span class="err">#</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">goes</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="n">password</span><span class="err">!</span>
</code></pre></div>
<p>Obviously, in this example you should replace the uppercase words with the
real thing, that is, your email login, and real email addresses.</p>
<p>Also, let me insist, you must enter the <em>app password</em> that was generated
previously, not your real GMail password.</p>
<p>And it should work already, this email should have been sent and received by
now.</p>
<p>So let me explain quickly what happened here.</p>
<p>In the file <code>message.txt</code>, we provided <code>From:</code> (the email address of the person
sending the email) and <code>To:</code> (the destination email address). Then we asked
msmtp to re-use those values to set the <a href="https://stackoverflow.com/a/1767262">envelope</a> of the email with
<code>--read-envelope-from</code> and <code>--read-recipients</code>.</p>
<p>What about the other parameters?</p>
<ul>
<li><code>--auth=on</code> because we want to authenticate with the server.</li>
<li><code>--tls=on</code> because we want to make sure that the communication with the
server is encrypted.</li>
<li><code>--host</code> and <code>--port</code> tells where to find the server. If you don't use GMail,
adjust that accordingly.</li>
<li><code>--user</code> is obviously your GMail username.</li>
</ul>
<p>For more details, you should refer to the <a href="https://marlam.de/msmtp/msmtp.html">msmtp documentation</a>.</p>
<h2>Write a configuration file</h2>
<p>So we could send an email, that's cool already.</p>
<p>However the command to do that was a bit long, and we don't want to juggle with
all these arguments every time we send an email. So let's write down all of
that into a configuration file.</p>
<p>There are a few locations that we can use to write down our configuration:
<code>/etc/msmtprc</code> for system wide configuration, and <code>~/.msmtprc</code> or
<code>~/.config/msmtp/config</code> for per-user configuration. All of that is, of course,
thoroughly documented in the <a href="https://manpages.debian.org/msmtp/msmtp.1.en.html#CONFIGURATION_FILES">msmtp manual page</a>.</p>
<p>In this tutorial we'll use <code>~/.msmtprc</code> for brevity. There we go:</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s1">'EOF'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span>
<span class="n">defaults</span>
<span class="n">tls</span><span class="w"> </span><span class="k">on</span>
<span class="n">account</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">auth</span><span class="w"> </span><span class="k">on</span>
<span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">port</span><span class="w"> </span><span class="mi">587</span>
<span class="k">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span>
<span class="k">from</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">EOF</span>
</code></pre></div>
<p>And for a quick explanation:</p>
<ul>
<li>under <code>defaults</code> are the default values for all the following accounts.</li>
<li>under <code>account</code> are the settings specific to this account, until another
<code>account</code> line is found.</li>
<li>finally, the last line defines which account is the default.</li>
</ul>
<p>All in all it's pretty simple, and it's becoming easier to send an email:</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="k">Write</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">dummy</span><span class="w"> </span><span class="n">email</span><span class="p">.</span><span class="w"> </span><span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">the</span>
<span class="err">#</span><span class="w"> </span><span class="n">header</span><span class="w"> </span><span class="s1">'From:'</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">no</span><span class="w"> </span><span class="n">longer</span><span class="w"> </span><span class="n">needed</span><span class="p">,</span>
<span class="err">#</span><span class="w"> </span><span class="n">it</span><span class="s1">'s already in '</span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span><span class="s1">'.</span>
<span class="s1">cat << '</span><span class="n">EOF</span><span class="err">'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">SOMEONE_ELSE</span><span class="nv">@SOMEWHERE_ELSE</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Flat</span><span class="w"> </span><span class="n">White</span>
<span class="n">The</span><span class="w"> </span><span class="n">milky</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">coffee</span>
<span class="n">EOF</span>
<span class="err">#</span><span class="w"> </span><span class="n">Send</span><span class="w"> </span><span class="n">it</span>
<span class="n">cat</span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="c1">--read-recipients</span>
</code></pre></div>
<p>Actually, <code>--account default</code> is not needed, as it's the default anyway if you
don't provide a <code>--account</code> argument. Furthermore <code>--read-recipients</code> can be
shortened as <code>-t</code>. So we can make it real short now:</p>
<div class="highlight"><pre><span></span><code>msmtp -t < message.txt
</code></pre></div>
<p>At this point, life is good! Except for one thing maybe: we still have to type
the password every time we send an email. Surely it must be possible to avoid
that annoyance...</p>
<h2>Store your password in the system keyring</h2>
<p>For this part, we'll make use of the <a href="https://lwn.net/Articles/490518/">libsecret</a> tool to store the password
in the system keyring via the <a href="https://specifications.freedesktop.org/secret-service/latest/">Secret Service API</a>. It means that your
desktop environment should implement the Secret Service specification, which is
the case for both GNOME and KDE.</p>
<p>Note that GNOME provides <a href="https://wiki.gnome.org/Apps/Seahorse">Seahorse</a> to have a look at your secrets, KDE has
the <a href="https://utils.kde.org/projects/kwalletmanager/">KDE Wallet</a>. There's also <a href="https://keepassxc.org/">KeePassXC</a>, which I have only heard of but
never used. I guess it can be your password manager of choice if you use
neither GNOME nor KDE.</p>
<p>For those running an up-to-date Debian unstable, you should have <code>msmtp >=
1.8.11-2</code>, and you're all good to go. For those having an older version than
that however, you will have to install the package <code>msmtp-gnome</code> in order to
have msmtp built with libsecret support. Note that this package depends on
<code>seahorse</code>, hence it pulls in a good part of the GNOME stack when you install
it. For those not running GNOME, that's unfortunate. All of this was discussed
and fixed in <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962689">#962689</a>.</p>
<p>Alright! So let's just make sure that the libsecret tools are installed:</p>
<div class="highlight"><pre><span></span><code><span class="n">sudo</span><span class="w"> </span><span class="n">apt</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">libsecret</span><span class="o">-</span><span class="n">tools</span>
</code></pre></div>
<p>And now we can store our password in the system keyring with this command:</p>
<div class="highlight"><pre><span></span><code><span class="n">secret</span><span class="o">-</span><span class="k">tool</span><span class="w"> </span><span class="n">store</span><span class="w"> </span><span class="o">--</span><span class="n">label</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span>\
<span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">smtp</span><span class="o">.</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="w"> </span>\
<span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">smtp</span><span class="w"> </span>\
<span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span>
</code></pre></div>
<p>If this looks a bit too magic, and you want something more visual, you can
actually fire a GUI like <code>seahorse</code> (for GNOME users), or <code>kwalletmanager5</code>
(for KDE users), and then you will see what passwords are stored in there.</p>
<p>Here's a screenshot of Seahorse, with a msmtp password stored:</p>
<p><img alt="seahorse with msmtp password" src="/images/seahorse-msmtp.png"></p>
<p>Let's try to send an email again:</p>
<div class="highlight"><pre><span></span><code>msmtp -t < message.txt
</code></pre></div>
<p>No need for a password anymore, msmtp got it from the system keyring!</p>
<p>For more details on how msmtp handle the passwords, and to see what other
methods are supported, refer to the <a href="https://marlam.de/msmtp/msmtp.html#Authentication">extensive documentation</a>.</p>
<h2>Use-cases and integration</h2>
<p>Let's go over a few use-cases, situations where you might end up sending emails
from the command-line, and what configuration is required to make it work with
msmtp.</p>
<h3>Git Send-Email</h3>
<p>Sending emails with git is a common workflow for some projects, like the Linux
kernel. How does <code>git send-email</code> actually send emails? From the
<a href="https://manpages.debian.org/git-email/git-send-email.1.en.html">git-send-email manual page</a>:</p>
<blockquote>
<p>the built-in default is to search for sendmail in /usr/sbin, /usr/lib and
$PATH if such program is available</p>
</blockquote>
<p>It is possible to override this default though:</p>
<blockquote>
<p>--smtp-server=<host><br>
[...] Alternatively it can specify a full pathname of a sendmail-like program
instead; the program must support the -i option.</p>
</blockquote>
<p>So in order to use msmtp here, you'd add a snippet like that to your
<code>~/.gitconfig</code> file:</p>
<div class="highlight"><pre><span></span><code><span class="k">[sendemail]</span>
<span class="w"> </span><span class="na">smtpserver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/usr/bin/msmtp</span>
</code></pre></div>
<p>For a full guide, you can also refer to <a href="https://git-send-email.io">https://git-send-email.io</a>.</p>
<h3>Debian developer tools</h3>
<p>Tools like <code>bts</code> or <code>reportbug</code> are also good examples of command-line tools
that need to send emails.</p>
<p>From the <a href="https://manpages.debian.org/devscripts/bts.1.en.html">bts manual page</a>:</p>
<blockquote>
<p>--sendmail=SENDMAILCMD<br>
Specify the sendmail command [...] Default is /usr/sbin/sendmail.</p>
</blockquote>
<p>So if you want bts to send emails with msmtp instead of sendmail, you must use
<code>bts --sendmail='/usr/bin/msmtp -t'</code>.</p>
<p>Note that bts also loads settings from the file <code>/etc/devscripts.conf</code> and
<code>~/.devscripts</code>, so you could also set <code>BTS_SENDMAIL_COMMAND='/usr/bin/msmtp
-t'</code> in one of those files.</p>
<p>From the <a href="https://manpages.debian.org/reportbug/reportbug.1.en.html">reportbug manual page</a>:</p>
<blockquote>
<p>--mta=MTA<br>
Specify an alternate MTA, instead of /usr/sbin/sendmail (the default).</p>
</blockquote>
<p>In order to use msmtp here, you'd write <code>reportbug --mta=/usr/bin/msmtp</code>.</p>
<p>Note that reportbug reads it settings from <code>/etc/reportbug.conf</code> and
<code>~/.reportbugrc</code>, so you could as well set <code>mta /usr/bin/msmtp</code> in one of those
files.</p>
<h3>So who is this sendmail again?</h3>
<p>By now, you probably noticed that <a href="https://en.wikipedia.org/wiki/Sendmail">sendmail</a> seems to be considered the
default tool for the job, the "traditional" command that has been around for
ages.</p>
<p>Rather than configuring every tool to use something else than sendmail,
wouldn't it be simpler to actually replace sendmail by msmtp? Like, create a
symlink that points to msmtp, something like <code>ln -sr /usr/bin/msmtp
/usr/sbin/sendmail</code>? So that msmtp acts as a drop-in replacement for sendmail,
and there's nothing else to configure?</p>
<p>Answer is yes, kind of. Actually, the <a href="https://marlam.de/msmtp/">first msmtp feature</a>
that is listed on the homepage is "<em>Sendmail compatible interface (command
line options and exit codes)</em>". Meaning that msmtp <em>is</em> a drop-in replacement
for sendmail, that seems to be the intent.</p>
<p>However, you should refrain from creating or modifying anything in <code>/usr</code>, as
it's the territory of the package manager, <code>apt</code>. Any change in <code>/usr</code> might be
overwritten by apt the next time you run an upgrade or install new packages.</p>
<p>In the case of msmtp, there is actually a package named <code>msmtp-mta</code> that will
create this symlink for you. So if you really want a definitive replacement for
sendmail, there you go:</p>
<div class="highlight"><pre><span></span><code>sudo apt install msmtp-mta
</code></pre></div>
<p>From this point, sendmail is now a symlink <code>/usr/sbin/sendmail →
/usr/bin/msmtp</code>, and there's no need to configure <code>git</code>, <code>bts</code>, <code>reportbug</code> or
any other tool that would rely on sendmail. Everything should work "out of the
box".</p>
<h2>Conclusion</h2>
<p>I hope that you enjoyed reading this article! If you have any comment, feel
free to send me a short email, preferably from your terminal!</p>Modify Vim syntax files for your taste2020-08-17T00:00:00+00:002020-08-17T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-17:/2020/08/17/modify-vim-syntax-files-for-your-taste/<p>In this short how-to, we'll see how to make small modifications to a Vim syntax
file, in order to change how a particular file format is highlighted. We'll go
for a simple use-case: modify the Markdown syntax file, so that H1 and H2
headings (titles and subtitles, if you prefer) are displayed in bold. Of
course, this won't be exactly as easy as expected, but no worries, we'll
succeed in the end.</p>
<p>In this short how-to, we'll see how to make small modifications to a Vim syntax
file, in order to change how a particular file format is highlighted. We'll go
for a simple use-case: modify the Markdown syntax file, so that H1 and H2
headings (titles and subtitles, if you prefer) are displayed in bold. Of
course, this won't be exactly as easy as expected, but no worries, we'll
succeed in the end.</p>
<h2>The calling</h2>
<p>Let's start with a screenshot: how <a href="https://www.vim.org/">Vim</a> displays <a href="https://commonmark.org/">Markdown</a> files for me,
someone who use the <a href="https://www.gnome.org/">GNOME</a> terminal with the <a href="https://ethanschoonover.com/solarized/">Solarized</a> light theme.</p>
<p><img alt="Vim - Markdown file with original highlighting" src="/images/vim-syntax-01.png"></p>
<p>I'm mostly happy with that, except for one or two little details. I'd like to
have the titles displayed in bold, for example, so that they're easier to spot
when I skim through a Markdown file. It seems like a simple thing to ask, so I
hope there can be a simple solution.</p>
<h2>The first steps</h2>
<p>Let's learn the basics.</p>
<p>In Vim world, the rules to highlight files formats are defined in the directory
<code>/usr/share/vim/vim82/syntax</code> (I bet you'll have to adjust this path depending
on the version of Vim that is installed on your system).</p>
<p>And so, for the Markdown file format, the rules are defined in the file
<code>/usr/share/vim/vim82/syntax/markdown.vim</code>.</p>
<p>The first thing we could do is to have a look at this file, try to make sense
of it, and maybe start to make some modifications.</p>
<p>But wait a moment. You should know that modifying a system file is not a great
idea. First because your changes will be lost as soon as an update kicks in and
the package manager replaces this file by a new version. Second, because you
will quickly forget what files you modified, and what were your modifications,
and if you do that too much, you might experience what is called "maintenance
headache" in the long run.</p>
<p>So instead, maybe you DO NOT modify this file, and instead you copy it in your
personal Vim folder, more precisely in <code>~/.vim/syntax</code>. Create this directory
if it does not exist:</p>
<div class="highlight"><pre><span></span><code>mkdir -p ~/.vim/syntax
cp /usr/share/vim/vim82/syntax/markdown.vim ~/.vim/syntax
</code></pre></div>
<p>The file in your personal folder takes precedence over the system file of the
same name in <code>/usr/share/vim/vim82/syntax/</code>, it is a <em>replacement</em> for the
existing syntax files. And so from now on, Vim uses the file
<code>~/.vim/syntax/markdown.vim</code>, and this is where we can make our modifications.</p>
<p>(And by the way, this is explained in the <a href="https://vimhelp.org/vim_faq.txt.html#faq-24.12">Vim faq-24.12</a>)</p>
<p>And so, it's already nice to know all of that, but wait, there's even better.</p>
<p>There's is another location of interest, and it is <code>~/.vim/after/syntax</code>. You
can drop syntax files in this directory, and these files are treated as
<em>additions</em> to the existing syntax. So if you only want to make slight
modifications, that's the way to go.</p>
<p>(And by the way, this is explained in the <a href="https://vimhelp.org/vim_faq.txt.html#faq-24.11">Vim faq-24.11</a>)</p>
<p>So let's forget about a syntax replacement in <code>~/.vim/syntax/markdown.vim</code>, and
instead let's go for some syntax additions in <code>~/.vim/after/syntax/markdown.vim</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir -p ~/.vim/after/syntax
touch ~/.vim/after/syntax/markdown.vim
</code></pre></div>
<p>Now, let's answer the initial question: how do we modify the highlighting rules
for Markdown files, so that the titles are displayed in bold? First, we have to
understand where are the rules that define the highlighting for titles. Here
there are, from the file <code>/usr/share/vim/vim82/syntax/markdown.vim</code>:</p>
<div class="highlight"><pre><span></span><code>hi def link markdownH1 htmlH1
hi def link markdownH2 htmlH2
hi def link markdownH3 htmlH3
...
</code></pre></div>
<p>You should know that <em>H1</em> means <em>Heading 1</em>, and so on, and so we want to make
H1 and H2 bold. What we can see here is that the headings in the Markdown files
are highlighted like the headings in HTML files, and this is obviously defined
in the file <code>/usr/share/vim/vim82/syntax/html.vim</code>. So let's have a look into
this file:</p>
<div class="highlight"><pre><span></span><code>hi def link htmlH1 Title
hi def link htmlH2 htmlH1
hi def link htmlH3 htmlH2
...
</code></pre></div>
<p>Let's keep digging a bit. Where is <code>Title</code> defined? For those using the
<code>default</code> color scheme like me, this is defined straight in the Vim source
code, in the file
<a href="https://github.com/vim/vim/blob/7ac616cb0a52bc72b449e19cf9db93bee116c15a/src/highlight.c#L190">src/highlight.c</a>.</p>
<div class="highlight"><pre><span></span><code>CENT("Title term=bold ctermfg=DarkMagenta",
"Title term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta"),
</code></pre></div>
<p>And for those using custom color schemes, it might be defined in a file under
<code>/usr/share/vim/vim82/colors/</code>.</p>
<p>Alright, so how do we override that? We can just define this kind of rules in
our syntax additions file at <code>~/.vim/after/syntax/markdown.vim</code>:</p>
<div class="highlight"><pre><span></span><code>hi link markdownH1 markdownHxBold
hi link markdownH2 markdownHxBold
hi markdownHxBold term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta cterm=bold
</code></pre></div>
<p>As you can see, the only addition we made, compared to what's defined in
<code>src/highlight.c</code>, is <code>cterm=bold</code>. And that's already enough to achieve the
initial goal, make the titles (ie. H1 and H2) bold. The result can be seen in
the following screenshot:</p>
<p><img alt="Vim - Markdown file with modified highlighting" src="/images/vim-syntax-02.png"></p>
<h2>The rabbit hole</h2>
<p>So we could stop right here, and life would be easy and good.</p>
<p>However, with this solution there's still something that is not perfect. We use
the color <code>DarkMagenta</code> as defined in the default color scheme. What I didn't
mention however, is that this is applicable for a <em>light</em> background. If you
have a <em>dark</em> background though, dark magenta won't be easy to read.</p>
<p>Actually, if you look a bit more into <code>src/highlight.c</code>, you will see that the
default color scheme comes in two variants, one for a light background, and one
for a dark background.</p>
<p>And so the definition for <code>Title</code> for a dark background is as follow:</p>
<div class="highlight"><pre><span></span><code>CENT("Title term=bold ctermfg=LightMagenta",
"Title term=bold ctermfg=LightMagenta gui=bold guifg=Magenta"),
</code></pre></div>
<p>Hmmm, so how do we do that in our syntax file? How can we support both light
and dark background, so that the color is right in both cases?</p>
<p>After a bit of research, and after looking at other syntax files, it seems that
the solution is to check for the value of the <code>background</code> option, and so our
syntax file becomes:</p>
<div class="highlight"><pre><span></span><code><span class="nv">hi</span><span class="w"> </span><span class="nv">link</span><span class="w"> </span><span class="nv">markdownH1</span><span class="w"> </span><span class="nv">markdownHxBold</span>
<span class="nv">hi</span><span class="w"> </span><span class="nv">link</span><span class="w"> </span><span class="nv">markdownH2</span><span class="w"> </span><span class="nv">markdownHxBold</span>
<span class="k">if</span><span class="w"> </span><span class="o">&</span><span class="nv">background</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"light"</span>
<span class="w"> </span><span class="nv">hi</span><span class="w"> </span><span class="nv">markdownHxBold</span><span class="w"> </span><span class="nv">term</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">ctermfg</span><span class="o">=</span><span class="nv">DarkMagenta</span><span class="w"> </span><span class="nv">gui</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">guifg</span><span class="o">=</span><span class="nv">Magenta</span><span class="w"> </span><span class="nv">cterm</span><span class="o">=</span><span class="nv">bold</span>
<span class="k">else</span>
<span class="w"> </span><span class="nv">hi</span><span class="w"> </span><span class="nv">markdownHxBold</span><span class="w"> </span><span class="nv">term</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">ctermfg</span><span class="o">=</span><span class="nv">LightMagenta</span><span class="w"> </span><span class="nv">gui</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">guifg</span><span class="o">=</span><span class="nv">Magenta</span><span class="w"> </span><span class="nv">cterm</span><span class="o">=</span><span class="nv">bold</span>
<span class="k">endif</span>
</code></pre></div>
<p>In case you wonder, in Vim script you prefix Vim options with <code>&</code>, and so you
get the value of the <code>background</code> option by writing <code>&background</code>. You can
learn this kind of things in the <a href="https://devhints.io/vimscript">Vim scripting cheatsheet</a>.</p>
<p>And so, it's easy enough, except for one thing: it doesn't work. The headings
always show up in <code>DarkMagenta</code>, even for a dark background.</p>
<p>This is why I called this paragraph "the rabbit hole", by the way.</p>
<p>So... Well after trying a few things, I noticed that in order to make it work,
I would have to reload the syntax files with <code>:syntax on</code>.</p>
<p>At this point, the most likely explanation is that the <code>background</code> option is
not set yet when the syntax files are loaded at startup, hence it needs to be
reloaded manually afterward.</p>
<p>And after muuuuuuch research, I found out that it's actually possible to set a
hook for when an option is modified. Meaning, it's possible to execute a
function when the <code>background</code> option is modified. Quite cool actually.</p>
<p>And so, there it goes in my <code>~/.vimrc</code>:</p>
<div class="highlight"><pre><span></span><code><span class="s2">" Reload syntax when the background changes </span>
<span class="n">autocmd</span><span class="w"> </span><span class="n">OptionSet</span><span class="w"> </span><span class="n">background</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">exists</span><span class="p">(</span><span class="s2">"g:syntax_on"</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">syntax</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">endif</span>
</code></pre></div>
<p>For humans, this line reads as:</p>
<ol>
<li>when the background option is modified -- <code>autocmd OptionSet background</code></li>
<li>check if the syntax is on -- <code>if exists("g:syntax_on")</code></li>
<li>if that's the case, reload it -- <code>syntax on</code></li>
</ol>
<p>With that in place, my Markdown syntax overrides work for both dark and light
background. Champagne!</p>
<h2>The happy end</h2>
<p>To finish, let me share my actual additions to the <code>markdown.vim</code> syntax. It
makes H1 and H2 bold, along with their delimiters, and it also colors the
inline code and the code blocks.</p>
<div class="highlight"><pre><span></span><code><span class="s">" H1 and H2 headings -> bold</span>
<span class="s">hi link markdownH1 markdownHxBold</span>
<span class="s">hi link markdownH2 markdownHxBold</span>
<span class="s">"</span><span class="w"> </span><span class="kr">Heading</span><span class="w"> </span><span class="n">delimiters</span><span class="w"> </span><span class="p">(</span><span class="n">eg</span><span class="w"> </span><span class="s">'#'</span><span class="p">)</span><span class="w"> </span><span class="kr">and</span><span class="w"> </span><span class="n">rules</span><span class="w"> </span><span class="p">(</span><span class="n">eg</span><span class="w"> </span><span class="s">'----'</span><span class="p">,</span><span class="w"> </span><span class="s">'===='</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">bold</span>
<span class="n">hi</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">markdownHeadingDelimiter</span><span class="w"> </span><span class="n">markdownHxBold</span>
<span class="n">hi</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">markdownRule</span><span class="w"> </span><span class="n">markdownHxBold</span>
<span class="s">" Code blocks and inline code -> highlighted</span>
<span class="s">hi link markdownCode htmlH1</span>
<span class="s">"</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">following</span><span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="n">requires</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">addition</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">vimrc</span><span class="o">:</span>
<span class="s">" autocmd OptionSet background if exists("</span><span class="n">g</span><span class="o">:</span><span class="n">syntax_on</span><span class="s">") | syntax on | endif</span>
<span class="s">if &background == "</span><span class="n">light</span><span class="s">"</span>
<span class="s"> hi markdownHxBold term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta cterm=bold</span>
<span class="s">else</span>
<span class="s"> hi markdownHxBold term=bold ctermfg=LightMagenta gui=bold guifg=Magenta cterm=bold</span>
<span class="s">endif</span>
</code></pre></div>
<p>And here's how it looks like with a light background:</p>
<p><img alt="Vim - Markdown file with final highlighting (light)" src="/images/vim-syntax-03.png"></p>
<p>And a dark background:</p>
<p><img alt="Vim - Markdown file with final highlighting (dark)" src="/images/vim-syntax-04.png"></p>
<p>That's all, that's very little changes compared to the highlighting from the
original syntax file, and now that we understand how it's supposed to be done,
it's not much effort to achieve it.</p>
<p>It's just that finding the workaround to make it work for both light and dark
background took forever, and leaves the usual, unanswered question: <a href="https://github.com/vim/vim/issues/6692">bug or
feature</a>?</p>GoAccess 1.4, a detailed tutorial2020-08-10T00:00:00+00:002020-08-10T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-10:/2020/08/10/goaccess-14-a-detailed-tutorial/<p>GoAccess v1.4 was just released a few weeks ago! Let's take this chance to
write a loooong tutorial. We'll go over every steps to install and operate
GoAccess. This is a tutorial aimed at those who don't play sysadmin every day,
and that's why it's so long, I did my best to provide thorough explanations all
along, so that it's more than just a "copy-and-paste" kind of tutorial. And for
those who do play sysadmin everyday: please try not to fall asleep while
reading, and don't hesitate to drop me an e-mail if you spot anything
inaccurate in here. Thanks!</p>
<p>GoAccess v1.4 was just released a few weeks ago! Let's take this chance to
write a loooong tutorial. We'll go over every steps to install and operate
GoAccess. This is a tutorial aimed at those who don't play sysadmin every day,
and that's why it's so long, I did my best to provide thorough explanations all
along, so that it's more than just a "copy-and-paste" kind of tutorial. And for
those who do play sysadmin everyday: please try not to fall asleep while
reading, and don't hesitate to drop me an e-mail if you spot anything
inaccurate in here. Thanks!</p>
<h2>Introduction</h2>
<p>So what's <a href="https://goaccess.io/">GoAccess</a> already? GoAccess is a web log analyzer, and it allows
you to visualize the traffic for your website, and get to know a bit more about
your visitors: how many visitors and hits, for which pages, coming from where
(geolocation, operating system, web browser...), etc... It does so by
parsing the access logs from your web server, be it Apache, NGINX or whatever.</p>
<p>GoAccess gives you different options to display the statistics, and in this
tutorial we'll focus on producing a HTML report. Meaning that you can see the
statistics for your website straight in your web browser, under the form of a
single HTML page.</p>
<p>For an example, you can have a look at the stats of my blog here:
<a href="https://goaccess.arnaudr.io">https://goaccess.arnaudr.io</a>.</p>
<p>GoAccess is written in C, it has very few dependencies, it had been around for
about 10 years, and it's distributed under the MIT license.</p>
<h2>Assumptions</h2>
<p>This tutorial is about installing and configuring, so I'll assume that all the
commands are run as root. I won't prefix each of them with <code>sudo</code>.</p>
<p>I use the <a href="https://httpd.apache.org/">Apache</a> web server, running on a <a href="https://www.debian.org/">Debian</a> system. I don't think
it matters so much for this tutorial though. If you're using <a href="https://www.nginx.com/">NGINX</a> it's
fine, you can keep reading.</p>
<p>Also, I will just use the name <code>SITE</code> for the name of the website that we want
to analyze with GoAccess. Just replace that with the real name of your site.</p>
<p>I also assume the following locations for your stuff:</p>
<ul>
<li>the website is at <code>/var/www/SITE</code></li>
<li>the logs are at <code>/var/log/apache2/SITE</code> (yes, there is a sub-directory)</li>
<li>we're going to save the GoAccess database in <code>/var/lib/goaccess-db/SITE</code>.</li>
</ul>
<p>If you have your stuff in <code>/srv/SITE/{log,www}</code> instead, no worries, just
adjust the paths accordingly, I bet you can do it.</p>
<h2>Installation</h2>
<p>The latest version of GoAccess is <code>v1.4</code>, and it's not yet available in the
Debian repositories. So for this part, you can follow the instructions from
the official <a href="https://goaccess.io/download#official-repo">GoAccess download page</a>. Install steps
are explained in details, so there's nothing left for me to say :)</p>
<p>When this is done, let's get started with the basics.</p>
<p>We're talking about the latest version <code>v1.4</code> here, let's make sure:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>goaccess<span class="w"> </span>--version
GoAccess<span class="w"> </span>-<span class="w"> </span><span class="m">1</span>.4.
...
</code></pre></div>
<p>Now let's try to create a HTML report. I assume that you already have a website
up and running.</p>
<p>GoAccess needs to parse the <strong>access logs</strong>. These logs are optional, they
might or might not be created by your web server, depending on how it's
configured. Usually, these log files are named <code>access.log</code>, unsurprisingly.</p>
<p>You can check if those logs exist on your system by running this command:</p>
<div class="highlight"><pre><span></span><code><span class="n">find</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="w"> </span><span class="o">-</span><span class="n">name</span><span class="w"> </span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>Another important thing to know is that these logs can be in different formats.
In this tutorial we'll assume that we work with the <strong>combined log format</strong>,
because it seems to be the most common default.</p>
<p>To check what kind of access logs your web server produces, you must look at
the configuration for your site.</p>
<p>For an Apache web server, you should have such a line in the file
<code>/etc/apache2/sites-enabled/SITE.conf</code>:</p>
<div class="highlight"><pre><span></span><code>CustomLog<span class="w"> </span><span class="cp">${</span><span class="n">APACHE_LOG_DIR</span><span class="cp">}</span>/SITE/access.log<span class="w"> </span>combined
</code></pre></div>
<p>For NGINX, it's quite similar. The configuration file would be something like
<code>/etc/nginx/sites-enabled/SITE</code>, and the line to enable access logs would be
something like:</p>
<div class="highlight"><pre><span></span><code><span class="n">access_log</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">SITE</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>Note that <a href="https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#access_log">NGINX writes the access logs in the combined format</a>
by default, that's why you don't see the word <code>combined</code> anywhere in the line
above: it's implicit.</p>
<p>Alright, so from now on we assume that yes, you have access log files
available, and yes, they are in the combined log format. If that's the case,
then you can already run GoAccess and generate a report, for example for the
log file <code>/var/log/apache2/access.log</code></p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>It's possible to give GoAccess more than one log files to process, so if you
have for example the file <code>access.log.1</code> around, you can use it as well:</p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="mi">1</span>
</code></pre></div>
<p>If GoAccess succeeds (and it should), you're on the right track!</p>
<p>All is left to do to complete this test is to have a look at the HTML report
created. It's a single HTML page, so you can easily <code>scp</code> it to your machine,
or just move it to the document root of your site, and then open it in your web
browser.</p>
<p>Looks good? So let's move on to more interesting things.</p>
<h2>Web server configuration</h2>
<p>This part is very short, because in terms of configuration of the web server,
there's very little to do. As I said above, the only thing you want from the
web server is to create access log files. Then you want to be sure that
GoAccess and your web server agree on the format for these files.</p>
<p>In the part above we used the combined log format, but GoAccess supports many
other common log formats out of the box, and even allows you to parse custom
log formats. For more details, refer to the option <code>--log-format</code> in the
<a href="https://goaccess.io/man#options">GoAccess manual page</a>.</p>
<p>Another common log format is named, well, <code>common</code>. It even has its own
<a href="https://en.wikipedia.org/wiki/Common_Log_Format">Wikipedia page</a>. But compared to <code>combined</code>, the common
log format contains less information, it doesn't include the <code>referrer</code> and
<code>user-agent</code> values, meaning that you won't have it in the GoAccess report.</p>
<p>So at this point you should understand that, unsurprisingly, GoAccess can only
tell you about what's in the access logs, no more no less.</p>
<p>And that's all in term of web server configuration.</p>
<h2>Configuration to run GoAccess unprivileged</h2>
<p>Now we're going to create a user and group for GoAccess, so that we don't have
to run it as root. The reason is that, well, for everything running unattended
on your server, the less code runs as root, the better. It's good practice and
common sense.</p>
<p>In this case, GoAccess is simply a log analyzer. So it just needs to read the
logs files from your web server, and there is no need to be root for that, an
unprivileged user can do the job just as well, assuming it has read permissions
on <code>/var/log/apache2</code> or <code>/var/log/nginx</code>.</p>
<p>The log files of the web server are usually part of the <code>adm</code> group (though it
might depend on your distro, I'm not sure). This is something you can check
easily with the following command:</p>
<div class="highlight"><pre><span></span><code><span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="o">-</span><span class="n">e</span><span class="w"> </span><span class="n">apache2</span><span class="w"> </span><span class="o">-</span><span class="n">e</span><span class="w"> </span><span class="n">nginx</span>
</code></pre></div>
<p>As a result you should get something like that:</p>
<div class="highlight"><pre><span></span><code><span class="n">drwxr</span><span class="o">-</span><span class="n">x</span><span class="o">---</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">adm</span><span class="w"> </span><span class="mi">20480</span><span class="w"> </span><span class="n">Jul</span><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span>
</code></pre></div>
<p>And as you can see, the directory <code>apache2</code> belongs to the group <code>adm</code>. It
means that you don't need to be root to read the logs, instead any unprivileged
user that belongs to the group <code>adm</code> can do it.</p>
<p>So, let's create the <code>goaccess</code> user, and add it to the <code>adm</code> group:</p>
<div class="highlight"><pre><span></span><code>adduser --system --group --no-create-home goaccess
addgroup goaccess adm
</code></pre></div>
<p>And now, let's run GoAccess unprivileged, and verify that it can still read
the log files:</p>
<div class="highlight"><pre><span></span><code><span class="n">setpriv</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">reuid</span><span class="o">=</span><span class="n">goaccess</span><span class="w"> </span><span class="o">--</span><span class="n">regid</span><span class="o">=</span><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">init</span><span class="o">-</span><span class="n">groups</span><span class="w"> </span><span class="o">--</span><span class="n">inh</span><span class="o">-</span><span class="n">caps</span><span class="o">=-</span><span class="n">all</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="w"> </span>\
<span class="w"> </span><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report2</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p><code>setpriv</code> is the command used to drop privileges. The syntax is quite verbose,
it's not super friendly for tutorials, but don't be scared and read the manual
page to learn what it does.</p>
<p>In any case, this command should work, and at this point, it means that you
have a <code>goaccess</code> user ready, and we'll use it to run GoAccess unprivileged.</p>
<h2>Integration, option A - Run GoAccess once a day, from a logrotate hook</h2>
<p>In this part we wire things together, so that GoAccess processes the log files
once a day, adds the new logs to its internal database, and generates a report
from all that aggregated data. The result will be a single HTML page.</p>
<h3>Introducing logrotate</h3>
<p>In order to do that, we'll use a logrotate hook. <a href="https://manpages.debian.org/stable/logrotate/logrotate.8.en.html">logrotate</a> is a little tool
that should already be installed on your server, and that runs once a day, and
that is in charge of rotating the log files. "Rotating the logs" means moving
<code>access.log</code> to <code>access.log.1</code> and so on. With logrotate, a new log file is
created every day, and log files that are too old are deleted. That's what
prevents your logs from filling up your disk basically :)</p>
<p>You can check that logrotate is indeed installed and enabled with this command
(assuming that your init system is <code>systemd</code>):</p>
<div class="highlight"><pre><span></span><code><span class="nv">systemctl</span><span class="w"> </span><span class="nv">status</span><span class="w"> </span><span class="k">logrotate</span>.<span class="nv">timer</span>
</code></pre></div>
<p>What's interesting for us is that <code>logrotate</code> allows you to run scripts before
and after the rotation is performed, so it's an ideal place from where to run
GoAccess. In short, we want to run GoAccess just before the logs are rotated
away, in the <code>prerotate</code> hook.</p>
<p>But let's do things in order. At first, we need to write a little wrapper
script that will be in charge of running GoAccess with the right arguments, and
that will process all of your sites.</p>
<h3>The wrapper script</h3>
<p>This wrapper is made to process more than one site, but if you have only one
site it works just as well, of course.</p>
<p>So let me just drop it on you like that, and I'll explain afterward. Here's my
wrapper script:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1"># Process log files /var/www/apache2/SITE/access.log,</span>
<span class="c1"># only if /var/lib/goaccess-db/SITE exists.</span>
<span class="c1"># Create HTML reports in $1, a directory that must exist.</span>
<span class="nb">set</span><span class="w"> </span>-eu
<span class="nv">OUTDIR</span><span class="o">=</span>
<span class="nv">LOGDIR</span><span class="o">=</span>/var/log/apache2
<span class="nv">DBDIR</span><span class="o">=</span>/var/lib/goaccess-db
fail<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"Usage: </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="s2"> OUTPUT_DIRECTORY"</span>
<span class="nv">OUTDIR</span><span class="o">=</span><span class="nv">$1</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$OUTDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$OUTDIR</span><span class="s2">' is not a directory"</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$LOGDIR</span><span class="s2">' is not a directory"</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$DBDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$DBDIR</span><span class="s2">' is not a directory"</span>
<span class="k">for</span><span class="w"> </span>d<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="k">$(</span>find<span class="w"> </span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">"</span><span class="w"> </span>-mindepth<span class="w"> </span><span class="m">1</span><span class="w"> </span>-maxdepth<span class="w"> </span><span class="m">1</span><span class="w"> </span>-type<span class="w"> </span>d<span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">site</span><span class="o">=</span><span class="k">$(</span>basename<span class="w"> </span><span class="s2">"</span><span class="nv">$d</span><span class="s2">"</span><span class="k">)</span>
<span class="w"> </span><span class="nv">dbdir</span><span class="o">=</span><span class="nv">$DBDIR</span>/<span class="nv">$site</span>
<span class="w"> </span><span class="nv">logfile</span><span class="o">=</span><span class="nv">$d</span>/access.log
<span class="w"> </span><span class="nv">outfile</span><span class="o">=</span><span class="nv">$OUTDIR</span>/<span class="nv">$site</span>.html
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$dbdir</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"‣ Skipping site '</span><span class="nv">$site</span><span class="s2">'"</span>
<span class="w"> </span><span class="k">continue</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"‣ Processing site '</span><span class="nv">$site</span><span class="s2">'"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>setpriv<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--reuid<span class="o">=</span>goaccess<span class="w"> </span>--regid<span class="o">=</span>goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--init-groups<span class="w"> </span>--inh-caps<span class="o">=</span>-all<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--agent-list<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--anonymize-ip<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--persist<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--restore<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--config-file<span class="w"> </span>/etc/goaccess/goaccess.conf<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-path<span class="w"> </span><span class="s2">"</span><span class="nv">$dbdir</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--log-format<span class="w"> </span><span class="s2">"COMBINED"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--output<span class="w"> </span><span class="s2">"</span><span class="nv">$outfile</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span>
<span class="k">done</span>
</code></pre></div>
<p>So you'd install this script at <code>/usr/local/bin/goaccess-wrapper</code> for example,
and make it executable:</p>
<div class="highlight"><pre><span></span><code>chmod +x /usr/local/bin/goaccess-wrapper
</code></pre></div>
<p>A few things to note:</p>
<ul>
<li>We run GoAccess with <code>--persist</code>, meaning that we save the parsed
logs in the internal database, and <code>--restore</code>, meaning that we include
everything from the database in the report. In other words, we aggregate the
data at every run, and the report grows bigger every time.</li>
<li>The parameter <code>--config-file /etc/goaccess/goaccess.conf</code> is a
workaround for <a href="https://github.com/allinurl/goaccess/issues/1849">#1849</a>.
It should not be needed for future versions of GoAccess <code>> 1.4</code>.</li>
</ul>
<p>As is, the script makes the assumption that the logs for your site are logged
in a sub-directory <code>/var/log/apache2/SITE/</code>. If it's not the case, adjust that
in the wrapper accordingly.</p>
<p>The name of this sub-directory is then used to find the GoAccess database
directory <code>/var/lib/goaccess-db/SITE/</code>. This directory is expected to exist,
meaning that if you don't create it yourself, the wrapper won't process this
particular site. It's a simple way to control which sites are processed by
this GoAccess wrapper, and which sites are not.</p>
<p>So if you want <code>goaccess-wrapper</code> to process the site <code>SITE</code>, just create a
directory with the name of this site under <code>/var/lib/goaccess-db</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
<span class="n">chown</span><span class="w"> </span><span class="n">goaccess</span><span class="p">:</span><span class="n">goaccess</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>Now let's create an output directory:</p>
<div class="highlight"><pre><span></span><code>mkdir /tmp/goaccess-reports
chown goaccess:goaccess /tmp/goaccess-reports
</code></pre></div>
<p>And let's give a try to the wrapper script:</p>
<div class="highlight"><pre><span></span><code>goaccess-wrapper /tmp/goaccess-reports
ls /tmp/goaccess-reports
</code></pre></div>
<p>Which should give you:</p>
<div class="highlight"><pre><span></span><code>SITE.html
</code></pre></div>
<p>At the same time, you can check that GoAccess populated the database with a
bunch of files:</p>
<div class="highlight"><pre><span></span><code><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<h3>Setting up the logrotate prerotate hook</h3>
<p>At this point, we have the wrapper in place. Let's now add a pre-rotate hook so
that <code>goaccess-wrapper</code> runs once a day, just before the logs are rotated away.</p>
<p>The logrotate config file for Apache2 is located at <code>/etc/logrotate.d/apache2</code>,
and for NGINX it's at <code>/etc/logrotate.d/nginx</code>. Among the many things you'll
see in this file, here's what is of interest for us:</p>
<ul>
<li><code>daily</code> means that your logs are rotated every day</li>
<li><code>sharedscripts</code> means that the pre-rotate and post-rotate scripts are
executed once total per rotation, and not once per log file.</li>
</ul>
<p>In the config file, there is also this snippet:</p>
<div class="highlight"><pre><span></span><code><span class="nv">prerotate</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span>[<span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="w"> </span>]<span class="c1">; then \</span>
<span class="w"> </span><span class="nv">run</span><span class="o">-</span><span class="nv">parts</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="c1">; \</span>
<span class="w"> </span><span class="nv">fi</span><span class="c1">; \</span>
<span class="nv">endscript</span>
</code></pre></div>
<p>It indicates that scripts in the directory <code>/etc/logrotate.d/httpd-prerotate/</code>
will be executed before the rotation takes place. Refer to the man page
<code>run-parts(8)</code> for more details...</p>
<p>Putting all of that together, it means that logs from the web server are
rotated once a day, and if we want to run scripts just before the rotation, we
can just drop them in the <code>httpd-prerotate</code> directory. Simple, right?</p>
<p>Let's first create this directory if it doesn't exist:</p>
<div class="highlight"><pre><span></span><code><span class="nv">mkdir</span><span class="w"> </span><span class="o">-</span><span class="nv">p</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="o">/</span>
</code></pre></div>
<p>And let's create a tiny script at <code>/etc/logrotate.d/httpd-prerotate/goaccess</code>:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="nb">exec</span><span class="w"> </span>goaccess-wrapper<span class="w"> </span>/tmp/goaccess-reports
</code></pre></div></td></tr></table></div>
<p>Don't forget to make it executable:</p>
<div class="highlight"><pre><span></span><code><span class="nv">chmod</span><span class="w"> </span><span class="o">+</span><span class="nv">x</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="o">/</span><span class="nv">goaccess</span>
</code></pre></div>
<p>As you can see, the only thing that this script does is to invoke the wrapper
with the right argument, ie. the output directory for the HTML reports that are
generated.</p>
<p>And that's all. Now you can just come back tomorrow, check the logs, and make
sure that the hook was executed and succeeded. For example, this kind of
command will tell you quickly if it worked:</p>
<div class="highlight"><pre><span></span><code><span class="nv">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">grep</span><span class="w"> </span><span class="k">logrotate</span>
</code></pre></div>
<h2>Integration, option B - Run GoAccess once a day, from a systemd service</h2>
<p>OK so we've just seen how to use a logrotate hook. One downside with that is
that we have to drop privileges in the wrapper script, because logrotate runs
as root, and we don't want to run GoAccess as root. Hence the rather convoluted
syntax with <code>setpriv</code>.</p>
<p>Rather than embedding this kind of thing in a wrapper script, we can instead
run the wrapper script from a <a href="https://systemd.io/">systemd</a> service, and define which user runs
the wrapper straight in the systemd service file.</p>
<h3>Introducing systemd niceties</h3>
<p>So we can create a <a href="https://www.freedesktop.org/software/systemd/man/systemd.service.html">systemd service</a>, along with a <a href="https://www.freedesktop.org/software/systemd/man/systemd.timer.html">systemd timer</a> that
fires daily. We can then set the <code>user</code> and <code>group</code> that execute the script
straight in the systemd service, and there's no need for <code>setpriv</code> anymore.
It's a bit more streamlined.</p>
<p>We can even go a bit further, and use <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">systemd parameterized units</a> (also
called templates), so that we have one service per site (instead of one service
that process all of our sites). That will simplify the wrapper script a lot,
and it also looks nicer in the logs.</p>
<p>With this approach however, it seems that we can't really run <em>exactly</em> before
the logs are rotated away, like we did in the section above. But that's OK.
What we'll do is that we'll run once a day, no matter the time, and we'll just
make sure to process both log files <code>access.log</code> and <code>access.log.1</code> (ie. the
current logs and the logs from yesterday). This way, we're sure not to miss any
line from the logs.</p>
<p>Note that GoAccess is smart enough to only consider newer entries from the log
files, and discard entries that are already in the database. In other words,
it's safe to parse the same log file more than once, GoAccess will do the right
thing. For more details see "INCREMENTAL LOG PROCESSING" from <code>man goaccess</code>.</p>
<h3>Implementation</h3>
<p>And here's how it all looks like.</p>
<p>First, a little wrapper script for GoAccess:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1"># Usage: $0 SITE DBDIR LOGDIR OUTDIR</span>
<span class="nb">set</span><span class="w"> </span>-eu
<span class="nv">SITE</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">DBDIR</span><span class="o">=</span><span class="nv">$2</span>
<span class="nv">LOGDIR</span><span class="o">=</span><span class="nv">$3</span>
<span class="nv">OUTDIR</span><span class="o">=</span><span class="nv">$4</span>
<span class="nv">LOGFILES</span><span class="o">=()</span>
<span class="k">for</span><span class="w"> </span>ext<span class="w"> </span><span class="k">in</span><span class="w"> </span>log<span class="w"> </span>log.1<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">logfile</span><span class="o">=</span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">/access.</span><span class="nv">$ext</span><span class="s2">"</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nv">LOGFILES</span><span class="o">+=(</span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="o">)</span>
<span class="k">done</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="si">${#</span><span class="nv">LOGFILES</span><span class="p">[@]</span><span class="si">}</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"No log files in '</span><span class="nv">$LOGDIR</span><span class="s2">'"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="k">fi</span>
goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--agent-list<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--anonymize-ip<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--persist<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--restore<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--config-file<span class="w"> </span>/etc/goaccess/goaccess.conf<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-path<span class="w"> </span><span class="s2">"</span><span class="nv">$DBDIR</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--log-format<span class="w"> </span><span class="s2">"COMBINED"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--output<span class="w"> </span><span class="s2">"</span><span class="nv">$OUTDIR</span><span class="s2">/</span><span class="nv">$SITE</span><span class="s2">.html"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">LOGFILES</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
</code></pre></div>
<p>This wrapper does very little. Actually, the only thing it does is to check for
the existence of the two log files <code>access.log</code> and <code>access.log.1</code>, to be sure
that we don't ask GoAccess to process a file that does not exist (GoAccess
would not be happy about that).</p>
<p>Save this file under <code>/usr/local/bin/goaccess-wrapper</code>, don't forget to make it
executable:</p>
<div class="highlight"><pre><span></span><code>chmod +x /usr/local/bin/goaccess-wrapper
</code></pre></div>
<p>Then, create a systemd parameterized unit file, so that we can run this wrapper
as a systemd service. Save it under <code>/etc/systemd/system/goaccess@.service</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Update GoAccess report - %i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/var/lib/goaccess-db/%i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/var/log/apache2/%i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/tmp/goaccess-reports</span>
<span class="na">PartOf</span><span class="o">=</span><span class="s">goaccess.service</span>
<span class="k">[Service]</span>
<span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
<span class="na">User</span><span class="o">=</span><span class="s">goaccess</span>
<span class="na">Group</span><span class="o">=</span><span class="s">goaccess</span>
<span class="na">Nice</span><span class="o">=</span><span class="s">19</span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/goaccess-wrapper </span>\
<span class="w"> </span><span class="s">%i </span>\
<span class="w"> </span><span class="s">/var/lib/goaccess-db/%i </span>\
<span class="w"> </span><span class="s">/var/log/apache2/%i </span>\
<span class="w"> </span><span class="s">/tmp/goaccess-reports</span>
</code></pre></div>
<p>So, what is a systemd parameterized unit? It's a service to which you can pass
an argument when you enable it. The <code>%i</code> in the unit definition will be
replaced by this argument. In our case, the argument will be the name of the
site that we want to process.</p>
<p>As you can see, we use the directive <code>ConditionPathIsDirectory=</code> extensively,
so that if ever one of the required directories does not exist, the unit will
just be skipped (and marked as such in the logs). It's a graceful way to fail.</p>
<p>We run the wrapper as the user and group <code>goaccess</code>, thanks to <code>User=</code> and
<code>Group=</code>. We also use <code>Nice=</code> to give a low priority to the process.</p>
<p>At this point, it's already possible to test. Just make sure that you created a
directory for the GoAccess database:</p>
<div class="highlight"><pre><span></span><code><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
<span class="n">chown</span><span class="w"> </span><span class="n">goaccess</span><span class="p">:</span><span class="n">goaccess</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>Also make sure that the output directory exists:</p>
<div class="highlight"><pre><span></span><code>mkdir /tmp/goaccess-reports
chown goaccess:goaccess /tmp/goaccess-reports
</code></pre></div>
<p>Then reload systemd and fire the unit to see if it works:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">goaccess</span><span class="err">@</span><span class="n">SITE</span><span class="o">.</span><span class="n">service</span>
<span class="n">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">tail</span>
</code></pre></div>
<p>And that should work already.</p>
<p>As you can see, the argument, <code>SITE</code>, is passed in the <code>systemctl start</code>
command. We just append it after the <code>@</code>, in the name of the unit.</p>
<p>Now, let's create another GoAccess service file, which sole purpose is to group
all the parameterized units together, so that we can start them all in one go.
Note that we don't use a systemd target for that, because ultimately we want to
run it once a day, and that would not be possible with a target. So instead we
use a dummy oneshot service.</p>
<p>So here it is, saved under <code>/etc/systemd/system/goaccess.service</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Update GoAccess reports</span>
<span class="na">Requires</span><span class="o">=</span><span class="w"> </span>\
<span class="w"> </span><span class="s">goaccess@SITE1.service </span>\
<span class="w"> </span><span class="s">goaccess@SITE2.service</span>
<span class="k">[Service]</span>
<span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">true</span>
</code></pre></div>
<p>As you can see, we simply list the sites that we want to process in the
<code>Requires=</code> directive. In this example we have two sites named <code>SITE1</code> and
<code>SITE2</code>.</p>
<p>Let's ensure that everything is still good:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">goaccess</span><span class="o">.</span><span class="n">service</span>
<span class="n">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">tail</span>
</code></pre></div>
<p>Check the logs, both sites <code>SITE1</code> and <code>SITE2</code> should have been processed.</p>
<p>And finally, let's create a timer, so that systemd runs <code>goaccess.service</code> once
a day. Save it under <code>/etc/systemd/system/goaccess.timer</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Daily update of GoAccess reports</span>
<span class="k">[Timer]</span>
<span class="na">OnCalendar</span><span class="o">=</span><span class="s">daily</span>
<span class="na">RandomizedDelaySec</span><span class="o">=</span><span class="s">1h</span>
<span class="na">Persistent</span><span class="o">=</span><span class="s">true</span>
<span class="k">[Install]</span>
<span class="na">WantedBy</span><span class="o">=</span><span class="s">timers.target</span>
</code></pre></div>
<p>Finally, enable the timer:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">enable</span><span class="w"> </span><span class="o">--</span><span class="n">now</span><span class="w"> </span><span class="n">goaccess</span><span class="o">.</span><span class="n">timer</span>
</code></pre></div>
<p>At this point, everything should be OK. Just come back tomorrow and check the
logs with something like:</p>
<div class="highlight"><pre><span></span><code>journalctl | grep goaccess
</code></pre></div>
<p>Last word: if you have only one site to process, of course you can simplify,
for example you can hardcode all the paths in the file <code>goaccess.service</code>
instead of using a parameterized unit. Up to you.</p>
<h2>Daily operations</h2>
<p>So in this part, we assume that you have GoAccess all setup and running, once
a day or so. Let's just go over a few things worth noting.</p>
<h3>Serve your report</h3>
<p>Up to now in this tutorial, we created the reports in <code>/tmp/goaccess-reports</code>,
but that was just for the sake of the example. You will probably want to save
your reports in a directory that is served by your web server, so that, well,
you can actually look at it in your web browser, that was the point, right?</p>
<p>So how to do that is a bit out of scope here, and I guess that if you want to
monitor your website, you already <em>have</em> a website, so you will have no trouble
serving the GoAccess HTML report.</p>
<p>However there's an important detail to be aware of: GoAccess shows all the IP
addresses of your visitors in the report. As long as the report is private it's
OK, but if ever you make your GoAccess report public, then you should
definitely invoke GoAccess with the option <code>--anonymize-ip</code>.</p>
<h3>Keep an eye on the logs</h3>
<p>In this tutorial, the reports we create, along with the GoAccess databases,
will grow bigger every day, forever. It also means that the GoAccess processing
time will grow a bit each day.</p>
<p>So maybe the first thing to do is to keep an eye on the logs, to see how long
it takes to GoAccess to do its job every day. Also, maybe you'd like to keep
an eye on the size of the GoAccess database with:</p>
<div class="highlight"><pre><span></span><code><span class="n">du</span><span class="w"> </span><span class="o">-</span><span class="n">sh</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>If your site has few visitors, I suspect it won't be a problem though.</p>
<p>You could also be a bit pro-active in preventing this problem in the future,
and for example you could break the reports into, say, monthly reports. Meaning
that every month, you would create a new database in a new directory, and also
start a new HTML report. This way you'd have monthly reports, and you make sure
to limit the GoAccess processing time, by limiting the database size to a
month.</p>
<p>This can be achieved very easily, by including something like <code>YEAR-MONTH</code> in
the database directory, and in the HTML report. You can handle that
automatically in the wrapper script, for example:</p>
<div class="highlight"><pre><span></span><code>sfx=$(date +'%Y-%m')
mkdir -p $DBDIR/$sfx
goaccess \
--db-path $DBDIR/$sfx \
--output "$OUTDIR/$SITE-$sfx.html" \
...
</code></pre></div>
<p>You get the idea.</p>
<h2>Further notes</h2>
<h3>Migration from older versions</h3>
<p>With the <code>--persist</code> option, GoAccess keeps all the information from the logs
in a database, so that it can re-use it later. In prior versions, GoAccess used
the <a href="https://dbdb.io/db/tokyo-cabinet">Tokyo Cabinet</a> key-value store for that. However starting
from <code>v1.4</code>, GoAccess dropped this dependency and now uses its own database format.</p>
<p>As a result, the previous database can't be used anymore, you will have to
remove it and restart from zero. At the moment there is no way to convert the
data from the old database to the new one. If you're interested, this is
discussed upstream at <a href="https://github.com/allinurl/goaccess/issues/1783">#1783</a>.</p>
<p>Another thing that changed with this new version is the name for some of the
command-line options. For example, <code>--load-from-disk</code> was dropped in favor of
<code>--restore</code>, and <code>--keep-db-files</code> became <code>--persist</code>. So you'll have to
look at the documentation a bit, and update your script(s) accordingly.</p>
<h3>Other ways to use GoAccess</h3>
<p>It's also possible to do it completely differently. You could keep GoAccess
running, pretty much like a daemon, with the <code>--real-time-html</code> option, and
have it process the logs continuously, rather than calling it on a regular
basis.</p>
<p>It's also possible to see the GoAccess report straight in the terminal, thanks
to <code>libncurses</code>, rather than creating a HTML report.</p>
<p>And much more, GoAccess is <a href="https://goaccess.io/features">packed with features</a>.</p>
<h2>Conclusion</h2>
<p>I hope that this tutorial helped some of you folks. Feel free to drop an e-mail
for comments.</p>Building your Pelican website with schroot2019-03-16T00:00:00+00:002019-03-16T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2019-03-16:/2019/03/16/building-your-pelican-website-with-schroot/<p>Lately I moved my blog to Pelican. I really like how simple and flexible it is.
So in this post I'd like to highlight one particular aspect of my Pelican's
workflow: how to setup a Debian-based environment to build your Pelican's
website, and how to leverage Pelican's Makefile to transparently use this build
environment. Overall, this post has more to do with the Debian tooling, and
little with Pelican.</p>
<p>Lately I moved my blog to Pelican. I really like how simple and flexible it is.
So in this post I'd like to highlight one particular aspect of my Pelican's
workflow: how to setup a Debian-based environment to build your Pelican's
website, and how to leverage Pelican's Makefile to transparently use this build
environment. Overall, this post has more to do with the Debian tooling, and
little with Pelican.</p>
<h2>Introduction</h2>
<p>First thing first, why would you setup a build environment for your project?</p>
<p>Imagine that you run Debian <em>stable</em> on your machine, then you want to build
your website with a fancy theme, that requires the latest bleeding edge
features from Pelican. But hey, in Debian stable you don't have these shiny new
things, and the version of Pelican you need is only available in Debian
<em>unstable</em>. How do you handle that? Will you start messing around with apt
configuration and pinning, and try to install an unstable package in your
stable system? Wrong, please stop.</p>
<p>Another scenario, the opposite: you run Debian unstable on your system. You
have all the new shiny things, but sometimes an update of your system might
break things. What if you update, and then can't build your website because of
this or that? Will you wait a few days until another update comes and fixes
everything? How many days before you can build your blog again? Or will you
dive in the issues, and debug, which is nice and fun, but can also keep you
busy all night, and is not exactly what you wanted to do in the first place,
right?</p>
<p>So, for both of these issues, there's one simple answer: setup a build
environment for your project. The most simple way is to use a <em>chroot</em>, which
is roughly another filesystem hierarchy that you create and install somewhere,
and in which you will run your build process. A more elaborate build
environment is a <em>container</em>, which brings more isolation from the host system,
and many more features, but for something as simple as building your website on
your own machine, it's kind of overkill.</p>
<p>So that's what I want to detail here, I'll show you the way to setup and use a
chroot. There are many tools for the job, and for example Pelican's official
documentation recommends <a href="https://docs.getpelican.com/en/stable/install.html">virtualenv</a>, which is kind of the standard Python
solution for that. However, I'm not too much of a Python developer, and I'm
more familiar with the Debian tools, so I'll show you the Debian way instead.</p>
<p>Version-wise, it's 2019, we're talking about Pelican 4.x, if ever it matters.</p>
<h2>Create the chroot</h2>
<p>To create a basic, minimal Debian system, the usual command is <a href="https://manpages.debian.org/unstable/debootstrap/debootstrap.8.en.html">debootstrap</a>.
Then in order to actually use this new system, we'll use <a href="https://manpages.debian.org/unstable/schroot/schroot.1.en.html">schroot</a>. So be
sure to have these two packages installed on your machine.</p>
<div class="highlight"><pre><span></span><code>sudo apt install debootstrap schroot
</code></pre></div>
<p>It seems that the standard location for chroots is <code>/srv/chroot</code>, so let's
create our chroot there. It also seems that the traditional naming scheme for
these chroots is something like <code>SUITE-ARCH-APPLICATION</code>, at least that's what
other tools like <a href="https://manpages.debian.org/unstable/sbuild/sbuild.1.en.html">sbuild</a> do. While you're free to do whatever you want, in
this tutorial we'll try to stick to the conventions.</p>
<p>Let's go and create a <code>bookworm</code> chroot:</p>
<div class="highlight"><pre><span></span><code><span class="n">SYSROOT</span><span class="o">=/</span><span class="n">srv</span><span class="o">/</span><span class="n">chroot</span><span class="o">/</span><span class="n">bookworm</span><span class="o">-</span><span class="n">amd64</span><span class="o">-</span><span class="n">pelican</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">$</span><span class="p">{</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="p">}</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">debootstrap</span><span class="w"> </span><span class="o">--</span><span class="n">variant</span><span class="o">=</span><span class="n">minbase</span><span class="w"> </span><span class="n">bookworm</span><span class="w"> </span><span class="o">$</span><span class="p">{</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="p">}</span>
</code></pre></div>
<p>And there we are, we just installed a minimal Debian system in <code>$SYSROOT</code>, how
easy and neat is that! Just run a quick <code>ls</code> there, and see by yourself:</p>
<div class="highlight"><pre><span></span><code>ls<span class="w"> </span><span class="cp">${</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="cp">}</span>
</code></pre></div>
<p>Now let's setup schroot to be able to use it. schroot will require a bit of a
configuration file that tells it how to use this chroot. This is where things
might get a bit complicated and cryptic for the newcomer.</p>
<p>So for now, stick with me, and create the schroot config file as follow:</p>
<div class="highlight"><pre><span></span><code>cat << EOF | sudo tee /etc/schroot/chroot.d/bookworm-amd64-pelican.conf
[bookworm-amd64-pelican]
users=$LOGNAME
root-users=$LOGNAME
source-users=$LOGNAME
source-root-users=$LOGNAME
type=directory
union-type=overlay
directory=/srv/chroot/bookworm-amd64-pelican
EOF
</code></pre></div>
<p>Here, we tell schroot who can use this chroot (<code>$LOGNAME</code>, it's you), as normal
user and root user. We also say where is the chroot directory located, and that
we want an <code>overlay</code>, which means that the chroot will actually be read-only,
and during operation a writable overlay will be stacked up on top of it, so
that modifications are possible, but are not saved when you exit the chroot.</p>
<p>In our case, it makes sense because we have no intention to modify the build
environment. The basic idea with a build environment is that it's identical for
every build, we don't want anything to change, we hate surprises. So we make it
read-only, but we also need a writable overlay on top of it, in case some
process might want to write things in <code>/var</code>, for example. We don't care about
these changes, so we're fine discarding this data after each build, when we
leave the chroot.</p>
<p>And now, for the last step, let's install Pelican in our chroot:</p>
<div class="highlight"><pre><span></span><code>schroot -c source:bookworm-amd64-pelican -u root -- \
bash -c "apt update && apt install --yes make pelican && apt clean"
</code></pre></div>
<p>In this command, we log into the source chroot as root, and we install the two
packages <code>make</code> and <code>pelican</code>. We also clean up after ourselves, to save a bit
of space on the disk.</p>
<p>At this point, our chroot is ready to be used. If you're new to all of this,
then read the next section, I'll try to explain a bit more how it works.</p>
<h2>A quick introduction to schroot</h2>
<p>In this part, let me try to explain a bit more how schroot works. If you're
already acquainted, you can skip this part.</p>
<p>So now that the chroot is ready, let's experiment a bit. For example, you might
want to start by listing the chroots available:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-l
chroot:bookworm-amd64-pelican
source:bookworm-amd64-pelican
</code></pre></div>
<p>Interestingly, there are two of them... So, this is due to the overlay thing
that I mentioned just above. Using the <em>regular chroot</em> (<code>chroot:</code>) gives you
the read-only version, for daily use, while the <em>source chroot</em> (<code>source:</code>)
allows you to make persistent modifications to the filesystem, for install and
maintenance basically. In effect, the <em>source chroot</em> has no overlay mounted on
top of it, and is writable.</p>
<p>So you can experiment some more. For example, to have a shell into your <em>regular
chroot</em>, run:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>chroot:bookworm-amd64-pelican
</code></pre></div>
<p>Notice that the <em>namespace</em> (eg. <code>chroot:</code> or <code>source:</code>) is optional, if you
omit it, schroot will be smart and choose the right namespace. So the command
above is equivalent to:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>bookworm-amd64-pelican
</code></pre></div>
<p>Let's try to see the overlay thing in action. For example, once inside the
chroot, you could create a file in some writable place of the filesystem.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">chroot</span><span class="p">)</span><span class="o">$</span><span class="w"> </span><span class="n">touch</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">this</span><span class="o">-</span><span class="k">is</span><span class="o">-</span><span class="n">an</span><span class="o">-</span><span class="n">empty</span><span class="o">-</span><span class="n">file</span>
<span class="p">(</span><span class="n">chroot</span><span class="p">)</span><span class="o">$</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">tmp</span>
<span class="n">this</span><span class="o">-</span><span class="k">is</span><span class="o">-</span><span class="n">an</span><span class="o">-</span><span class="n">empty</span><span class="o">-</span><span class="n">file</span>
</code></pre></div>
<p>Then log out with <code><Ctrl-D></code>, and log in again. Have a look in <code>/var/tmp</code>: the
file is gone. The overlay in action.</p>
<p>Now, there's a bit more to that. If you look into the current directory, you
will see that you're not within any isolated environment, you can still see all
your files, for example:</p>
<div class="highlight"><pre><span></span><code>(chroot)$ pwd
/home/arno/my-pelican-blog
(chroot)$ ls
content Makefile pelicanconf.py ...
</code></pre></div>
<p>Not only are all your files available in the chroot, you can also create new
files, delete existing ones, and so on. It doesn't even matter if you're inside
or outside the chroot, and the reason is simple: by default, schroot will mount
the <code>/home</code> directory inside the chroot, so that you can access all your files
transparently. For more details, just type <code>mount</code> inside the chroot, and see
what's listed.</p>
<p>So, this default of schroot is actually what makes it super convenient to use,
because you don't have to bother about bind-mounting every directory you care
about inside the chroot, which is actually quite annoying. Having <code>/home</code>
directly available saves time, because what you want to isolate are the tools
you need for the job (so basically <code>/usr</code>), but what you need is the data you
work with (which is supposedly in <code>/home</code>). And schroot gives you just that,
out of the box, without having to fiddle too much with the configuration.</p>
<p>If you're not familiar with chroots, containers, VMs, or more generally bind
mounts, maybe it's still very confusing. But you'd better get used to it, as
virtual environment are very standard in software development nowadays.</p>
<p>But anyway, let's get back to the topic. How to make use of this chroot to
build our Pelican website?</p>
<h2>Chroot usage with Pelican</h2>
<p>Pelican provides two helpers to build and manage your project: one is a
<code>Makefile</code>, and the other is a Python script called <code>fabfile.py</code>. As I said
before, I'm not really a seasoned Pythonista, but it happens that I'm quite a
fan of <a href="https://manpages.debian.org/unstable/make-guile/make.1.en.html">make</a>, hence I will focus on the Makefile for this part.</p>
<p>So, here's how your daily blogging workflow might look like, now that
everything is in place.</p>
<p>Open a first terminal, and edit your blog posts with your favorite editor:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>nano<span class="w"> </span>content/bla-bla-bla.md
</code></pre></div>
<p>Then open a second terminal, enter the chroot, build your blog and serve it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>bookworm-amd64-pelican
<span class="o">(</span>chroot<span class="o">)</span>$<span class="w"> </span>make<span class="w"> </span>html
<span class="o">(</span>chroot<span class="o">)</span>$<span class="w"> </span>make<span class="w"> </span>serve
</code></pre></div>
<p>And finally, open your web browser at <a href="http://localhost:8000">http://localhost:8000</a> and enjoy
yourself.</p>
<p>This is easy and neat, but guess what, we can even do better. Open the Makefile
and have a look at the very first lines:</p>
<div class="highlight"><pre><span></span><code>PY?=python3
PELICAN?=pelican
</code></pre></div>
<p>It turns out that the Pelican developers know how to write Makefiles, and they
were kind enough to allow their users to easily override the default commands.
In our case, it means that we can just replace these two lines with these ones:</p>
<div class="highlight"><pre><span></span><code>PY?=schroot -c bookworm-amd64-pelican -- python3
PELICAN?=schroot -c bookworm-amd64-pelican -- pelican
</code></pre></div>
<p>And after these changes, we can now completely forget about the chroot, and
simply type <code>make html</code> and <code>make serve</code>. The chroot invocation is now handled
automatically in the Makefile. How neat!</p>
<h2>Maintenance</h2>
<p>So you might want to update your chroot from time to time, and you do that with
<a href="https://manpages.debian.org/unstable/apt/apt.8.en.html">apt</a>, like for any Debian system. Remember the distinction between <em>regular
chroot</em> and <em>source chroot</em> due to the overlay? If you want to actually modify
your chroot, what you want is the <em>source chroot</em>. And here's the one-liner:</p>
<div class="highlight"><pre><span></span><code>schroot -c source:$PROJECT -u root -- \
bash -c "apt update && apt --yes dist-upgrade && apt clean"
</code></pre></div>
<p>If one day you stop using it, just delete the chroot directory, and the schroot
configuration file:</p>
<div class="highlight"><pre><span></span><code>sudo rm /etc/schroot/chroot.d/bookworm-amd64-pelican.conf
sudo rm -fr /srv/chroot/bookworm-amd64-pelican
</code></pre></div>
<p>And that's about it.</p>
<h2>Last words</h2>
<p>The general idea of keeping your build environments separated from your host
environment is a very important one if you're a software developer, and
especially if you're doing consulting and working on several projects at the
same time. Installing all the build tools and dependencies directly on your
system can work at the beginning, but it won't get you very far.</p>
<p><code>schroot</code> is only one of the many tools that exist to address this, and I
<em>think</em> it's Debian specific. Maybe you have never heard of it, as chroots in
general are far from the container hype, even though they have some common
use-cases. schroot has been around for a while, it works great, it's simple,
flexible, what else? Just give it a try!</p>
<p>It's also well integrated with other Debian tools, for example you might use it
through <code>sbuild</code> to build Debian packages (another daily task that is better
done in a dedicated build environment), so I think it's a tool worth knowing if
you're doing some Debian work.</p>
<p>That's about it, in the end it was mostly a post about schroot, hope you liked
it.</p>The docker.io Debian package is back to life2018-07-04T00:00:00+00:002018-07-04T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2018-07-04:/2018/07/04/the-dockerio-debian-package-is-back-to-life/<p>Last week, a new version of <code>docker.io</code>, the Docker package provided by Debian,
was uploaded to <a href="https://packages.debian.org/sid/docker.io">Debian Unstable</a>.
Quickly afterward, the package moved to <a href="https://packages.debian.org/testing/docker.io">Debian Testing</a>.
And this is good news for Debian users, as before that the package was more
or less abandoned in "unstable", and the future …</p><p>Last week, a new version of <code>docker.io</code>, the Docker package provided by Debian,
was uploaded to <a href="https://packages.debian.org/sid/docker.io">Debian Unstable</a>.
Quickly afterward, the package moved to <a href="https://packages.debian.org/testing/docker.io">Debian Testing</a>.
And this is good news for Debian users, as before that the package was more
or less abandoned in "unstable", and the future was uncertain.</p>
<p>The most striking fact about this change: it's the <em>first time in two years</em> that
<em>docker.io</em> has migrated to "testing". Another interesting fact is that, version-wise,
the package is moving from <code>1.13.1</code> from early 2017 to version <code>18.03</code> from
March 2018: that's a <em>one-year leap forward</em>.</p>
<p>Let me give you a very rough summary of how things came to be. I personally
started to work on that early in 2018. I joined the <em>Debian Go Packaging Team</em>
and I started to work on the many, many Docker dependencies that needed to be
updated in order to update the Docker package itself. I could get some of this
work uploaded to Debian, but ultimately I was a bit stuck on how to solve the
circular dependencies that plague the Docker package. This is where another
Debian Developer, Dmitry Smirnov, jumped in. We discussed the current status
and issues, and then he basically did all the job, from updating the package to
tackling all the long-time opened bugs.</p>
<p>This is for the short story, let me know give you some more details.</p>
<h2>The Docker package in Debian</h2>
<p>To better understand why this update of the <code>docker.io</code> package is such a good
news, let's have quick look at the current Debian offer:</p>
<div class="highlight"><pre><span></span><code>rmadison -u debian docker.io
</code></pre></div>
<p>If you're running Debian 8 Jessie, you can install Docker 1.6.2, through
backports. This version was released on May 14, 2015. That's 3 years old, but
Debian Jessie is fairly old as well.</p>
<p>If you're running Debian 9 Stretch (ie. Debian stable), then you have no
install candidate. No-thing. The current Debian doesn't provide any package for
Docker. That's a bit sad.</p>
<p>What's even more sad is that for quite a while, looking into Debian unstable
didn't look promising either. There used to be a package there, but it had bugs
that prevented it to migrate to Debian testing. This package was stuck at the
version <code>1.13.1</code>, released on Feb 8, 2017. Looking at the git history, there
was not much happening.</p>
<p>As for the reason for this sad state of things, I can only guess. Packaging
Docker is a tedious work, mainly due to a very big dependency tree. After
handling all these dependencies, there are other issues to tackle, some related
to Go packaging itself, and others due to Docker release process and
development workflow. In the end, it's quite difficult to find the right
approach to package Docker, and it's easy to make mistakes that cost hours of
works. I did this kind of mistakes. More than once.</p>
<p>So packaging Docker is not for the faint of heart, and maybe it's too much of
a burden for one developer alone. There was a <code>docker-maint</code> mailing list that
suggests an attempt to coordinate the effort, however this list was already dead
by the time I found it. It looks like the people involved walked away.</p>
<p>Another explanation for the disinterest in the Docker package could be that
Docker itself already provides a Debian package on docker.com. One can
always fall back to this solution, so why bothering with the extra-work of
doing a Debian package proper?</p>
<p>That's what the next part is about!</p>
<h2>Docker.io vs Docker-ce</h2>
<p>You have two options to install Docker on Debian: you can get the package from
docker.com (this package is named <code>docker-ce</code>), or you can get it from the
Debian repositories (this package is named <code>docker.io</code>). You can rebuild both
of these packages from source: for <em>docker-ce</em> you can fetch the source code
with git (it includes the packaging files), and for <em>docker.io</em> you can just
get the source package with <em>apt</em>, like for every other Debian package.</p>
<p>So what's the difference between these two packages?</p>
<p>No suspense, straight answer: what differs is the build process, and mostly,
the way dependencies are handled.</p>
<p>Docker is written in Go, and Golang comes with some tooling that allows
applications to keep a local copy of their dependencies in their source tree.
In Go-talk, this is called <em>vendoring</em>. Docker makes heavy use of that (like
many other Go applications), which means that the code is more or less
self-contained. You can build Docker without having to solve external
dependencies, as everything needed is already in-tree.</p>
<p>That's how the <code>docker-ce</code> package provided by Docker is built, and that's what
makes the packaging files for this package trivial. You can look at these files
at <a href="https://github.com/docker/docker-ce/tree/master/components/packaging/deb">https://github.com/docker/docker-ce/tree/master/components/packaging/deb</a>.
So everything is in-tree, there's almost no external build dependency, and
hence it's real easy for Docker to provide a new package for <code>docker-ce</code> every
month.</p>
<p>On the other hand, the <code>docker.io</code> package provided by Debian takes a
completely different approach: Docker is built against the libraries that are
packaged in Debian, instead of using the local copies that are present in the
Docker source tree. So if Docker is using <em>libABC</em> version 1.0, then it has a
build dependency on <em>libABC</em>. You can have a look at the current build
dependencies at <a href="https://salsa.debian.org/docker-team/docker/blob/master/debian/control">https://salsa.debian.org/docker-team/docker/blob/master/debian/control</a>.</p>
<p>There are more than 100 dependencies there, and that's one reason why the
Debian package is quite time-consuming to maintain. To give you a rough
estimation, in order to get the current "stable" release of Docker to Debian
"unstable", it took up to 40 uploads of related packages to stabilize the
dependency tree.</p>
<p>It's quite an effort. And once again, why bother? For this part I'll quote
Dmitry as he puts it better than me:</p>
<blockquote>
<p>Debian cares about reusable libraries, and packaging them individually allows to
build software from tested components, as Golang runs no tests for vendored
libraries. It is a mind blowing argument given that perhaps there is more code
in "vendor" than in the source tree.</p>
<p>Private vendoring have all disadvantages of <a href="https://wiki.debian.org/StaticLinking">static linking</a>,
making it impossible to provide meaningful security support. On top of that, it
is easy to lose control of vendored tree; it is difficult to track changes in
vendored dependencies and there is no incentive to upgrade vendored components.</p>
</blockquote>
<p>That's about it, whether it matters is up to you and your use-case. But it's
definitely something you should know about if you want to make an informed
decision on which package you're about to install and use.</p>
<p>To finish with this article, I'd like to give more details on the packaging of
<em>docker.io</em>, and what was done to get this new version in Debian.</p>
<h2>Under the hood of the docker.io package</h2>
<p>Let's have a brief overview of the difficulties we had to tackle while packaging
this new version of Docker.</p>
<p>The most outstanding one is circular dependencies. It's especially present in
the top-level dependencies of Docker: <code>docker/swarmkit</code>, <code>docker/libnetwork</code>,
<code>containerd</code>... All of these are Docker build dependencies, and all of these
depend on Docker to build. Good luck with that ;)</p>
<p>To solve this issue, the new <em>docker.io</em> package leverages MUT (Multiple
Upstream Tarball) to have these different components downloaded and built all
at once, instead of being packaged separately. In this particular case it
definitely makes sense, as we're really talking about different parts of
Docker. Even if they live in different git repositories, these components are
not standalone libraries, and there's absolutely no good reason to package them
separately.</p>
<p>Another issue with Docker is "micro-packaging", ie. wasting time packaging
small git repositories that, in the end, are only used by one application
(Docker in our case). This issue is quite interesting, really. Let me try to
explain.</p>
<p>Golang makes it extremely easy to split a codebase among several git
repositories. It's so easy that some projects (Docker in our case) do it
extensively, as part of their daily workflow. And in the end, at a first glance
you can't really say if a dependency of Docker is really a standalone project
(that would require a proper packaging), or only just a part of Docker
codebase, that happens to live in a different git repository. In this second
case, there's really no reason to package it independently of Docker.</p>
<p>As a packager, if you're not a bit careful, you can easily fall in this trap,
and start packaging every single dependency without thinking: that's
"micro-packaging". It's bad in the sense that it increases the maintenance cost
on the long-run, and doesn't bring any benefit. As I said before, <em>docker.io</em> has
currently 100+ dependencies, and probably a few of them fall in this category.</p>
<p>While working on this new version of <em>docker.io</em>, we decided to stop packaging
such dependencies. The guideline is that if a dependency has no <a href="https://semver.org/">semantic
versioning</a>, and no consumer other than Docker, then it's
not a library, it's just a part of Docker codebase.</p>
<p>Even though some tools like <a href="https://people.debian.org/%7Estapelberg/2015/07/27/dh-make-golang.html">dh-make-golang</a>
make it very easy to package simple Go packages, it doesn't mean that everything
should be packaged. Understanding that, and taking a bit of time to think before
packaging, is the key to successful Go packaging!</p>
<h2>Last words</h2>
<p>I could go on for a while on the technical details, there's a lot to say, but
let's not bore you to death, so that's it. I hope by now you understand that:</p>
<ol>
<li>There's now an up-to-date <code>docker.io</code> package in Debian.</li>
<li><code>docker.io</code> and <code>docker-ce</code> both give you a Docker binary, but through a
very different build process.</li>
<li>Maintaining the <code>docker.io</code> package is not an easy task.</li>
</ol>
<p>If you care about having a Docker package in Debian, feel free to try it out,
and feel free to join the maintenance effort!</p>
<p>Let's finish with a few credits. I've been working on that topic, albeit
sparingly, for the last 4 months, thanks to the support of
<a href="https://www.collabora.com/">Collabora</a>. As for Dmitry Smirnov, the work he did
on the <em>docker.io</em> package represents a three weeks, full-time effort, which
was sponsored by <a href="https://raid6.com.au">Libre Solutions Pty Ltd</a>.</p>
<p>I'd like to thank the <a href="https://go-team.pages.debian.net/">Debian Go Packaging
Team</a> for their support, and also the
reviewers of this article, namely Dmitry Smirnov and Héctor Orón Martínez.</p>
<p>Last but not least, I will attend <a href="https://debconf18.debconf.org/">DebConf18</a>
in Taiwan, where I will give a speak on this topic. There's also a BoF on Go
Packaging planned.</p>
<p>See you there!</p>GRUB Recovery for UEFI-GPT-LUKS-LVM2018-03-26T00:00:00+00:002018-03-26T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2018-03-26:/2018/03/26/grub-recovery-for-uefi-gpt-luks-lvm/<p>Yet another GRUB recovery article... Yep, I can't deny, that's what it's about!</p>
<p>Yet another GRUB recovery article... Yep, I can't deny, that's what it's about!</p>
<h2>Introduction</h2>
<p><a href="https://www.gnu.org/software/grub/">GRUB</a> is a bootloader, ie. a piece of
software that gets loaded very early on when you boot your machine, and which
is in charge of booting the Operating System. GRUB is the bootloader of choice
for many Linux distributions.</p>
<p>From time to time, it happens that the poor GRUB is stepped over by some very
rude operating systems. In my case, I needed the infamous Windows, and as I didn't
want to install it, I googled my way to a tutorial that explained how to
create a bootable USB stick with Windows. Great, I thought, and it worked!
Except that after doing my things in Windows, I restarted and found that the
machine couldn't boot anymore. GRUB had been wiped away, silently, without
any warning, by the infamous Windows...</p>
<p>If you find yourself in a similar situation, fear not! We can always boot a
<a href="https://en.wikipedia.org/wiki/Live_CD">live system</a> from a USB stick, and
repair the boot partition (ie. re-install GRUB) from there. It's fairly easy,
it's just a matter of knowing the right commands.</p>
<p>There's already many tutorial of this kind available, each one slightly
different from each other depending on the particular hardware and software
details of the person writing the article. And same goes with this article, so
let me tell you immediately about my configuration:</p>
<ul>
<li>We're talking about recovering GRUB with a GParted Live CD.</li>
<li>This happens on a UEFI machine.</li>
<li>The disk is partitioned with GPT.</li>
<li>The disk happens to be a SSD.</li>
<li>The whole disk is encrypted with LUKS.</li>
<li>The LUKS partition is "partitioned" using LVM.</li>
</ul>
<p>If you have no idea what I'm talking about, you're probably on the wrong page.
But if you understand what I mean, and even if your configuration is a bit
different, read on. I'll do my best to explain what's going on so that you
can adapt these commands to your particular setup.</p>
<p><strong>A word of caution though</strong>: we're dealing with the disks and filesystems here,
so if you screw up you can lose data. I will just assume that you understand
what's going on, and that you're able to adapt the commands below to your own
setup. Mostly, it's about changing the device names here and there, and I can't
do that for you.</p>
<h2>At first, let's understand how things work</h2>
<p>So let's get started with some explanations. If you already know how things
work and just want to copy/paste some commands, skip this part. But if you
want to understand a bit what we're doing here, read on.</p>
<p>The commands in this part are meant to be run when your machine is up and
running, the purpose it to look at how things are setup. So of course, if right
now your machine can't boot, you can't run these commands. I mean, don't run it
from a live system, it won't give you the expected result.</p>
<h4>Overview of the disk partitions</h4>
<p>Let's have an overview of the partitions present on the disk, the best command
for that being <code>lsblk</code>. On my laptop, here's the output:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">lsblk</span>
<span class="n">NAME</span><span class="w"> </span><span class="nl">MAJ</span><span class="p">:</span><span class="nf">MIN</span><span class="w"> </span><span class="n">RM</span><span class="w"> </span><span class="k">SIZE</span><span class="w"> </span><span class="n">RO</span><span class="w"> </span><span class="n">TYPE</span><span class="w"> </span><span class="n">MOUNTPOINT</span>
<span class="n">nvme0n1</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">477</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">disk</span><span class="w"> </span>
<span class="err">├─</span><span class="n">nvme0n1p1</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">512</span><span class="n">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span><span class="o">/</span><span class="n">boot</span><span class="o">/</span><span class="n">efi</span>
<span class="err">├─</span><span class="n">nvme0n1p2</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">244</span><span class="n">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span><span class="o">/</span><span class="n">boot</span>
<span class="err">└─</span><span class="n">nvme0n1p3</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">3</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">476.2</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span>
<span class="w"> </span><span class="err">└─</span><span class="n">nvme0n1p3_crypt</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">476.2</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">crypt</span><span class="w"> </span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">root</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">23.3</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="nf">var</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">40</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span><span class="nf">var</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">swap_1</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">3</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">15.9</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">[</span><span class="n">SWAP</span><span class="o">]</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">tmp</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">4</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">10</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span>
<span class="w"> </span><span class="err">└─</span><span class="n">debian</span><span class="c1">--vg-home 253:5 0 385.9G 0 lvm /home</span>
</code></pre></div>
<p>As I explained above, I have an UEFI machine, the hard drive is a SSD,
partitioned with GPT, and the disk is encrypted. You can see all of that
above, ie:</p>
<ul>
<li><code>nvme0n1</code> is what you get if you have a <a href="https://en.wikipedia.org/wiki/Solid-state_drive">SSD</a>
drive. For comparison, a more ancient <a href="https://en.wikipedia.org/wiki/Hard_disk_drive">HDD</a>
would be named something like <code>sda</code>. <code>nvme</code> stands for Non-Volatile Memory
Express (roughly), and <code>n1</code> stands for disk number one. Additionally,
the suffix <code>p1</code>, <code>p2</code> and so on is appended to give the partition number.</li>
<li><code>nvme0n1p1</code> and <code>nvme0n1p2</code> are the two partitions involved in the boot
process.</li>
<li><code>nvme0n1p3</code> is the <a href="https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup">LUKS</a>
encrypted partition that takes up the whole disk. When the partition is
unlocked, it is available unencrypted at <code>nvme0n1p3_crypt</code>, thanks to the
<a href="https://en.wikipedia.org/wiki/Device_mapper">Linux device mapper</a>.</li>
<li>Additionally, all the <code>debian--</code> devices are the different logical volumes
presents on the partition. If you're familiar with Linux distributions, you
can recognize the usual partitions <code>/</code>, <code>/home</code> and the swap partition. I
also decided to have a separate <code>/tmp</code> and <code>/var</code> partitions, but that's a
matter of taste.</li>
</ul>
<p>The logical volumes are not exactly partitions the way we know it. Ultimately,
they appear to the user as <em>block devices</em>, however they are <em>virtual block devices</em>,
you won't find them directly in <code>/dev</code>, but instead in the sub-directory
<code>/dev/mapper</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>/dev/mapper/
control
nvme0n1p3_crypt
debian--vg-home
debian--vg-root
debian--vg-swap_1
debian--vg-tmp
debian--vg-var
</code></pre></div>
<p>In case you wonder, such layout is nothing fancy. It's what you get with the
Debian Buster installer, assuming that you select the whole disk encryption
during the install process.</p>
<p>Ok, let's dive a bit more into details.</p>
<h4>The boot partition</h4>
<p>There are two partitions involved in the boot process.</p>
<p>The first partition of the disk, <code>nvme0n1p1</code> is the <a href="https://en.wikipedia.org/wiki/EFI_system_partition">EFI System Partition</a>.
At boot time, the UEFI firmware loads this partition and will boot your system
from there. In our case, this is where GRUB lives.</p>
<p>On a Debian system, this partition is usually mounted at <code>/boot/efi</code>, and you
can have a look at what's inside.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>tree<span class="w"> </span>/boot/efi
/boot/efi
└──<span class="w"> </span>EFI
<span class="w"> </span>├──<span class="w"> </span>debian
<span class="w"> </span>│<span class="w"> </span>└──<span class="w"> </span>grubx64.efi
...
</code></pre></div>
<p>Indeed, GRUB lives there, among other things.</p>
<h4>The kernel and initrd partition</h4>
<p>I mentioned that my disk is encrypted, right? It means that at boot time, I
need to enter my password in order to decrypt the operating system. The
decryption can be implemented in different places, and actually recent versions
of GRUB can handle that. However, in the setup I describe here, this is not the
case. GRUB will just boot and <em>unencrypted</em> kernel, and it's the kernel who's in
charge of decrypting the encrypted partition.</p>
<p>It means that we need an unencrypted partition somewhere to store the kernel
and the initrd: this is the purpose of the second partition: <code>nvme0n1p2</code>.</p>
<p>On a Debian system, this partition is usually mounted at <code>/boot</code>. Once again,
the best is to have a look and convince yourself.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>/boot
config-4.13.0-1-amd64
config-4.14.0-3-amd64
efi
grub
initrd.img-4.13.0-1-amd64
initrd.img-4.14.0-3-amd64
lost+found
System.map-4.13.0-1-amd64
System.map-4.14.0-3-amd64
vmlinuz-4.13.0-1-amd64
vmlinuz-4.14.0-3-amd64
</code></pre></div>
<p><code>vmlinuz</code> is the usual filename for the Linux kernel, while <code>initrd.img</code> is the
initial ramdisk, ie. a temporary, minimal root filesystem that gets loaded by
the kernel at first, and perform some initial setup before mounting your "real"
root filesystem.</p>
<h4>Some more commands</h4>
<p>Another good command to have a hierachical view of the mount points is <code>findmnt</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>findmnt
...
├─/boot<span class="w"> </span>/dev/nvme0n1p2<span class="w"> </span>ext2<span class="w"> </span>rw,relatime,...
│<span class="w"> </span>└─/boot/efi<span class="w"> </span>/dev/nvme0n1p1<span class="w"> </span>vfat<span class="w"> </span>rw,relatime,...
...
</code></pre></div>
<p>The output is interesting, because it shows clearly how the <code>efi</code> mount point
is nested inside the <code>/boot</code> mount point. This is interesting because during
the recovery process we will have to reproduce this layout.</p>
<p>At last, you can also have some useful information with <code>fdisk</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>fdisk<span class="w"> </span>-l
Disk<span class="w"> </span>/dev/nvme0n1:<span class="w"> </span><span class="m">477</span><span class="w"> </span>GiB,<span class="w"> </span><span class="m">512110190592</span><span class="w"> </span>bytes,<span class="w"> </span><span class="m">1000215216</span><span class="w"> </span>sectors
Units:<span class="w"> </span>...
Sector<span class="w"> </span>size<span class="w"> </span>...
I/O<span class="w"> </span>size<span class="w"> </span>...
Disklabel<span class="w"> </span>type:<span class="w"> </span>gpt
Disk<span class="w"> </span>identifier:<span class="w"> </span>...
Device<span class="w"> </span>Start<span class="w"> </span>End<span class="w"> </span>Sectors<span class="w"> </span>Size<span class="w"> </span>Type
/dev/nvme0n1p1<span class="w"> </span><span class="m">2048</span><span class="w"> </span><span class="m">1050623</span><span class="w"> </span><span class="m">1048576</span><span class="w"> </span>512M<span class="w"> </span>EFI<span class="w"> </span>System
/dev/nvme0n1p2<span class="w"> </span><span class="m">1050624</span><span class="w"> </span><span class="m">1550335</span><span class="w"> </span><span class="m">499712</span><span class="w"> </span>244M<span class="w"> </span>Linux<span class="w"> </span>filesystem
/dev/nvme0n1p3<span class="w"> </span><span class="m">1550336</span><span class="w"> </span><span class="m">1000214527</span><span class="w"> </span><span class="m">998664192</span><span class="w"> </span><span class="m">476</span>.2G<span class="w"> </span>Linux<span class="w"> </span>filesystem
...
</code></pre></div>
<p>One interesting detail here is the <code>Disklabel type</code>, which should be <code>gpt</code> if your
disk is partitioned with <a href="https://en.wikipedia.org/wiki/GUID_Partition_Table">GPT</a>,
or <code>dos</code> otherwise.</p>
<p>Ok, I hope you get a good overview of things now, time to get started with the
real thing: recovering GRUB.</p>
<h2>Get yourself a GParted Live CD</h2>
<p>Let's get started with the recovery process!</p>
<p>The first thing to do is to prepare a USB stick (or a CD-ROM if you're a bit
old-fashioned) with a live system of some kind. I usually go with a
<a href="https://gparted.org/livecd.php">GParted Live CD/USB</a>
for this task, as it's been around forever and always did the job. But there
are other alternatives, and you might want to give a try to
<a href="http://www.rodsbooks.com/refind">ReFind</a> if you want to try out new stuff.</p>
<p>So, let's visit <a href="https://gparted.org/download.php">https://gparted.org/download.php</a>, download an iso, and install
that on a USB stick. I won't cover these details, it's nothing complicated and
there's plenty of explanation available on the GParted website already.</p>
<p>When you're done, plug the USB stick, reboot your machine, get a boot prompt,
and then choose to boot from the USB stick.</p>
<p><strong>Be sure that your machine is configured to boot in EFI mode</strong>, and not in
legacy BIOS mode. If I'm not mistaken, GRUB will detect that later on, and will
install itself accordingly. So if you boot in legacy BIOS mode, then grub will
install itself the legacy way, and you will not be able to boot it through UEFI.</p>
<h2>Your first steps in GParted</h2>
<p>The desktop looks a bit old fashioned, it's not super pretty, but that's not
the point. The point is that GParted comes furnished with all the tools you
need to deal with disks and filesystems, all of that up-to-date and well
maintained.</p>
<p>There are a few icons on the desktop, one is about changing the display
resolution, and it can be very useful. Other than that, just right click
somewhere on the desktop, and choose to open a root terminal.</p>
<p>And that's pretty much all we need to know.</p>
<h2>The recovery process</h2>
<p>Now, it's just a matter of <em>chrooting</em> properly.
<a href="https://en.wikipedia.org/wiki/Chroot">chroot</a> (literally "change root") is the
magic of executing a process in another root directory. Right now, if you
type <code>ls /</code> in your terminal, you will see the current root directory of the
system. And which system? The live system, the one you booted from the USB
stick.</p>
<p>However, in order to recover GRUB, we will need to run some commands in the root
directory of your machine, the one you can't boot at the moment. The reason is
that GRUB needs to access resources in several places of this hierarchy. So the
right way to achieve that is to chroot. And we need a bit of work to prepare
the environment in which we will chroot, it's a bit more complicated than just
doing <code>cd</code> somewhere, or just remounting a directory.</p>
<p>So, yep, that's where all the difficulty is, simply because it's the kind of
commands we never have to run in the daily life. And there are different
elements in the picture, we will have to deal with the encryption, and the
logical volumes, and all the rest.</p>
<p>So, let me guide you in the process.</p>
<h4>Mapping every devices</h4>
<p>Please remember that I told you to open a <em>root</em> terminal!</p>
<div class="highlight"><pre><span></span><code># whoami
root
</code></pre></div>
<p>A lot of magic will happen in <code>/dev/mapper</code>, so you might as well have a look
right now.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>Yep, right now it's more or less empty, but we will populate it.</p>
<p>The first thing we need to handle is the encryption: we need to unlock our
encrypted partition before we can work with it.</p>
<p>This is actually super easy, as long as you know the command :)</p>
<div class="highlight"><pre><span></span><code># cryptsetup luksOpen /dev/nvme0n1p3 cryptdisk
</code></pre></div>
<p>And have a look at <code>/dev/mapper</code> immediately to see what happened.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>Done, so from now on your encrypted partition is unlocked, let's move on to the
next step, which is to activate the LVM partitions. I'm not sure this is always
needed, but it doesn't hurt either.</p>
<p>Just enter these LVM commands.</p>
<div class="highlight"><pre><span></span><code># vgscan
# vgchange -ay
</code></pre></div>
<p>And once again, be curious, check what's up in <code>/dev/mapper</code>.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>As you can see, all your partitions are now visible there. That's great, we can
start to mount them now.</p>
<h4>Preparing the chroot</h4>
<p>We will prepare the chroot in <code>/mnt</code>. Right now, it's empty.</p>
<div class="highlight"><pre><span></span><code># ls /mnt
</code></pre></div>
<p>We need to mount the root partition first.</p>
<div class="highlight"><pre><span></span><code># mount /dev/mapper/debian--vg-root /mnt
# ls /mnt
</code></pre></div>
<p>If you read the explanations above, then you know that the boot partitions
are supposed to be mounted in <code>/boot</code>. Since one mount point is nested into
the other, the order for these commands matter.</p>
<div class="highlight"><pre><span></span><code># mount /dev/nvme0n1p2 /mnt/boot
# mount /dev/nvme0n1p1 /mnt/boot/efi
</code></pre></div>
<p>Additionally, other system partitions need to be mounted, if any. So if you
have a separate <code>/var</code> partition, it's time to mount it.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mount /dev/mapper/debian--vg-var /mnt/var</span>
</code></pre></div>
<p>We also need to bind mount all the system things, like the /dev hierarchy and
other pseudo filesystems. These are some special directories that contain some
runtime things that the system needs to operate properly.</p>
<div class="highlight"><pre><span></span><code># mount --bind /dev /mnt/dev
# mount --bind /dev/pts /mnt/dev/pts
# mount --bind /proc /mnt/proc
# mount --bind /run /mnt/run
# mount --bind /sys /mnt/sys
</code></pre></div>
<p>At last, we need to mount the EFI configurations variables, that are exposed by
the kernel via <em>efivarfs</em> (assuming that your systemd was booted in EFI mode).</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars</span>
</code></pre></div>
<p>If ever this line did not work, try:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># modprobe efivarfs</span>
<span class="c1"># mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars</span>
</code></pre></div>
<p>If it still doesn't work, maybe you simply didn't boot in EFI mode?</p>
<h4>Entering the chroot and recovering GRUB</h4>
<p>At this point, everything is ready! Just enter the chroot, ie. "change root
directory":</p>
<div class="highlight"><pre><span></span><code># chroot /mnt
</code></pre></div>
<p>As you noticed, your prompt changed. That's the indication that you are now in
a different environment.</p>
<p>It's now super easy. We want to re-install GRUB, and we just need two commands
for that.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">grub</span><span class="o">-</span><span class="nv">install</span><span class="w"> </span><span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">nvme0n1</span>
<span class="nv">Installing</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">x86_64</span><span class="o">-</span><span class="nv">efi</span><span class="w"> </span><span class="nv">platform</span>.
<span class="nv">Installation</span><span class="w"> </span><span class="nv">finished</span>.<span class="w"> </span><span class="nv">No</span><span class="w"> </span><span class="nv">error</span><span class="w"> </span><span class="nv">reported</span>.
#<span class="w"> </span><span class="nv">update</span><span class="o">-</span><span class="nv">grub</span>
<span class="nv">Generating</span><span class="w"> </span><span class="nv">grub</span><span class="w"> </span><span class="nv">configuration</span><span class="w"> </span><span class="nv">file</span>...
<span class="nv">Found</span><span class="w"> </span><span class="nv">background</span><span class="w"> </span><span class="nv">image</span>:<span class="w"> </span><span class="nv">blablabla</span>
<span class="nv">Found</span><span class="w"> </span><span class="nv">linux</span><span class="w"> </span><span class="nv">image</span>:<span class="w"> </span><span class="o">/</span><span class="nv">boot</span><span class="o">/</span><span class="nv">vmlinuz</span><span class="o">-</span><span class="mi">4</span>.<span class="mi">14</span>.<span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="nv">amd64</span>
...
<span class="nv">Adding</span><span class="w"> </span><span class="nv">boot</span><span class="w"> </span><span class="nv">menu</span><span class="w"> </span><span class="nv">entry</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">EFI</span><span class="w"> </span><span class="nv">firmware</span><span class="w"> </span><span class="nv">configuration</span>
<span class="nv">done</span>
</code></pre></div>
<p>If you see this warning along the way, fear not, it's harmless.</p>
<div class="highlight"><pre><span></span><code><span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">connect</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">lvmetad</span><span class="o">.</span><span class="w"> </span><span class="n">Falling</span><span class="w"> </span><span class="n">back</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">scanning</span><span class="o">.</span>
</code></pre></div>
<p>Done! Now you want to exit the chroot.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="k">exit</span>
</code></pre></div>
<p>See how your prompt changed? You're out of the chroot, back to the real world!
Time to reboot and see if GRUB is happy again.</p>
<div class="highlight"><pre><span></span><code># reboot
</code></pre></div>
<p>After that, I'm not even sure your system will boot ;) You might need to enter
your UEFI manager, and then manually add a boot entry: select the file
<code>bootx64.efi</code>. On my laptop, the UEFI manager is a bit dumb, I also need to
give a name when I create the entry, otherwise it fails with some unhelpful
error message.</p>
<h2>Thanks</h2>
<p>Here are some readings that made this article possible.</p>
<ul>
<li><a href="https://ubuntuforums.org/showthread.php?t=2266650">https://ubuntuforums.org/showthread.php?t=2266650</a></li>
<li><a href="https://wiki.debian.org/GrubEFIReinstall">https://wiki.debian.org/GrubEFIReinstall</a></li>
</ul>
<p>Edited in 2021, May to improve the chroot setup procedure, thanks to Oliver
Kollenberg.</p>Using GMail to send system mails on Debian2017-07-04T00:00:00+00:002017-07-04T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2017-07-04:/2017/07/04/using-gmail-to-send-system-mails-on-debian/<p>If you're a Debian user like me, chances are that, at some point, you will
need to send emails from the command-line. It could be on a Debian server,
where you want to send notifications (login, logout, boot, daily report, ...).
Or it could be on your own machine, where you use some tools like <code>reportbug</code>
or <code>git send-email</code>.</p>
<p>If you're a Debian user like me, chances are that, at some point, you will
need to send emails from the command-line. It could be on a Debian server,
where you want to send notifications (login, logout, boot, daily report, ...).
Or it could be on your own machine, where you use some tools like <code>reportbug</code>
or <code>git send-email</code>.</p>
<p>On this page we will go through the configuration needed to make it happen.
I assume that we will send the mails through <a href="https://mail.google.com">GMail</a>.
Then we'll go through the config steps of <a href="http://www.exim.org/">Exim4</a>, which
is the MTA (Mail Transfer Agent) usually installed on Debian.</p>
<p>At last, we will look at the config steps for <a href="http://msmtp.sourceforge.net/">msmtp</a>,
a SMTP client, which can be a lightweight alternative to Exim, if for some
reasons you want an alternative.</p>
<p><strong>UPDATE</strong>: This tutorial is outdated, please refer to the new tutorial:
<a href="https://arnaudr.io/2020/08/24/send-emails-from-your-terminal-with-msmtp/">https://arnaudr.io/2020/08/24/send-emails-from-your-terminal-with-msmtp/</a></p>
<h2>GMail setup</h2>
<p>On the GMail-side, there's a little bit to do.</p>
<p>First, you want to ensure that the <em>2-Step Verification</em> is enabled. For that
to happen, visit <a href="https://myaccount.google.com/security">https://myaccount.google.com/security</a> and enable the 2-Step
Verification feature.</p>
<p>This will require that you authorize all your devices once (mail client, phone
and so on), which can be a bit tedious, but it's only for once. And after it's
done, you're left with a more secured account, so it's not that bad, right?</p>
<p>Enabling the 2-Step Verification will unlock the feature we want: <em>App Passwords</em>.
Visit <a href="https://myaccount.google.com/apppasswords">https://myaccount.google.com/apppasswords</a>, and go generate a password
for your application. You should use it only for <em>one</em> application, and never use
it anywhere else. The idea is that for each application that will send emails
through your GMail account, you create a password. Simple, right?</p>
<p>There are real benefits with app passwords:</p>
<ul>
<li>you won't have to write down your real password in a configuration file.</li>
<li>gmail won't block your emails for security reasons (happens if you
send emails from different servers in different locations, for example).</li>
<li>you can change your GMail password without impacting applications.</li>
<li>you can revoke an app password anytime without impacting anything else.</li>
</ul>
<p>OK, once you've done the GMail setup, keep going.</p>
<h2>Exim</h2>
<p>If you're on a Debian system, it's likely that <code>exim</code> is already installed.
You can easily check that with <code>dpkg -l | grep exim</code>. If it's not there, then
install!</p>
<div class="highlight"><pre><span></span><code>apt-get install exim4 mailutils
</code></pre></div>
<p>Then, all you need is a little conf.</p>
<div class="highlight"><pre><span></span><code>dpkg-reconfigure exim4-config
</code></pre></div>
<p>And here's the answers I used (most of it is the default):</p>
<ul>
<li>General type of mail configuration: <code>mail sent by smarthost; no local mail</code></li>
<li>System mail name: <code>localhost</code></li>
<li>IP-addresses to listen on for incoming SMTP connections: <code>127.0.0.1 ; ::1</code></li>
<li>Other destinations for which mail is accepted: <em>leave blank</em></li>
<li>Visible domain name for local users: <code>localhost</code> </li>
<li><strong>IP address or host name of the outgoing smarthost: <code>smtp.gmail.com::587</code></strong></li>
<li>Keep number of DNS-queries minimal (Dial-on-Demand)? <code>No</code></li>
<li>Split configuration into small files? <code>No</code></li>
</ul>
<p>Now, it's time to store your password in plain text. This is where you're happy
to use an app password instead of your account password.</p>
<p>In case you're a bit lost, I'm talking here about an app password that you
have to generate from your Google account page, mentioned above.</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'*.google.com:YourAddress@gmail.com:YourAppPassword'</span><span class="w"> </span><span class="err">\</span>
<span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">exim4</span><span class="o">/</span><span class="n">passwd</span><span class="p">.</span><span class="n">client</span>
</code></pre></div>
<p>Then ensure the file has the right ownership and permissions:</p>
<div class="highlight"><pre><span></span><code>chown root:Debian-exim /etc/exim4/passwd.client
chmod 640 /etc/exim4/passwd.client
</code></pre></div>
<p>At last, restart exim:</p>
<div class="highlight"><pre><span></span><code>update-exim4.conf
systemctl restart exim4
</code></pre></div>
<p>If you made it up to here, it's time for a reward! Send yourself a little email
to see if everything is OK.</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'Time for coffee buddy'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">mail</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="ss">"Coffee break"</span><span class="w"> </span><span class="n">AnyAddress</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span>
</code></pre></div>
<h4>Aliases</h4>
<p>Aliases are used to redirect mails for local recipients. For example, if you wish
to associate an email address with <code>root</code>, add this kind of line to the file
<code>/etc/aliases</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="o">:</span><span class="w"> </span><span class="n">RootAddress</span><span class="err">@</span><span class="n">Wherever</span><span class="o">.</span><span class="na">com</span>
</code></pre></div>
<p>Now, sending an email to root is easier.</p>
<div class="highlight"><pre><span></span><code>echo 'Root are you there?' | mail -s "Root message" root
</code></pre></div>
<h4>Tips and Tricks</h4>
<p>On the GMail App Passwords page, you can see the last time a password was used.
If you don't see this date being bumped while you send an email, you know that
your mail didn't even reach this step.</p>
<p>You can also look at the "Send Mails" section of your GMail mailbox, your mails
should leave a trace there.</p>
<p>If the mail is send but doesn't seem to arrive, always check your Spam.</p>
<h4>References</h4>
<ul>
<li><code>man exim4-config_files</code></li>
<li><a href="https://wiki.debian.org/GmailAndExim4">Debian Wiki - Gmail and Exim4</a></li>
<li><a href="https://www.talk-about-it.ca/setting-up-exim4-with-gmail-and-2-factor-authentication">Setting up Exim4 with GMail and 2-factor Authentication</a></li>
</ul>
<h2>MSMTP</h2>
<p>Msmtp is lightweigth, very simple to install and configure. If you just want
to be able to send emails, it's a great alternative to Exim. Here's the way
to go.</p>
<p>Install is a one-liner as usual.</p>
<div class="highlight"><pre><span></span><code>apt-get install msmtp
</code></pre></div>
<p>Create the configuration for the current user.</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span>
<span class="err">#</span><span class="w"> </span><span class="k">Set</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="k">values</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="ow">all</span><span class="w"> </span><span class="n">following</span><span class="w"> </span><span class="n">accounts</span><span class="p">.</span>
<span class="n">defaults</span>
<span class="n">auth</span><span class="w"> </span><span class="k">on</span>
<span class="n">tls</span><span class="w"> </span><span class="k">on</span>
<span class="n">tls_trust_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">certs</span><span class="o">/</span><span class="n">ca</span><span class="o">-</span><span class="n">certificates</span><span class="p">.</span><span class="n">crt</span>
<span class="n">logfile</span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtp</span><span class="p">.</span><span class="nf">log</span>
<span class="err">#</span><span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="n">gmail</span><span class="w"> </span><span class="n">account</span>
<span class="n">account</span><span class="w"> </span><span class="n">gmail</span>
<span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">port</span><span class="w"> </span><span class="mi">587</span>
<span class="k">from</span><span class="w"> </span><span class="n">YourAddress</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="k">user</span><span class="w"> </span><span class="n">YourAddress</span>
<span class="n">password</span><span class="w"> </span><span class="n">YourAppPassword</span>
<span class="err">#</span><span class="w"> </span><span class="k">Set</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="n">account</span>
<span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">EOF</span>
</code></pre></div>
<p>Explicitly set the right permissions on the configuration file.</p>
<div class="highlight"><pre><span></span><code>chmod 600 ~/.msmtprc
</code></pre></div>
<p>Now is the time to test!</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="k">default</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">Recipient</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span>
<span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="n">YourAddress</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Cafe</span><span class="w"> </span><span class="n">Sua</span><span class="w"> </span><span class="n">Da</span>
<span class="n">Iced</span><span class="o">-</span><span class="n">coffee</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">condensed</span><span class="w"> </span><span class="n">milk</span>
<span class="n">EOF</span>
</code></pre></div>
<h4>MTA</h4>
<p>Maybe you don't want to explicitly use msmtp to send emails. Maybe you have
some applications that expect a regular MTA program to be installed, and
will try to use the standard command <code>sendmail</code> to send emails.</p>
<p>Good news, it's just a matter of symlinking msmtp, and there is a package
that does just that for you.</p>
<div class="highlight"><pre><span></span><code>apt-get install msmstp-mta
</code></pre></div>
<p>Now you can test one last time.</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sendmail</span><span class="w"> </span><span class="n">Recipient</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span><span class="w"> </span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Flat</span><span class="w"> </span><span class="n">White</span>
<span class="n">The</span><span class="w"> </span><span class="n">milky</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">making</span><span class="w"> </span><span class="n">coffee</span>
<span class="n">EOF</span>
</code></pre></div>
<h4>References</h4>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/Msmtp">https://wiki.archlinux.org/index.php/Msmtp</a></li>
</ul>