Arnaud Rhttps://arnaudr.io/2023-01-18T00:00:00+00:00Build container images in GitLab CI (iptables-legacy at the rescue)2023-01-18T00:00:00+00:002023-01-18T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2023-01-18:/2023/01/18/build-container-images-in-gitlab-ci-iptables-legacy-at-the-rescue/<p>It's 2023 and these days, building a container image in a CI pipeline should be
straightforward. So let's try.</p>
<p>It's 2023 and these days, building a container image in a CI pipeline should be
straightforward. So let's try.</p>
<p>For this blog post we'll focus on GitLab SaaS only, that is,
<a href="https://gitlab.com">gitlab.com</a>, as it's what I use for work and for personal
projects.</p>
<p>To get started, we just need two files in our Git repository:</p>
<ul>
<li>a <code>Containerfile</code> (or <code>Dockerfile</code> if you prefer to name it this way) that
defines how to build a container image.</li>
<li>a <code>.gitlab-ci.yml</code> file that defines what the CI should do. In the example
below, we want to build a container image and push it to the GitLab registry
associated with the GitLab repo.</li>
</ul>
<p>Here is our Git tree:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-A
Containerfile<span class="w"> </span>.git<span class="w"> </span>.gitlab-ci.yml
$<span class="w"> </span>cat<span class="w"> </span>Containerfile<span class="w"> </span>
FROM<span class="w"> </span>debian:stable
RUN<span class="w"> </span>apt-get<span class="w"> </span>update
CMD<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>hello<span class="w"> </span>world
$<span class="w"> </span>cat<span class="w"> </span>.gitlab-ci.yml<span class="w"> </span>
build-container-image:
<span class="w"> </span>stage:<span class="w"> </span>build
<span class="w"> </span>image:<span class="w"> </span>debian:testing
<span class="w"> </span>before_script:
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>update
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>buildah<span class="w"> </span>ca-certificates
<span class="w"> </span>script:
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>login<span class="w"> </span>-u<span class="w"> </span><span class="nv">$CI_REGISTRY_USER</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$CI_JOB_TOKEN</span><span class="w"> </span><span class="nv">$CI_REGISTRY</span>
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>push<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span>
</code></pre></div>
<p>A few remarks:</p>
<ul>
<li>We use <code>buildah</code>, but we could have used <code>podman</code>.</li>
<li>However we don't use <code>docker</code>, because its client/server design makes it
cumbersome to use in a CI environment: it requires a separate container to
run the Docker daemon, plus setting the <code>DOCKER_HOST</code> variable. Why bother?</li>
</ul>
<p>Now let's push that. Does the CI pass? No, of course, otherwise I wouldn't be
writing this blog post ;)</p>
<p>The CI fails at the <code>buildah build</code> command, with a rather cryptic error:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>--tag<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="o">[</span>...<span class="o">]</span>
STEP<span class="w"> </span><span class="m">2</span>/3:<span class="w"> </span>RUN<span class="w"> </span>apt-get<span class="w"> </span>update
error<span class="w"> </span>running<span class="w"> </span>container:<span class="w"> </span>did<span class="w"> </span>not<span class="w"> </span>get<span class="w"> </span>container<span class="w"> </span>start<span class="w"> </span>message<span class="w"> </span>from<span class="w"> </span>parent:<span class="w"> </span>EOF
Error:<span class="w"> </span>building<span class="w"> </span>at<span class="w"> </span>STEP<span class="w"> </span><span class="s2">"RUN apt-get update"</span>:<span class="w"> </span>netavark:<span class="w"> </span>code:<span class="w"> </span><span class="m">4</span>,<span class="w"> </span>msg:<span class="w"> </span>iptables<span class="w"> </span>v1.8.8<span class="w"> </span><span class="o">(</span>nf_tables<span class="o">)</span>:<span class="w"> </span>Could<span class="w"> </span>not<span class="w"> </span>fetch<span class="w"> </span>rule<span class="w"> </span><span class="nb">set</span><span class="w"> </span>generation<span class="w"> </span>id:<span class="w"> </span>Invalid<span class="w"> </span>argument
</code></pre></div>
<p>The hint here is <code>nf_tables</code>... Back in July 2021, GitLab did a major update of
their shared runners infrastructure, and broke nftables support in the process,
<a href="https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5184#note_637383773">as it seems</a>.
So we have to use iptables instead.</p>
<p>Let's fix our <code>.gitlab-ci.yml</code>, which now looks like that:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>.gitlab-ci.yml<span class="w"> </span>
build-container-image:
<span class="w"> </span>stage:<span class="w"> </span>build
<span class="w"> </span>image:<span class="w"> </span>debian:testing
<span class="w"> </span>before_script:
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>update
<span class="w"> </span>-<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>buildah<span class="w"> </span>ca-certificates
<span class="w"> </span>-<span class="w"> </span><span class="p">|</span>
<span class="w"> </span><span class="c1"># Switch to iptables legacy, as GitLab CI doesn't support nftables.</span>
<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>--no-install-recommends<span class="w"> </span>iptables
<span class="w"> </span>update-alternatives<span class="w"> </span>--set<span class="w"> </span>iptables<span class="w"> </span>/usr/sbin/iptables-legacy
<span class="w"> </span>script:
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span><span class="w"> </span>.
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>login<span class="w"> </span>-u<span class="w"> </span><span class="nv">$CI_REGISTRY_USER</span><span class="w"> </span>-p<span class="w"> </span><span class="nv">$CI_JOB_TOKEN</span><span class="w"> </span><span class="nv">$CI_REGISTRY</span>
<span class="w"> </span>-<span class="w"> </span>buildah<span class="w"> </span>push<span class="w"> </span><span class="nv">$CI_REGISTRY_IMAGE</span>
</code></pre></div>
<p>And push again. Does that work? Yes!</p>
<p>If you're interested in this issue, feel free to fork
<a href="https://gitlab.com/arnaudr/gitlab-build-container-image">https://gitlab.com/arnaudr/gitlab-build-container-image</a> and try it by
yourself.</p>
<p>It's been more than a year since this change, and I'm surprised that I didn't
find much about it on the Internet, neither mentions of the issue, nor of a
workaround. Maybe nobody builds container images in GitLab CI, or maybe they do
it another way, I don't know. In any case, now it's documented in this blog,
hopefully some will find it useful.</p>
<p>Happy 2023!</p>Inhibit suspending the computer with GtkApplication2020-09-25T00:00:00+00:002020-09-25T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-09-25:/2020/09/25/inhibit-suspending-the-computer-with-gtkapplication/<p>A deep dive into the topic of "system sleep inhibition", or how to prevent the
computer from suspending while your application is running. This is a post for
developers who write GTK applications for GNU/Linux. In here I'll try to cover
pretty much everything there is to know on the topic. Or at least, everything I
know.</p>
<p>A deep dive into the topic of "system sleep inhibition", or how to prevent the
computer from suspending while your application is running. This is a post for
developers who write GTK applications for GNU/Linux. In here I'll try to cover
pretty much everything there is to know on the topic. Or at least, everything I
know.</p>
<p>Just a little precision regarding the terms we use here: preventing the
"computer/session/system" from "suspending/sleeping" are the same things:
different words, same meaning.</p>
<h2>Introduction</h2>
<p>The assumption is that you're developing an application for the GNU/Linux
desktop, and that for some reason you want to make sure that the computer
doesn't go to sleep while your application is running. A typical example is a
video player: in all evidence, if you watch a 2-hour long movie, you don't want
your computer to suspend in the middle.</p>
<p>Another use-case (the one I'm concerned about) is a radio player, or more
generally a music player. <a href="https://gitlab.com/goodvibes/goodvibes">Goodvibes</a> is this little radio player that I've
been maintaining for a while, and it provides an option to prevent the system
from going to sleep while a radio is playing.</p>
<p>There are other use-cases of course.</p>
<p>So how do you do that? The thing to understand is that it's not really your
application which does that, but rather a component of the <em>Desktop
Environment</em>. Your application just communicates with this component, and asks
it to prevent the system from suspending.</p>
<p>Now, what is this mysterious "component"? Answer is: it depends.</p>
<p>If you're familiar with the wild world of Linux Desktop, you know that there's
a whole bunch of different Desktop Environments out there. <a href="https://www.gnome.org/">GNOME</a> (and GNOME
related like <a href="https://cinnamon-spices.linuxmint.com/">Cinnamon</a>, <a href="https://mate-desktop.org/">MATE</a>, <a href="https://elementary.io/">Elementary</a>, etc...), <a href="https://kde.org/">KDE</a>,
<a href="https://www.xfce.org/">XFCE</a>, and even more "barebone" systems without even a Desktop Environment
<em>per se</em>, like <a href="https://swaywm.org/">Sway</a>. Another situation to consider is that your
application might be running in a sandboxed environment, like <a href="https://flatpak.org/">Flatpak</a>,
<a href="https://snapcraft.io/">Snap</a> or <a href="https://appimage.org/">AppImage</a>.</p>
<p>Depending on the environment in which your application is running, the way to
inhibit will be different. In all likelihood, your application will have to
talk to some API over D-Bus, but the API will differ.</p>
<p>Does it mean that, if you want to ensure that your application works for all of
those environments, you will have to support all of them by yourself? Answer
is: no!</p>
<p>Don't even try! First it's a pain to do so, then it takes a lot of time to test
your solution on all of these environments, and finally, it will also require
maintenance over the years (disclaimer: I've been there).</p>
<p>The right answer is: leverage the toolkit! GTK, through the <a href="https://developer.gnome.org/gtk3/stable/GtkApplication.html">GtkApplication</a>
class, provides the two methods <code>gtk_application_inhibit ()</code> and
<code>gtk_application_uninhibit ()</code>. These methods do all the job for you, they do
all the painful job of supporting the various D-Bus APIs available out there,
and the GTK maintainers are very aware when new APIs are introduced.</p>
<p>So that's really the only sane way to go. And if ever your application does not
use <code>GtkApplication</code> yet, then it's about time to give it a try!</p>
<h2>A dive into the GtkApplication implementation</h2>
<p>Now let's be curious a bit, and let's have a look at the implementation in GTK
(version 3.x), to understand what really happens when your application calls
<code>gtk_application_inhibit ()</code>.</p>
<p>It all starts in the file <a href="https://gitlab.gnome.org/GNOME/gtk/-/blob/3.24.23/gtk/gtkapplication.c">gtk/gtkapplication.c</a>, it is where
you can find the method <code>gtk_application_inhibit ()</code>. It doesn't tell us much
though. The actual implementation can be found in the file
<a href="https://gitlab.gnome.org/GNOME/gtk/-/blob/3.24.23/gtk/gtkapplication-dbus.c">gtk/gtkapplication-dbus.c</a>, and that's where we can see
exactly what GTK does in order to inhibit.</p>
<p>At first, GTK will try to reach the <em>Session Manager</em>.</p>
<p>For GNOME and GNOME-derived desktop environments, it means calling the
following D-Bus method:</p>
<ul>
<li>bus name: <code>org.gnome.SessionManager</code></li>
<li>object path: <code>/org/gnome/SessionManager</code></li>
<li>method: <code>org.gnome.SessionManager.Inhibit</code></li>
<li>API details → <a href="https://gitlab.gnome.org/GNOME/gnome-session/-/blob/gnome-3-36/gnome-session/org.gnome.SessionManager.xml">org.gnome.SessionManager.xml</a></li>
</ul>
<p>For XFCE, GTK also supports the XFCE Session Manager:</p>
<ul>
<li>bus name: <code>org.xfce.SessionManager</code></li>
<li>object path: <code>/org/xfce/SessionManager</code></li>
<li>method: <code>org.xfce.Session.Manager.Inhibit</code></li>
<li>API details → <a href="https://gitlab.xfce.org/xfce/xfce4-session/-/blob/xfce-4.14/xfce4-session/xfsm-manager-dbus.xml">xfsm-manager-dbus.xml</a></li>
</ul>
<p>If none of these Session Managers are found, GTK will try the <em>Freedesktop
Portal</em>, ie. the following D-Bus method:</p>
<ul>
<li>bus name: <code>org.freedesktop.portal.Desktop</code></li>
<li>object path: <code>/org/freedesktop/portal/desktop</code></li>
<li>method: <code>org.freedesktop.portal.Inhibit.Inhibit</code></li>
<li>API documentation → <a href="https://flatpak.github.io/xdg-desktop-portal/portal-docs.html#gdbus-org.freedesktop.portal.Inhibit">Portal for inhibiting session transitions</a></li>
</ul>
<p>If it's not found, then it's the end of the game... Or is it?</p>
<p>For GTK3, yes, but it's interesting to note that GTK4 supports yet another way
of inhibiting. This time, it's not through D-Bus. It goes through a <em>Wayland
protocol</em>, meaning that it only works for Wayland compositors. It seems that
this support was added for minimalist desktops like Sway.</p>
<p>Also, this protocol is not exactly about what we're talking about here, it's
not about inhibiting system sleep. Instead it's about « <em>inhibiting the idle
behavior such as screen blanking, locking, and screensaving</em> ». This is an
important detail that I didn't mention yet, because it's a bit unclear to me.
But basically, you can inhibit various things, and "inhibiting the idle
behavior" is more about the screen. Does that also prevent the computer from
going to sleep, or is it completely unrelated? I have to admit I don't know...</p>
<p>So, yeah, this new thing in GTK4 might not be relevant for this blog post, but
I thought I'd mention it anyway, just in case you find it interesting. Details
can be found at:</p>
<ul>
<li>Original <a href="https://gitlab.gnome.org/GNOME/gtk/-/issues/2202">GitLab Issue #2202</a></li>
<li><a href="https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/2226">GitLab Merge Request #2226</a></li>
<li>Wayland protocol → <a href="https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/master/unstable/idle-inhibit/idle-inhibit-unstable-v1.xml">idle-inhibit-unstable-v1.xml</a></li>
</ul>
<h2>Testing it</h2>
<p>This post wouldn't be complete without actually trying to use
<code>gtk_application_inhibit ()</code> in various environments, and seeing it at work,
and feeling good about it, right?</p>
<p>So, let's come back to my personal life for a while. A few days ago I reworked
how Goodvibes inhibits the system sleep, and I started to use
<code>gtk_application_inhibit ()</code>, instead of rolling my own solution. Then I went
on to try it with different environments. Here are some results.</p>
<h3>GNOME 3.36</h3>
<p>Under GNOME, it works as expected, no surprise. GTK goes through the D-Bus
service <code>org.gnome.SessionManager</code>. I couldn't see anything in the logs, but
it's possible to verify programatically that suspend is inhibited, thanks to
the method <code>IsInhibited</code> provided by the GNOME Session Manager.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>busctl<span class="w"> </span>--user<span class="w"> </span>call<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>org.gnome.SessionManager<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>/org/gnome/SessionManager<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>org.gnome.SessionManager<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>IsInhibited<span class="w"> </span>u<span class="w"> </span><span class="m">4</span>
b<span class="w"> </span><span class="nb">true</span>
</code></pre></div>
<p>Note that the parameter, an unsigned integer, is the flag that says <em>what</em>
exactly we're checking, among the various things that can be inhibited. <code>4</code> is
for <a href="https://developer.gnome.org/gtk3/stable/GtkApplication.html#GtkApplicationInhibitFlags">suspending the session or computer</a>.</p>
<h3>KDE 5.18</h3>
<p>Under KDE, it also works as expected. We can see some logs in the systemd
journal when we inhibit:</p>
<div class="highlight"><pre><span></span><code>org.freedesktop.impl.portal.desktop.kde[9956]: xdp-kde-inhibit:
Inhibit called with parameters:
handle: "/org/freedesktop/portal/desktop/request/1_149/t"
app_id: ""
window: ""
flags: 4
options: QMap(("reason", QVariant(QString, "Playing")))
</code></pre></div>
<p>In all evidence, GTK goes through the Freedesktop Portal
<code>org.freedesktop.portal.Desktop</code> to do the job.</p>
<h3>XFCE 4.14</h3>
<p>Now is the surprise, and the reward you get for testing!</p>
<p>On XFCE it does <em>NOT</em> work:</p>
<div class="highlight"><pre><span></span><code>(goodvibes:12772): Gtk-WARNING **: 10:23:57.000:
Calling org.xfce.Session.Manager.Inhibit failed:
GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod:
No such method “Inhibit”
</code></pre></div>
<p>No such method? Indeed. Interestingly, after looking at the XFCE Session
Manager <a href="https://gitlab.xfce.org/xfce/xfce4-session">git repository</a>, it seems that the <code>Inhibit</code> method
never existed, so basically <code>gtk_application_inhibit ()</code> has never worked on
XFCE. I <a href="https://gitlab.xfce.org/xfce/xfce4-session/-/issues/71">opened an issue</a> just to see if I can get more crunchy details.</p>
<h3>XFCE Workaround</h3>
<p>So in this situation, either you live with it, either you go to some lengths to
implement some kind of workaround. I went for the latter, basically because I
have too much time on my hands.</p>
<p>It just happens that, on XFCE, there's a D-Bus service sitting there and which
provides an <code>Inhibit</code> method as well:</p>
<ul>
<li>bus name: <code>org.freedesktop.PowerManagement</code></li>
<li>object path: <code>/org/freedesktop/PowerManagement/Inhibit</code></li>
<li>method: <code>org.freedesktop.PowerManagement.Inhibit.Inhibit()</code></li>
</ul>
<p>Hey. Wait a second. So there's <em>yet another way</em> to inhibit? Answer is: yes!</p>
<p>Now to be fair, this D-Bus API seems to be a relic of the past. I did a bit of
search, and here's what I found:</p>
<ul>
<li>The <code>org.freedesktop.PowerManagement</code> spec was <a href="https://blogs.gnome.org/hughsie/2007/03/27/orgfreedesktoppowermanagement/">announced in March 2007</a></li>
<li>There seems to be an <a href="https://www.freedesktop.org/wiki/Specifications/power-management-spec/">official specification</a> ...</li>
<li>... which is obsolete. All of the links to the actual spec are dead.</li>
<li>And I couldn't find much more information on this mysterious D-Bus API.</li>
</ul>
<p>Thing is, on XFCE, it's there, and it works. You can use it and inhibit the
system sleep. It's functional. I'm not sure if it's "the right way" to do it
for XFCE, but I couldn't find any other way anyway.</p>
<p>So ultimately, in Goodvibes I added a <a href="https://gitlab.com/goodvibes/goodvibes/-/commit/1fc0dee7472448340f2707691132485484271e2c">workaround to support this D-Bus
API</a>. If ever <code>gtk_application_inhibit ()</code> fails,
Goodvibes falls back to using this old API, if available. XFCE users are
covered.</p>
<p>By the way, this API also provides a method to check if inhibition is enabled:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>busctl<span class="w"> </span>--user<span class="w"> </span>call<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>org.freedesktop.PowerManagement<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>/org/freedesktop/PowerManagement/Inhibit<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>org.freedesktop.PowerManagement.Inhibit<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>HasInhibit
b<span class="w"> </span><span class="nb">true</span>
</code></pre></div>
<h2>Conclusion</h2>
<p>There's a few conclusions I can draw from this adventure.</p>
<p>Interoperability for Linux desktop applications never fails to be a timesink.
Learning how things work deep down, digging up bits of history from the
Internet to understand how things came to be, actually testing on different
desktop environments, hitting some issues here and there... These kind of
things take forever, but...</p>
<p>It's also a great way to learn how things work, and to understand better how
Linux desktops are actually developed. I mean, if you're into this kind of
things, and willing to spend time doing research on the Internet.</p>
<p>Ultimately, the only sane way to go is to leverage well-known toolkits (GTK,
Qt) or libraries (GLib) to get your stuff done. More than often, the specs from
freedesktop.org have an implementation in these libraries, so look hard for
your solution in there, and stick to it if you can.</p>
<p>I stress that last point because initially, I implemented my own solution
simply because I didn't know about <code>gtk_application_inhibit ()</code>. Then when I
found out, I didn't want to ditch all the work I had done, so I kept using it
in my application for a few years. Actually it worked well, but finally a bug
was reported, and I just didn't want to debug all that stuff I made years
before. So I agreed with myself that it was time to get rid of it, but the
price to pay was a bit of refactoring.</p>
<p>Now I'm at peace with that, but I can't help but think that a good deal of time
could have been saved, if only from the beginning I had looked harder for an
existing solution, rather than rushing into implementing something. A mistake
done too often.</p>Send emails from your terminal with msmtp2020-08-24T00:00:00+00:002020-08-24T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-24:/2020/08/24/send-emails-from-your-terminal-with-msmtp/<p>In this tutorial, we'll configure everything needed to send emails from the
terminal. We'll use <a href="https://marlam.de/msmtp/">msmtp</a>, a lightweight SMTP client. For the sake of
the example, we'll use a GMail account, but any other email provider can do.
Your OS is expected to be <a href="https://www.debian.org/">Debian</a>, as usual on this blog, although it
doesn't really matter. We will also see how to store the credentials for the
email account in the system keyring. And finally, we'll go the extra mile, and
see how to configure various command-line utilities so that they automatically
use msmtp to send emails. Even better, we'll make msmtp the default email
sender, to actually avoid configuring these utilities one by one.</p>
<p>In this tutorial, we'll configure everything needed to send emails from the
terminal. We'll use <a href="https://marlam.de/msmtp/">msmtp</a>, a lightweight SMTP client. For the sake of
the example, we'll use a GMail account, but any other email provider can do.
Your OS is expected to be <a href="https://www.debian.org/">Debian</a>, as usual on this blog, although it
doesn't really matter. We will also see how to store the credentials for the
email account in the system keyring. And finally, we'll go the extra mile, and
see how to configure various command-line utilities so that they automatically
use msmtp to send emails. Even better, we'll make msmtp the default email
sender, to actually avoid configuring these utilities one by one.</p>
<h2>Prerequisites</h2>
<p>Strong prerequisites (if you don't recognize yourself here, you probably landed
on the wrong page):</p>
<ul>
<li>You run Linux on your computer (let's assume a Debian-like distro).</li>
<li>You want to send emails from your terminal.</li>
</ul>
<p>Weak prerequisites (if your setup doesn't match those points exactly, that's
fine, you can still read on):</p>
<ul>
<li>Your email account is a GMail one.</li>
<li>Your desktop environment is GNOME.</li>
</ul>
<h2>GMail account setup</h2>
<p>For a GMail account, there's a bit of configuration to do. For other email
providers, I have no idea, maybe you can just skip this part, or maybe you will
have to go through a similar procedure.</p>
<p>If you want an external program (<code>msmtp</code> in this case) to talk to the GMail
servers on your behalf, and send emails, you can't just use your usual GMail
password. Instead, GMail requires you to generate so-called <em>app passwords</em>,
one for each application that needs to access your GMail account.</p>
<p>This approach has several advantages:</p>
<ul>
<li>it will basically work, GMail won't block you because it thinks that you're
trying to sign in from an unknown device, a weird location or whatever.</li>
<li>your main GMail password remains secret, you won't have to write it down
in any configuration file or anywhere else.</li>
<li>you can change your main GMail password, no breakage, apps will still work as
each of them use their own passwords.</li>
<li>you can revoke an app password anytime, without impacting anything else.</li>
</ul>
<p>So app passwords are a good idea, it just requires a bit of work to set it up.
Let's see what it takes.</p>
<p>First, <em>2-Step Verification</em> must be enabled on your GMail account. Visit
<a href="https://myaccount.google.com/security">https://myaccount.google.com/security</a>, and if that's not the case, enable it.
You'll need to authorize all of your devices (computer(s), phone(s) and so on),
and it can be a bit tedious, granted. But you only have to do it once in a
lifetime, and after it's done, you're left with a more secure account, so it's
not that bad, right?</p>
<p>Enabling the 2-Step Verification will unlock the feature we need: <em>App
passwords</em>. Visit <a href="https://myaccount.google.com/apppasswords">https://myaccount.google.com/apppasswords</a>, and under
"<em>Signing in to Google</em>", click "<em>App passwords</em>", and generate one. An app
password is a 16 characters string, something like <code>qwertyuiopqwerty</code>. It's
supposed to be used from only one place, ie. from ONE application that is
installed on ONE device. That's why it's common to give it a name of the form
<code>application@device</code>, so in our case it could be <code>msmtp@laptop</code>, but really
it's free form, choose whatever name suits you, as long as it makes sense to
you.</p>
<p>So let's give a name to this app password, write it down for now, and we're
done with the GMail config.</p>
<h2>Send your first email</h2>
<p>Time to get started with <code>msmtp</code>.</p>
<p>First thing first, installation, trivial:</p>
<div class="highlight"><pre><span></span><code>sudo apt install msmtp
</code></pre></div>
<p>Let's try to send an email. At this point, we did not create any configuration
file for msmtp yet, so we have to provide every details on the command line.</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="k">Write</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">dummy</span><span class="w"> </span><span class="n">email</span>
<span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span>
<span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">SOMEONE_ELSE</span><span class="nv">@SOMEWHERE_ELSE</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Cafe</span><span class="w"> </span><span class="n">Sua</span><span class="w"> </span><span class="n">Da</span>
<span class="n">Iced</span><span class="o">-</span><span class="n">coffee</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">condensed</span><span class="w"> </span><span class="n">milk</span>
<span class="n">EOF</span>
<span class="err">#</span><span class="w"> </span><span class="n">Send</span><span class="w"> </span><span class="n">it</span>
<span class="n">cat</span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">auth</span><span class="o">=</span><span class="k">on</span><span class="w"> </span><span class="o">--</span><span class="n">tls</span><span class="o">=</span><span class="k">on</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">port</span><span class="w"> </span><span class="mi">587</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">read</span><span class="o">-</span><span class="n">envelope</span><span class="o">-</span><span class="k">from</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="k">read</span><span class="o">-</span><span class="n">recipients</span>
<span class="err">#</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="n">prompts</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="nl">password</span><span class="p">:</span>
<span class="err">#</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">goes</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="n">password</span><span class="err">!</span>
</code></pre></div>
<p>Obviously, in this example you should replace the uppercase words with the
real thing, that is, your email login, and real email addresses.</p>
<p>Also, let me insist, you must enter the <em>app password</em> that was generated
previously, not your real GMail password.</p>
<p>And it should work already, this email should have been sent and received by
now.</p>
<p>So let me explain quickly what happened here.</p>
<p>In the file <code>message.txt</code>, we provided <code>From:</code> (the email address of the person
sending the email) and <code>To:</code> (the destination email address). Then we asked
msmtp to re-use those values to set the <a href="https://stackoverflow.com/a/1767262">envelope</a> of the email with
<code>--read-envelope-from</code> and <code>--read-recipients</code>.</p>
<p>What about the other parameters?</p>
<ul>
<li><code>--auth=on</code> because we want to authenticate with the server.</li>
<li><code>--tls=on</code> because we want to make sure that the communication with the
server is encrypted.</li>
<li><code>--host</code> and <code>--port</code> tells where to find the server. If you don't use GMail,
adjust that accordingly.</li>
<li><code>--user</code> is obviously your GMail username.</li>
</ul>
<p>For more details, you should refer to the <a href="https://marlam.de/msmtp/msmtp.html">msmtp documentation</a>.</p>
<h2>Write a configuration file</h2>
<p>So we could send an email, that's cool already.</p>
<p>However the command to do that was a bit long, and we don't want to juggle with
all these arguments every time we send an email. So let's write down all of
that into a configuration file.</p>
<p>There are a few locations that we can use to write down our configuration:
<code>/etc/msmtprc</code> for system wide configuration, and <code>~/.msmtprc</code> or
<code>~/.config/msmtp/config</code> for per-user configuration. All of that is, of course,
thoroughly documented in the <a href="https://manpages.debian.org/msmtp/msmtp.1.en.html#CONFIGURATION_FILES">msmtp manual page</a>.</p>
<p>In this tutorial we'll use <code>~/.msmtprc</code> for brevity. There we go:</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s1">'EOF'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span>
<span class="n">defaults</span>
<span class="n">tls</span><span class="w"> </span><span class="k">on</span>
<span class="n">account</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">auth</span><span class="w"> </span><span class="k">on</span>
<span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">port</span><span class="w"> </span><span class="mi">587</span>
<span class="k">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span>
<span class="k">from</span><span class="w"> </span><span class="n">YOUR_LOGIN</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">EOF</span>
</code></pre></div>
<p>And for a quick explanation:</p>
<ul>
<li>under <code>defaults</code> are the default values for all the following accounts.</li>
<li>under <code>account</code> are the settings specific to this account, until another
<code>account</code> line is found.</li>
<li>finally, the last line defines which account is the default.</li>
</ul>
<p>All in all it's pretty simple, and it's becoming easier to send an email:</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="k">Write</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">dummy</span><span class="w"> </span><span class="n">email</span><span class="p">.</span><span class="w"> </span><span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">the</span>
<span class="err">#</span><span class="w"> </span><span class="n">header</span><span class="w"> </span><span class="s1">'From:'</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">no</span><span class="w"> </span><span class="n">longer</span><span class="w"> </span><span class="n">needed</span><span class="p">,</span>
<span class="err">#</span><span class="w"> </span><span class="n">it</span><span class="s1">'s already in '</span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span><span class="s1">'.</span>
<span class="s1">cat << '</span><span class="n">EOF</span><span class="err">'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">SOMEONE_ELSE</span><span class="nv">@SOMEWHERE_ELSE</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Flat</span><span class="w"> </span><span class="n">White</span>
<span class="n">The</span><span class="w"> </span><span class="n">milky</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">coffee</span>
<span class="n">EOF</span>
<span class="err">#</span><span class="w"> </span><span class="n">Send</span><span class="w"> </span><span class="n">it</span>
<span class="n">cat</span><span class="w"> </span><span class="n">message</span><span class="p">.</span><span class="n">txt</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="o">--</span><span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="c1">--read-recipients</span>
</code></pre></div>
<p>Actually, <code>--account default</code> is not needed, as it's the default anyway if you
don't provide a <code>--account</code> argument. Furthermore <code>--read-recipients</code> can be
shortened as <code>-t</code>. So we can make it real short now:</p>
<div class="highlight"><pre><span></span><code>msmtp -t < message.txt
</code></pre></div>
<p>At this point, life is good! Except for one thing maybe: we still have to type
the password every time we send an email. Surely it must be possible to avoid
that annoyance...</p>
<h2>Store your password in the system keyring</h2>
<p>For this part, we'll make use of the <a href="https://lwn.net/Articles/490518/">libsecret</a> tool to store the password
in the system keyring via the <a href="https://specifications.freedesktop.org/secret-service/latest/">Secret Service API</a>. It means that your
desktop environment should implement the Secret Service specification, which is
the case for both GNOME and KDE.</p>
<p>Note that GNOME provides <a href="https://wiki.gnome.org/Apps/Seahorse">Seahorse</a> to have a look at your secrets, KDE has
the <a href="https://utils.kde.org/projects/kwalletmanager/">KDE Wallet</a>. There's also <a href="https://keepassxc.org/">KeePassXC</a>, which I have only heard of but
never used. I guess it can be your password manager of choice if you use
neither GNOME nor KDE.</p>
<p>For those running an up-to-date Debian unstable, you should have <code>msmtp >=
1.8.11-2</code>, and you're all good to go. For those having an older version than
that however, you will have to install the package <code>msmtp-gnome</code> in order to
have msmtp built with libsecret support. Note that this package depends on
<code>seahorse</code>, hence it pulls in a good part of the GNOME stack when you install
it. For those not running GNOME, that's unfortunate. All of this was discussed
and fixed in <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962689">#962689</a>.</p>
<p>Alright! So let's just make sure that the libsecret tools are installed:</p>
<div class="highlight"><pre><span></span><code><span class="n">sudo</span><span class="w"> </span><span class="n">apt</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">libsecret</span><span class="o">-</span><span class="n">tools</span>
</code></pre></div>
<p>And now we can store our password in the system keyring with this command:</p>
<div class="highlight"><pre><span></span><code><span class="n">secret</span><span class="o">-</span><span class="k">tool</span><span class="w"> </span><span class="n">store</span><span class="w"> </span><span class="o">--</span><span class="n">label</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span>\
<span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">smtp</span><span class="o">.</span><span class="n">gmail</span><span class="o">.</span><span class="n">com</span><span class="w"> </span>\
<span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="n">smtp</span><span class="w"> </span>\
<span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="n">YOUR_LOGIN</span>
</code></pre></div>
<p>If this looks a bit too magic, and you want something more visual, you can
actually fire a GUI like <code>seahorse</code> (for GNOME users), or <code>kwalletmanager5</code>
(for KDE users), and then you will see what passwords are stored in there.</p>
<p>Here's a screenshot of Seahorse, with a msmtp password stored:</p>
<p><img alt="seahorse with msmtp password" src="/images/seahorse-msmtp.png"></p>
<p>Let's try to send an email again:</p>
<div class="highlight"><pre><span></span><code>msmtp -t < message.txt
</code></pre></div>
<p>No need for a password anymore, msmtp got it from the system keyring!</p>
<p>For more details on how msmtp handle the passwords, and to see what other
methods are supported, refer to the <a href="https://marlam.de/msmtp/msmtp.html#Authentication">extensive documentation</a>.</p>
<h2>Use-cases and integration</h2>
<p>Let's go over a few use-cases, situations where you might end up sending emails
from the command-line, and what configuration is required to make it work with
msmtp.</p>
<h3>Git Send-Email</h3>
<p>Sending emails with git is a common workflow for some projects, like the Linux
kernel. How does <code>git send-email</code> actually send emails? From the
<a href="https://manpages.debian.org/git-email/git-send-email.1.en.html">git-send-email manual page</a>:</p>
<blockquote>
<p>the built-in default is to search for sendmail in /usr/sbin, /usr/lib and
$PATH if such program is available</p>
</blockquote>
<p>It is possible to override this default though:</p>
<blockquote>
<p>--smtp-server=<host><br>
[...] Alternatively it can specify a full pathname of a sendmail-like program
instead; the program must support the -i option.</p>
</blockquote>
<p>So in order to use msmtp here, you'd add a snippet like that to your
<code>~/.gitconfig</code> file:</p>
<div class="highlight"><pre><span></span><code><span class="k">[sendemail]</span>
<span class="w"> </span><span class="na">smtpserver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/usr/bin/msmtp</span>
</code></pre></div>
<p>For a full guide, you can also refer to <a href="https://git-send-email.io">https://git-send-email.io</a>.</p>
<h3>Debian developer tools</h3>
<p>Tools like <code>bts</code> or <code>reportbug</code> are also good examples of command-line tools
that need to send emails.</p>
<p>From the <a href="https://manpages.debian.org/devscripts/bts.1.en.html">bts manual page</a>:</p>
<blockquote>
<p>--sendmail=SENDMAILCMD<br>
Specify the sendmail command [...] Default is /usr/sbin/sendmail.</p>
</blockquote>
<p>So if you want bts to send emails with msmtp instead of sendmail, you must use
<code>bts --sendmail='/usr/bin/msmtp -t'</code>.</p>
<p>Note that bts also loads settings from the file <code>/etc/devscripts.conf</code> and
<code>~/.devscripts</code>, so you could also set <code>BTS_SENDMAIL_COMMAND='/usr/bin/msmtp
-t'</code> in one of those files.</p>
<p>From the <a href="https://manpages.debian.org/reportbug/reportbug.1.en.html">reportbug manual page</a>:</p>
<blockquote>
<p>--mta=MTA<br>
Specify an alternate MTA, instead of /usr/sbin/sendmail (the default).</p>
</blockquote>
<p>In order to use msmtp here, you'd write <code>reportbug --mta=/usr/bin/msmtp</code>.</p>
<p>Note that reportbug reads it settings from <code>/etc/reportbug.conf</code> and
<code>~/.reportbugrc</code>, so you could as well set <code>mta /usr/bin/msmtp</code> in one of those
files.</p>
<h3>So who is this sendmail again?</h3>
<p>By now, you probably noticed that <a href="https://en.wikipedia.org/wiki/Sendmail">sendmail</a> seems to be considered the
default tool for the job, the "traditional" command that has been around for
ages.</p>
<p>Rather than configuring every tool to use something else than sendmail,
wouldn't it be simpler to actually replace sendmail by msmtp? Like, create a
symlink that points to msmtp, something like <code>ln -sr /usr/bin/msmtp
/usr/sbin/sendmail</code>? So that msmtp acts as a drop-in replacement for sendmail,
and there's nothing else to configure?</p>
<p>Answer is yes, kind of. Actually, the <a href="https://marlam.de/msmtp/">first msmtp feature</a>
that is listed on the homepage is "<em>Sendmail compatible interface (command
line options and exit codes)</em>". Meaning that msmtp <em>is</em> a drop-in replacement
for sendmail, that seems to be the intent.</p>
<p>However, you should refrain from creating or modifying anything in <code>/usr</code>, as
it's the territory of the package manager, <code>apt</code>. Any change in <code>/usr</code> might be
overwritten by apt the next time you run an upgrade or install new packages.</p>
<p>In the case of msmtp, there is actually a package named <code>msmtp-mta</code> that will
create this symlink for you. So if you really want a definitive replacement for
sendmail, there you go:</p>
<div class="highlight"><pre><span></span><code>sudo apt install msmtp-mta
</code></pre></div>
<p>From this point, sendmail is now a symlink <code>/usr/sbin/sendmail →
/usr/bin/msmtp</code>, and there's no need to configure <code>git</code>, <code>bts</code>, <code>reportbug</code> or
any other tool that would rely on sendmail. Everything should work "out of the
box".</p>
<h2>Conclusion</h2>
<p>I hope that you enjoyed reading this article! If you have any comment, feel
free to send me a short email, preferably from your terminal!</p>Modify Vim syntax files for your taste2020-08-17T00:00:00+00:002020-08-17T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-17:/2020/08/17/modify-vim-syntax-files-for-your-taste/<p>In this short how-to, we'll see how to make small modifications to a Vim syntax
file, in order to change how a particular file format is highlighted. We'll go
for a simple use-case: modify the Markdown syntax file, so that H1 and H2
headings (titles and subtitles, if you prefer) are displayed in bold. Of
course, this won't be exactly as easy as expected, but no worries, we'll
succeed in the end.</p>
<p>In this short how-to, we'll see how to make small modifications to a Vim syntax
file, in order to change how a particular file format is highlighted. We'll go
for a simple use-case: modify the Markdown syntax file, so that H1 and H2
headings (titles and subtitles, if you prefer) are displayed in bold. Of
course, this won't be exactly as easy as expected, but no worries, we'll
succeed in the end.</p>
<h2>The calling</h2>
<p>Let's start with a screenshot: how <a href="https://www.vim.org/">Vim</a> displays <a href="https://commonmark.org/">Markdown</a> files for me,
someone who use the <a href="https://www.gnome.org/">GNOME</a> terminal with the <a href="https://ethanschoonover.com/solarized/">Solarized</a> light theme.</p>
<p><img alt="Vim - Markdown file with original highlighting" src="/images/vim-syntax-01.png"></p>
<p>I'm mostly happy with that, except for one or two little details. I'd like to
have the titles displayed in bold, for example, so that they're easier to spot
when I skim through a Markdown file. It seems like a simple thing to ask, so I
hope there can be a simple solution.</p>
<h2>The first steps</h2>
<p>Let's learn the basics.</p>
<p>In Vim world, the rules to highlight files formats are defined in the directory
<code>/usr/share/vim/vim82/syntax</code> (I bet you'll have to adjust this path depending
on the version of Vim that is installed on your system).</p>
<p>And so, for the Markdown file format, the rules are defined in the file
<code>/usr/share/vim/vim82/syntax/markdown.vim</code>.</p>
<p>The first thing we could do is to have a look at this file, try to make sense
of it, and maybe start to make some modifications.</p>
<p>But wait a moment. You should know that modifying a system file is not a great
idea. First because your changes will be lost as soon as an update kicks in and
the package manager replaces this file by a new version. Second, because you
will quickly forget what files you modified, and what were your modifications,
and if you do that too much, you might experience what is called "maintenance
headache" in the long run.</p>
<p>So instead, maybe you DO NOT modify this file, and instead you copy it in your
personal Vim folder, more precisely in <code>~/.vim/syntax</code>. Create this directory
if it does not exist:</p>
<div class="highlight"><pre><span></span><code>mkdir -p ~/.vim/syntax
cp /usr/share/vim/vim82/syntax/markdown.vim ~/.vim/syntax
</code></pre></div>
<p>The file in your personal folder takes precedence over the system file of the
same name in <code>/usr/share/vim/vim82/syntax/</code>, it is a <em>replacement</em> for the
existing syntax files. And so from now on, Vim uses the file
<code>~/.vim/syntax/markdown.vim</code>, and this is where we can make our modifications.</p>
<p>(And by the way, this is explained in the <a href="https://vimhelp.org/vim_faq.txt.html#faq-24.12">Vim faq-24.12</a>)</p>
<p>And so, it's already nice to know all of that, but wait, there's even better.</p>
<p>There's is another location of interest, and it is <code>~/.vim/after/syntax</code>. You
can drop syntax files in this directory, and these files are treated as
<em>additions</em> to the existing syntax. So if you only want to make slight
modifications, that's the way to go.</p>
<p>(And by the way, this is explained in the <a href="https://vimhelp.org/vim_faq.txt.html#faq-24.11">Vim faq-24.11</a>)</p>
<p>So let's forget about a syntax replacement in <code>~/.vim/syntax/markdown.vim</code>, and
instead let's go for some syntax additions in <code>~/.vim/after/syntax/markdown.vim</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir -p ~/.vim/after/syntax
touch ~/.vim/after/syntax/markdown.vim
</code></pre></div>
<p>Now, let's answer the initial question: how do we modify the highlighting rules
for Markdown files, so that the titles are displayed in bold? First, we have to
understand where are the rules that define the highlighting for titles. Here
there are, from the file <code>/usr/share/vim/vim82/syntax/markdown.vim</code>:</p>
<div class="highlight"><pre><span></span><code>hi def link markdownH1 htmlH1
hi def link markdownH2 htmlH2
hi def link markdownH3 htmlH3
...
</code></pre></div>
<p>You should know that <em>H1</em> means <em>Heading 1</em>, and so on, and so we want to make
H1 and H2 bold. What we can see here is that the headings in the Markdown files
are highlighted like the headings in HTML files, and this is obviously defined
in the file <code>/usr/share/vim/vim82/syntax/html.vim</code>. So let's have a look into
this file:</p>
<div class="highlight"><pre><span></span><code>hi def link htmlH1 Title
hi def link htmlH2 htmlH1
hi def link htmlH3 htmlH2
...
</code></pre></div>
<p>Let's keep digging a bit. Where is <code>Title</code> defined? For those using the
<code>default</code> color scheme like me, this is defined straight in the Vim source
code, in the file
<a href="https://github.com/vim/vim/blob/7ac616cb0a52bc72b449e19cf9db93bee116c15a/src/highlight.c#L190">src/highlight.c</a>.</p>
<div class="highlight"><pre><span></span><code>CENT("Title term=bold ctermfg=DarkMagenta",
"Title term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta"),
</code></pre></div>
<p>And for those using custom color schemes, it might be defined in a file under
<code>/usr/share/vim/vim82/colors/</code>.</p>
<p>Alright, so how do we override that? We can just define this kind of rules in
our syntax additions file at <code>~/.vim/after/syntax/markdown.vim</code>:</p>
<div class="highlight"><pre><span></span><code>hi link markdownH1 markdownHxBold
hi link markdownH2 markdownHxBold
hi markdownHxBold term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta cterm=bold
</code></pre></div>
<p>As you can see, the only addition we made, compared to what's defined in
<code>src/highlight.c</code>, is <code>cterm=bold</code>. And that's already enough to achieve the
initial goal, make the titles (ie. H1 and H2) bold. The result can be seen in
the following screenshot:</p>
<p><img alt="Vim - Markdown file with modified highlighting" src="/images/vim-syntax-02.png"></p>
<h2>The rabbit hole</h2>
<p>So we could stop right here, and life would be easy and good.</p>
<p>However, with this solution there's still something that is not perfect. We use
the color <code>DarkMagenta</code> as defined in the default color scheme. What I didn't
mention however, is that this is applicable for a <em>light</em> background. If you
have a <em>dark</em> background though, dark magenta won't be easy to read.</p>
<p>Actually, if you look a bit more into <code>src/highlight.c</code>, you will see that the
default color scheme comes in two variants, one for a light background, and one
for a dark background.</p>
<p>And so the definition for <code>Title</code> for a dark background is as follow:</p>
<div class="highlight"><pre><span></span><code>CENT("Title term=bold ctermfg=LightMagenta",
"Title term=bold ctermfg=LightMagenta gui=bold guifg=Magenta"),
</code></pre></div>
<p>Hmmm, so how do we do that in our syntax file? How can we support both light
and dark background, so that the color is right in both cases?</p>
<p>After a bit of research, and after looking at other syntax files, it seems that
the solution is to check for the value of the <code>background</code> option, and so our
syntax file becomes:</p>
<div class="highlight"><pre><span></span><code><span class="nv">hi</span><span class="w"> </span><span class="nv">link</span><span class="w"> </span><span class="nv">markdownH1</span><span class="w"> </span><span class="nv">markdownHxBold</span>
<span class="nv">hi</span><span class="w"> </span><span class="nv">link</span><span class="w"> </span><span class="nv">markdownH2</span><span class="w"> </span><span class="nv">markdownHxBold</span>
<span class="k">if</span><span class="w"> </span><span class="o">&</span><span class="nv">background</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"light"</span>
<span class="w"> </span><span class="nv">hi</span><span class="w"> </span><span class="nv">markdownHxBold</span><span class="w"> </span><span class="nv">term</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">ctermfg</span><span class="o">=</span><span class="nv">DarkMagenta</span><span class="w"> </span><span class="nv">gui</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">guifg</span><span class="o">=</span><span class="nv">Magenta</span><span class="w"> </span><span class="nv">cterm</span><span class="o">=</span><span class="nv">bold</span>
<span class="k">else</span>
<span class="w"> </span><span class="nv">hi</span><span class="w"> </span><span class="nv">markdownHxBold</span><span class="w"> </span><span class="nv">term</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">ctermfg</span><span class="o">=</span><span class="nv">LightMagenta</span><span class="w"> </span><span class="nv">gui</span><span class="o">=</span><span class="nv">bold</span><span class="w"> </span><span class="nv">guifg</span><span class="o">=</span><span class="nv">Magenta</span><span class="w"> </span><span class="nv">cterm</span><span class="o">=</span><span class="nv">bold</span>
<span class="k">endif</span>
</code></pre></div>
<p>In case you wonder, in Vim script you prefix Vim options with <code>&</code>, and so you
get the value of the <code>background</code> option by writing <code>&background</code>. You can
learn this kind of things in the <a href="https://devhints.io/vimscript">Vim scripting cheatsheet</a>.</p>
<p>And so, it's easy enough, except for one thing: it doesn't work. The headings
always show up in <code>DarkMagenta</code>, even for a dark background.</p>
<p>This is why I called this paragraph "the rabbit hole", by the way.</p>
<p>So... Well after trying a few things, I noticed that in order to make it work,
I would have to reload the syntax files with <code>:syntax on</code>.</p>
<p>At this point, the most likely explanation is that the <code>background</code> option is
not set yet when the syntax files are loaded at startup, hence it needs to be
reloaded manually afterward.</p>
<p>And after muuuuuuch research, I found out that it's actually possible to set a
hook for when an option is modified. Meaning, it's possible to execute a
function when the <code>background</code> option is modified. Quite cool actually.</p>
<p>And so, there it goes in my <code>~/.vimrc</code>:</p>
<div class="highlight"><pre><span></span><code><span class="s2">" Reload syntax when the background changes </span>
<span class="n">autocmd</span><span class="w"> </span><span class="n">OptionSet</span><span class="w"> </span><span class="n">background</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">exists</span><span class="p">(</span><span class="s2">"g:syntax_on"</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">syntax</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">endif</span>
</code></pre></div>
<p>For humans, this line reads as:</p>
<ol>
<li>when the background option is modified -- <code>autocmd OptionSet background</code></li>
<li>check if the syntax is on -- <code>if exists("g:syntax_on")</code></li>
<li>if that's the case, reload it -- <code>syntax on</code></li>
</ol>
<p>With that in place, my Markdown syntax overrides work for both dark and light
background. Champagne!</p>
<h2>The happy end</h2>
<p>To finish, let me share my actual additions to the <code>markdown.vim</code> syntax. It
makes H1 and H2 bold, along with their delimiters, and it also colors the
inline code and the code blocks.</p>
<div class="highlight"><pre><span></span><code><span class="s">" H1 and H2 headings -> bold</span>
<span class="s">hi link markdownH1 markdownHxBold</span>
<span class="s">hi link markdownH2 markdownHxBold</span>
<span class="s">"</span><span class="w"> </span><span class="kr">Heading</span><span class="w"> </span><span class="n">delimiters</span><span class="w"> </span><span class="p">(</span><span class="n">eg</span><span class="w"> </span><span class="s">'#'</span><span class="p">)</span><span class="w"> </span><span class="kr">and</span><span class="w"> </span><span class="n">rules</span><span class="w"> </span><span class="p">(</span><span class="n">eg</span><span class="w"> </span><span class="s">'----'</span><span class="p">,</span><span class="w"> </span><span class="s">'===='</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">bold</span>
<span class="n">hi</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">markdownHeadingDelimiter</span><span class="w"> </span><span class="n">markdownHxBold</span>
<span class="n">hi</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">markdownRule</span><span class="w"> </span><span class="n">markdownHxBold</span>
<span class="s">" Code blocks and inline code -> highlighted</span>
<span class="s">hi link markdownCode htmlH1</span>
<span class="s">"</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">following</span><span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="n">requires</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">addition</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">vimrc</span><span class="o">:</span>
<span class="s">" autocmd OptionSet background if exists("</span><span class="n">g</span><span class="o">:</span><span class="n">syntax_on</span><span class="s">") | syntax on | endif</span>
<span class="s">if &background == "</span><span class="n">light</span><span class="s">"</span>
<span class="s"> hi markdownHxBold term=bold ctermfg=DarkMagenta gui=bold guifg=Magenta cterm=bold</span>
<span class="s">else</span>
<span class="s"> hi markdownHxBold term=bold ctermfg=LightMagenta gui=bold guifg=Magenta cterm=bold</span>
<span class="s">endif</span>
</code></pre></div>
<p>And here's how it looks like with a light background:</p>
<p><img alt="Vim - Markdown file with final highlighting (light)" src="/images/vim-syntax-03.png"></p>
<p>And a dark background:</p>
<p><img alt="Vim - Markdown file with final highlighting (dark)" src="/images/vim-syntax-04.png"></p>
<p>That's all, that's very little changes compared to the highlighting from the
original syntax file, and now that we understand how it's supposed to be done,
it's not much effort to achieve it.</p>
<p>It's just that finding the workaround to make it work for both light and dark
background took forever, and leaves the usual, unanswered question: <a href="https://github.com/vim/vim/issues/6692">bug or
feature</a>?</p>GoAccess 1.4, a detailed tutorial2020-08-10T00:00:00+00:002020-08-10T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2020-08-10:/2020/08/10/goaccess-14-a-detailed-tutorial/<p>GoAccess v1.4 was just released a few weeks ago! Let's take this chance to
write a loooong tutorial. We'll go over every steps to install and operate
GoAccess. This is a tutorial aimed at those who don't play sysadmin every day,
and that's why it's so long, I did my best to provide thorough explanations all
along, so that it's more than just a "copy-and-paste" kind of tutorial. And for
those who do play sysadmin everyday: please try not to fall asleep while
reading, and don't hesitate to drop me an e-mail if you spot anything
inaccurate in here. Thanks!</p>
<p>GoAccess v1.4 was just released a few weeks ago! Let's take this chance to
write a loooong tutorial. We'll go over every steps to install and operate
GoAccess. This is a tutorial aimed at those who don't play sysadmin every day,
and that's why it's so long, I did my best to provide thorough explanations all
along, so that it's more than just a "copy-and-paste" kind of tutorial. And for
those who do play sysadmin everyday: please try not to fall asleep while
reading, and don't hesitate to drop me an e-mail if you spot anything
inaccurate in here. Thanks!</p>
<h2>Introduction</h2>
<p>So what's <a href="https://goaccess.io/">GoAccess</a> already? GoAccess is a web log analyzer, and it allows
you to visualize the traffic for your website, and get to know a bit more about
your visitors: how many visitors and hits, for which pages, coming from where
(geolocation, operating system, web browser...), etc... It does so by
parsing the access logs from your web server, be it Apache, NGINX or whatever.</p>
<p>GoAccess gives you different options to display the statistics, and in this
tutorial we'll focus on producing a HTML report. Meaning that you can see the
statistics for your website straight in your web browser, under the form of a
single HTML page.</p>
<p>For an example, you can have a look at the stats of my blog here:
<a href="https://goaccess.arnaudr.io">https://goaccess.arnaudr.io</a>.</p>
<p>GoAccess is written in C, it has very few dependencies, it had been around for
about 10 years, and it's distributed under the MIT license.</p>
<h2>Assumptions</h2>
<p>This tutorial is about installing and configuring, so I'll assume that all the
commands are run as root. I won't prefix each of them with <code>sudo</code>.</p>
<p>I use the <a href="https://httpd.apache.org/">Apache</a> web server, running on a <a href="https://www.debian.org/">Debian</a> system. I don't think
it matters so much for this tutorial though. If you're using <a href="https://www.nginx.com/">NGINX</a> it's
fine, you can keep reading.</p>
<p>Also, I will just use the name <code>SITE</code> for the name of the website that we want
to analyze with GoAccess. Just replace that with the real name of your site.</p>
<p>I also assume the following locations for your stuff:</p>
<ul>
<li>the website is at <code>/var/www/SITE</code></li>
<li>the logs are at <code>/var/log/apache2/SITE</code> (yes, there is a sub-directory)</li>
<li>we're going to save the GoAccess database in <code>/var/lib/goaccess-db/SITE</code>.</li>
</ul>
<p>If you have your stuff in <code>/srv/SITE/{log,www}</code> instead, no worries, just
adjust the paths accordingly, I bet you can do it.</p>
<h2>Installation</h2>
<p>The latest version of GoAccess is <code>v1.4</code>, and it's not yet available in the
Debian repositories. So for this part, you can follow the instructions from
the official <a href="https://goaccess.io/download#official-repo">GoAccess download page</a>. Install steps
are explained in details, so there's nothing left for me to say :)</p>
<p>When this is done, let's get started with the basics.</p>
<p>We're talking about the latest version <code>v1.4</code> here, let's make sure:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>goaccess<span class="w"> </span>--version
GoAccess<span class="w"> </span>-<span class="w"> </span><span class="m">1</span>.4.
...
</code></pre></div>
<p>Now let's try to create a HTML report. I assume that you already have a website
up and running.</p>
<p>GoAccess needs to parse the <strong>access logs</strong>. These logs are optional, they
might or might not be created by your web server, depending on how it's
configured. Usually, these log files are named <code>access.log</code>, unsurprisingly.</p>
<p>You can check if those logs exist on your system by running this command:</p>
<div class="highlight"><pre><span></span><code><span class="n">find</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="w"> </span><span class="o">-</span><span class="n">name</span><span class="w"> </span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>Another important thing to know is that these logs can be in different formats.
In this tutorial we'll assume that we work with the <strong>combined log format</strong>,
because it seems to be the most common default.</p>
<p>To check what kind of access logs your web server produces, you must look at
the configuration for your site.</p>
<p>For an Apache web server, you should have such a line in the file
<code>/etc/apache2/sites-enabled/SITE.conf</code>:</p>
<div class="highlight"><pre><span></span><code>CustomLog<span class="w"> </span><span class="cp">${</span><span class="n">APACHE_LOG_DIR</span><span class="cp">}</span>/SITE/access.log<span class="w"> </span>combined
</code></pre></div>
<p>For NGINX, it's quite similar. The configuration file would be something like
<code>/etc/nginx/sites-enabled/SITE</code>, and the line to enable access logs would be
something like:</p>
<div class="highlight"><pre><span></span><code><span class="n">access_log</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">SITE</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>Note that <a href="https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#access_log">NGINX writes the access logs in the combined format</a>
by default, that's why you don't see the word <code>combined</code> anywhere in the line
above: it's implicit.</p>
<p>Alright, so from now on we assume that yes, you have access log files
available, and yes, they are in the combined log format. If that's the case,
then you can already run GoAccess and generate a report, for example for the
log file <code>/var/log/apache2/access.log</code></p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p>It's possible to give GoAccess more than one log files to process, so if you
have for example the file <code>access.log.1</code> around, you can use it as well:</p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="mi">1</span>
</code></pre></div>
<p>If GoAccess succeeds (and it should), you're on the right track!</p>
<p>All is left to do to complete this test is to have a look at the HTML report
created. It's a single HTML page, so you can easily <code>scp</code> it to your machine,
or just move it to the document root of your site, and then open it in your web
browser.</p>
<p>Looks good? So let's move on to more interesting things.</p>
<h2>Web server configuration</h2>
<p>This part is very short, because in terms of configuration of the web server,
there's very little to do. As I said above, the only thing you want from the
web server is to create access log files. Then you want to be sure that
GoAccess and your web server agree on the format for these files.</p>
<p>In the part above we used the combined log format, but GoAccess supports many
other common log formats out of the box, and even allows you to parse custom
log formats. For more details, refer to the option <code>--log-format</code> in the
<a href="https://goaccess.io/man#options">GoAccess manual page</a>.</p>
<p>Another common log format is named, well, <code>common</code>. It even has its own
<a href="https://en.wikipedia.org/wiki/Common_Log_Format">Wikipedia page</a>. But compared to <code>combined</code>, the common
log format contains less information, it doesn't include the <code>referrer</code> and
<code>user-agent</code> values, meaning that you won't have it in the GoAccess report.</p>
<p>So at this point you should understand that, unsurprisingly, GoAccess can only
tell you about what's in the access logs, no more no less.</p>
<p>And that's all in term of web server configuration.</p>
<h2>Configuration to run GoAccess unprivileged</h2>
<p>Now we're going to create a user and group for GoAccess, so that we don't have
to run it as root. The reason is that, well, for everything running unattended
on your server, the less code runs as root, the better. It's good practice and
common sense.</p>
<p>In this case, GoAccess is simply a log analyzer. So it just needs to read the
logs files from your web server, and there is no need to be root for that, an
unprivileged user can do the job just as well, assuming it has read permissions
on <code>/var/log/apache2</code> or <code>/var/log/nginx</code>.</p>
<p>The log files of the web server are usually part of the <code>adm</code> group (though it
might depend on your distro, I'm not sure). This is something you can check
easily with the following command:</p>
<div class="highlight"><pre><span></span><code><span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="o">-</span><span class="n">e</span><span class="w"> </span><span class="n">apache2</span><span class="w"> </span><span class="o">-</span><span class="n">e</span><span class="w"> </span><span class="n">nginx</span>
</code></pre></div>
<p>As a result you should get something like that:</p>
<div class="highlight"><pre><span></span><code><span class="n">drwxr</span><span class="o">-</span><span class="n">x</span><span class="o">---</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">adm</span><span class="w"> </span><span class="mi">20480</span><span class="w"> </span><span class="n">Jul</span><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span>
</code></pre></div>
<p>And as you can see, the directory <code>apache2</code> belongs to the group <code>adm</code>. It
means that you don't need to be root to read the logs, instead any unprivileged
user that belongs to the group <code>adm</code> can do it.</p>
<p>So, let's create the <code>goaccess</code> user, and add it to the <code>adm</code> group:</p>
<div class="highlight"><pre><span></span><code>adduser --system --group --no-create-home goaccess
addgroup goaccess adm
</code></pre></div>
<p>And now, let's run GoAccess unprivileged, and verify that it can still read
the log files:</p>
<div class="highlight"><pre><span></span><code><span class="n">setpriv</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">reuid</span><span class="o">=</span><span class="n">goaccess</span><span class="w"> </span><span class="o">--</span><span class="n">regid</span><span class="o">=</span><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">init</span><span class="o">-</span><span class="n">groups</span><span class="w"> </span><span class="o">--</span><span class="n">inh</span><span class="o">-</span><span class="n">caps</span><span class="o">=-</span><span class="n">all</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="w"> </span>\
<span class="w"> </span><span class="n">goaccess</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nb">log</span><span class="o">-</span><span class="n">format</span><span class="w"> </span><span class="n">COMBINED</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">output</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">report2</span><span class="o">.</span><span class="n">html</span><span class="w"> </span>\
<span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">access</span><span class="o">.</span><span class="n">log</span>
</code></pre></div>
<p><code>setpriv</code> is the command used to drop privileges. The syntax is quite verbose,
it's not super friendly for tutorials, but don't be scared and read the manual
page to learn what it does.</p>
<p>In any case, this command should work, and at this point, it means that you
have a <code>goaccess</code> user ready, and we'll use it to run GoAccess unprivileged.</p>
<h2>Integration, option A - Run GoAccess once a day, from a logrotate hook</h2>
<p>In this part we wire things together, so that GoAccess processes the log files
once a day, adds the new logs to its internal database, and generates a report
from all that aggregated data. The result will be a single HTML page.</p>
<h3>Introducing logrotate</h3>
<p>In order to do that, we'll use a logrotate hook. <a href="https://manpages.debian.org/stable/logrotate/logrotate.8.en.html">logrotate</a> is a little tool
that should already be installed on your server, and that runs once a day, and
that is in charge of rotating the log files. "Rotating the logs" means moving
<code>access.log</code> to <code>access.log.1</code> and so on. With logrotate, a new log file is
created every day, and log files that are too old are deleted. That's what
prevents your logs from filling up your disk basically :)</p>
<p>You can check that logrotate is indeed installed and enabled with this command
(assuming that your init system is <code>systemd</code>):</p>
<div class="highlight"><pre><span></span><code><span class="nv">systemctl</span><span class="w"> </span><span class="nv">status</span><span class="w"> </span><span class="k">logrotate</span>.<span class="nv">timer</span>
</code></pre></div>
<p>What's interesting for us is that <code>logrotate</code> allows you to run scripts before
and after the rotation is performed, so it's an ideal place from where to run
GoAccess. In short, we want to run GoAccess just before the logs are rotated
away, in the <code>prerotate</code> hook.</p>
<p>But let's do things in order. At first, we need to write a little wrapper
script that will be in charge of running GoAccess with the right arguments, and
that will process all of your sites.</p>
<h3>The wrapper script</h3>
<p>This wrapper is made to process more than one site, but if you have only one
site it works just as well, of course.</p>
<p>So let me just drop it on you like that, and I'll explain afterward. Here's my
wrapper script:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1"># Process log files /var/www/apache2/SITE/access.log,</span>
<span class="c1"># only if /var/lib/goaccess-db/SITE exists.</span>
<span class="c1"># Create HTML reports in $1, a directory that must exist.</span>
<span class="nb">set</span><span class="w"> </span>-eu
<span class="nv">OUTDIR</span><span class="o">=</span>
<span class="nv">LOGDIR</span><span class="o">=</span>/var/log/apache2
<span class="nv">DBDIR</span><span class="o">=</span>/var/lib/goaccess-db
fail<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"Usage: </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="s2"> OUTPUT_DIRECTORY"</span>
<span class="nv">OUTDIR</span><span class="o">=</span><span class="nv">$1</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$OUTDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$OUTDIR</span><span class="s2">' is not a directory"</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$LOGDIR</span><span class="s2">' is not a directory"</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$DBDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>fail<span class="w"> </span><span class="s2">"'</span><span class="nv">$DBDIR</span><span class="s2">' is not a directory"</span>
<span class="k">for</span><span class="w"> </span>d<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="k">$(</span>find<span class="w"> </span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">"</span><span class="w"> </span>-mindepth<span class="w"> </span><span class="m">1</span><span class="w"> </span>-maxdepth<span class="w"> </span><span class="m">1</span><span class="w"> </span>-type<span class="w"> </span>d<span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">site</span><span class="o">=</span><span class="k">$(</span>basename<span class="w"> </span><span class="s2">"</span><span class="nv">$d</span><span class="s2">"</span><span class="k">)</span>
<span class="w"> </span><span class="nv">dbdir</span><span class="o">=</span><span class="nv">$DBDIR</span>/<span class="nv">$site</span>
<span class="w"> </span><span class="nv">logfile</span><span class="o">=</span><span class="nv">$d</span>/access.log
<span class="w"> </span><span class="nv">outfile</span><span class="o">=</span><span class="nv">$OUTDIR</span>/<span class="nv">$site</span>.html
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$dbdir</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"‣ Skipping site '</span><span class="nv">$site</span><span class="s2">'"</span>
<span class="w"> </span><span class="k">continue</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"‣ Processing site '</span><span class="nv">$site</span><span class="s2">'"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>setpriv<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--reuid<span class="o">=</span>goaccess<span class="w"> </span>--regid<span class="o">=</span>goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--init-groups<span class="w"> </span>--inh-caps<span class="o">=</span>-all<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--agent-list<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--anonymize-ip<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--persist<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--restore<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--config-file<span class="w"> </span>/etc/goaccess/goaccess.conf<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-path<span class="w"> </span><span class="s2">"</span><span class="nv">$dbdir</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--log-format<span class="w"> </span><span class="s2">"COMBINED"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--output<span class="w"> </span><span class="s2">"</span><span class="nv">$outfile</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span>
<span class="k">done</span>
</code></pre></div>
<p>So you'd install this script at <code>/usr/local/bin/goaccess-wrapper</code> for example,
and make it executable:</p>
<div class="highlight"><pre><span></span><code>chmod +x /usr/local/bin/goaccess-wrapper
</code></pre></div>
<p>A few things to note:</p>
<ul>
<li>We run GoAccess with <code>--persist</code>, meaning that we save the parsed
logs in the internal database, and <code>--restore</code>, meaning that we include
everything from the database in the report. In other words, we aggregate the
data at every run, and the report grows bigger every time.</li>
<li>The parameter <code>--config-file /etc/goaccess/goaccess.conf</code> is a
workaround for <a href="https://github.com/allinurl/goaccess/issues/1849">#1849</a>.
It should not be needed for future versions of GoAccess <code>> 1.4</code>.</li>
</ul>
<p>As is, the script makes the assumption that the logs for your site are logged
in a sub-directory <code>/var/log/apache2/SITE/</code>. If it's not the case, adjust that
in the wrapper accordingly.</p>
<p>The name of this sub-directory is then used to find the GoAccess database
directory <code>/var/lib/goaccess-db/SITE/</code>. This directory is expected to exist,
meaning that if you don't create it yourself, the wrapper won't process this
particular site. It's a simple way to control which sites are processed by
this GoAccess wrapper, and which sites are not.</p>
<p>So if you want <code>goaccess-wrapper</code> to process the site <code>SITE</code>, just create a
directory with the name of this site under <code>/var/lib/goaccess-db</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
<span class="n">chown</span><span class="w"> </span><span class="n">goaccess</span><span class="p">:</span><span class="n">goaccess</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>Now let's create an output directory:</p>
<div class="highlight"><pre><span></span><code>mkdir /tmp/goaccess-reports
chown goaccess:goaccess /tmp/goaccess-reports
</code></pre></div>
<p>And let's give a try to the wrapper script:</p>
<div class="highlight"><pre><span></span><code>goaccess-wrapper /tmp/goaccess-reports
ls /tmp/goaccess-reports
</code></pre></div>
<p>Which should give you:</p>
<div class="highlight"><pre><span></span><code>SITE.html
</code></pre></div>
<p>At the same time, you can check that GoAccess populated the database with a
bunch of files:</p>
<div class="highlight"><pre><span></span><code><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<h3>Setting up the logrotate prerotate hook</h3>
<p>At this point, we have the wrapper in place. Let's now add a pre-rotate hook so
that <code>goaccess-wrapper</code> runs once a day, just before the logs are rotated away.</p>
<p>The logrotate config file for Apache2 is located at <code>/etc/logrotate.d/apache2</code>,
and for NGINX it's at <code>/etc/logrotate.d/nginx</code>. Among the many things you'll
see in this file, here's what is of interest for us:</p>
<ul>
<li><code>daily</code> means that your logs are rotated every day</li>
<li><code>sharedscripts</code> means that the pre-rotate and post-rotate scripts are
executed once total per rotation, and not once per log file.</li>
</ul>
<p>In the config file, there is also this snippet:</p>
<div class="highlight"><pre><span></span><code><span class="nv">prerotate</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span>[<span class="w"> </span><span class="o">-</span><span class="nv">d</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="w"> </span>]<span class="c1">; then \</span>
<span class="w"> </span><span class="nv">run</span><span class="o">-</span><span class="nv">parts</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="c1">; \</span>
<span class="w"> </span><span class="nv">fi</span><span class="c1">; \</span>
<span class="nv">endscript</span>
</code></pre></div>
<p>It indicates that scripts in the directory <code>/etc/logrotate.d/httpd-prerotate/</code>
will be executed before the rotation takes place. Refer to the man page
<code>run-parts(8)</code> for more details...</p>
<p>Putting all of that together, it means that logs from the web server are
rotated once a day, and if we want to run scripts just before the rotation, we
can just drop them in the <code>httpd-prerotate</code> directory. Simple, right?</p>
<p>Let's first create this directory if it doesn't exist:</p>
<div class="highlight"><pre><span></span><code><span class="nv">mkdir</span><span class="w"> </span><span class="o">-</span><span class="nv">p</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="o">/</span>
</code></pre></div>
<p>And let's create a tiny script at <code>/etc/logrotate.d/httpd-prerotate/goaccess</code>:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="nb">exec</span><span class="w"> </span>goaccess-wrapper<span class="w"> </span>/tmp/goaccess-reports
</code></pre></div></td></tr></table></div>
<p>Don't forget to make it executable:</p>
<div class="highlight"><pre><span></span><code><span class="nv">chmod</span><span class="w"> </span><span class="o">+</span><span class="nv">x</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="k">logrotate</span>.<span class="nv">d</span><span class="o">/</span><span class="nv">httpd</span><span class="o">-</span><span class="nv">prerotate</span><span class="o">/</span><span class="nv">goaccess</span>
</code></pre></div>
<p>As you can see, the only thing that this script does is to invoke the wrapper
with the right argument, ie. the output directory for the HTML reports that are
generated.</p>
<p>And that's all. Now you can just come back tomorrow, check the logs, and make
sure that the hook was executed and succeeded. For example, this kind of
command will tell you quickly if it worked:</p>
<div class="highlight"><pre><span></span><code><span class="nv">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">grep</span><span class="w"> </span><span class="k">logrotate</span>
</code></pre></div>
<h2>Integration, option B - Run GoAccess once a day, from a systemd service</h2>
<p>OK so we've just seen how to use a logrotate hook. One downside with that is
that we have to drop privileges in the wrapper script, because logrotate runs
as root, and we don't want to run GoAccess as root. Hence the rather convoluted
syntax with <code>setpriv</code>.</p>
<p>Rather than embedding this kind of thing in a wrapper script, we can instead
run the wrapper script from a <a href="https://systemd.io/">systemd</a> service, and define which user runs
the wrapper straight in the systemd service file.</p>
<h3>Introducing systemd niceties</h3>
<p>So we can create a <a href="https://www.freedesktop.org/software/systemd/man/systemd.service.html">systemd service</a>, along with a <a href="https://www.freedesktop.org/software/systemd/man/systemd.timer.html">systemd timer</a> that
fires daily. We can then set the <code>user</code> and <code>group</code> that execute the script
straight in the systemd service, and there's no need for <code>setpriv</code> anymore.
It's a bit more streamlined.</p>
<p>We can even go a bit further, and use <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">systemd parameterized units</a> (also
called templates), so that we have one service per site (instead of one service
that process all of our sites). That will simplify the wrapper script a lot,
and it also looks nicer in the logs.</p>
<p>With this approach however, it seems that we can't really run <em>exactly</em> before
the logs are rotated away, like we did in the section above. But that's OK.
What we'll do is that we'll run once a day, no matter the time, and we'll just
make sure to process both log files <code>access.log</code> and <code>access.log.1</code> (ie. the
current logs and the logs from yesterday). This way, we're sure not to miss any
line from the logs.</p>
<p>Note that GoAccess is smart enough to only consider newer entries from the log
files, and discard entries that are already in the database. In other words,
it's safe to parse the same log file more than once, GoAccess will do the right
thing. For more details see "INCREMENTAL LOG PROCESSING" from <code>man goaccess</code>.</p>
<h3>Implementation</h3>
<p>And here's how it all looks like.</p>
<p>First, a little wrapper script for GoAccess:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1"># Usage: $0 SITE DBDIR LOGDIR OUTDIR</span>
<span class="nb">set</span><span class="w"> </span>-eu
<span class="nv">SITE</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">DBDIR</span><span class="o">=</span><span class="nv">$2</span>
<span class="nv">LOGDIR</span><span class="o">=</span><span class="nv">$3</span>
<span class="nv">OUTDIR</span><span class="o">=</span><span class="nv">$4</span>
<span class="nv">LOGFILES</span><span class="o">=()</span>
<span class="k">for</span><span class="w"> </span>ext<span class="w"> </span><span class="k">in</span><span class="w"> </span>log<span class="w"> </span>log.1<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">logfile</span><span class="o">=</span><span class="s2">"</span><span class="nv">$LOGDIR</span><span class="s2">/access.</span><span class="nv">$ext</span><span class="s2">"</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nv">LOGFILES</span><span class="o">+=(</span><span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span><span class="o">)</span>
<span class="k">done</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="si">${#</span><span class="nv">LOGFILES</span><span class="p">[@]</span><span class="si">}</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"No log files in '</span><span class="nv">$LOGDIR</span><span class="s2">'"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="k">fi</span>
goaccess<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--agent-list<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--anonymize-ip<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--persist<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--restore<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--config-file<span class="w"> </span>/etc/goaccess/goaccess.conf<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--db-path<span class="w"> </span><span class="s2">"</span><span class="nv">$DBDIR</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--log-format<span class="w"> </span><span class="s2">"COMBINED"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--output<span class="w"> </span><span class="s2">"</span><span class="nv">$OUTDIR</span><span class="s2">/</span><span class="nv">$SITE</span><span class="s2">.html"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">LOGFILES</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
</code></pre></div>
<p>This wrapper does very little. Actually, the only thing it does is to check for
the existence of the two log files <code>access.log</code> and <code>access.log.1</code>, to be sure
that we don't ask GoAccess to process a file that does not exist (GoAccess
would not be happy about that).</p>
<p>Save this file under <code>/usr/local/bin/goaccess-wrapper</code>, don't forget to make it
executable:</p>
<div class="highlight"><pre><span></span><code>chmod +x /usr/local/bin/goaccess-wrapper
</code></pre></div>
<p>Then, create a systemd parameterized unit file, so that we can run this wrapper
as a systemd service. Save it under <code>/etc/systemd/system/goaccess@.service</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Update GoAccess report - %i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/var/lib/goaccess-db/%i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/var/log/apache2/%i</span>
<span class="na">ConditionPathIsDirectory</span><span class="o">=</span><span class="s">/tmp/goaccess-reports</span>
<span class="na">PartOf</span><span class="o">=</span><span class="s">goaccess.service</span>
<span class="k">[Service]</span>
<span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
<span class="na">User</span><span class="o">=</span><span class="s">goaccess</span>
<span class="na">Group</span><span class="o">=</span><span class="s">goaccess</span>
<span class="na">Nice</span><span class="o">=</span><span class="s">19</span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/goaccess-wrapper </span>\
<span class="w"> </span><span class="s">%i </span>\
<span class="w"> </span><span class="s">/var/lib/goaccess-db/%i </span>\
<span class="w"> </span><span class="s">/var/log/apache2/%i </span>\
<span class="w"> </span><span class="s">/tmp/goaccess-reports</span>
</code></pre></div>
<p>So, what is a systemd parameterized unit? It's a service to which you can pass
an argument when you enable it. The <code>%i</code> in the unit definition will be
replaced by this argument. In our case, the argument will be the name of the
site that we want to process.</p>
<p>As you can see, we use the directive <code>ConditionPathIsDirectory=</code> extensively,
so that if ever one of the required directories does not exist, the unit will
just be skipped (and marked as such in the logs). It's a graceful way to fail.</p>
<p>We run the wrapper as the user and group <code>goaccess</code>, thanks to <code>User=</code> and
<code>Group=</code>. We also use <code>Nice=</code> to give a low priority to the process.</p>
<p>At this point, it's already possible to test. Just make sure that you created a
directory for the GoAccess database:</p>
<div class="highlight"><pre><span></span><code><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
<span class="n">chown</span><span class="w"> </span><span class="n">goaccess</span><span class="p">:</span><span class="n">goaccess</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>Also make sure that the output directory exists:</p>
<div class="highlight"><pre><span></span><code>mkdir /tmp/goaccess-reports
chown goaccess:goaccess /tmp/goaccess-reports
</code></pre></div>
<p>Then reload systemd and fire the unit to see if it works:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">goaccess</span><span class="err">@</span><span class="n">SITE</span><span class="o">.</span><span class="n">service</span>
<span class="n">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">tail</span>
</code></pre></div>
<p>And that should work already.</p>
<p>As you can see, the argument, <code>SITE</code>, is passed in the <code>systemctl start</code>
command. We just append it after the <code>@</code>, in the name of the unit.</p>
<p>Now, let's create another GoAccess service file, which sole purpose is to group
all the parameterized units together, so that we can start them all in one go.
Note that we don't use a systemd target for that, because ultimately we want to
run it once a day, and that would not be possible with a target. So instead we
use a dummy oneshot service.</p>
<p>So here it is, saved under <code>/etc/systemd/system/goaccess.service</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Update GoAccess reports</span>
<span class="na">Requires</span><span class="o">=</span><span class="w"> </span>\
<span class="w"> </span><span class="s">goaccess@SITE1.service </span>\
<span class="w"> </span><span class="s">goaccess@SITE2.service</span>
<span class="k">[Service]</span>
<span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">true</span>
</code></pre></div>
<p>As you can see, we simply list the sites that we want to process in the
<code>Requires=</code> directive. In this example we have two sites named <code>SITE1</code> and
<code>SITE2</code>.</p>
<p>Let's ensure that everything is still good:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">goaccess</span><span class="o">.</span><span class="n">service</span>
<span class="n">journalctl</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">tail</span>
</code></pre></div>
<p>Check the logs, both sites <code>SITE1</code> and <code>SITE2</code> should have been processed.</p>
<p>And finally, let's create a timer, so that systemd runs <code>goaccess.service</code> once
a day. Save it under <code>/etc/systemd/system/goaccess.timer</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">Daily update of GoAccess reports</span>
<span class="k">[Timer]</span>
<span class="na">OnCalendar</span><span class="o">=</span><span class="s">daily</span>
<span class="na">RandomizedDelaySec</span><span class="o">=</span><span class="s">1h</span>
<span class="na">Persistent</span><span class="o">=</span><span class="s">true</span>
<span class="k">[Install]</span>
<span class="na">WantedBy</span><span class="o">=</span><span class="s">timers.target</span>
</code></pre></div>
<p>Finally, enable the timer:</p>
<div class="highlight"><pre><span></span><code><span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">enable</span><span class="w"> </span><span class="o">--</span><span class="n">now</span><span class="w"> </span><span class="n">goaccess</span><span class="o">.</span><span class="n">timer</span>
</code></pre></div>
<p>At this point, everything should be OK. Just come back tomorrow and check the
logs with something like:</p>
<div class="highlight"><pre><span></span><code>journalctl | grep goaccess
</code></pre></div>
<p>Last word: if you have only one site to process, of course you can simplify,
for example you can hardcode all the paths in the file <code>goaccess.service</code>
instead of using a parameterized unit. Up to you.</p>
<h2>Daily operations</h2>
<p>So in this part, we assume that you have GoAccess all setup and running, once
a day or so. Let's just go over a few things worth noting.</p>
<h3>Serve your report</h3>
<p>Up to now in this tutorial, we created the reports in <code>/tmp/goaccess-reports</code>,
but that was just for the sake of the example. You will probably want to save
your reports in a directory that is served by your web server, so that, well,
you can actually look at it in your web browser, that was the point, right?</p>
<p>So how to do that is a bit out of scope here, and I guess that if you want to
monitor your website, you already <em>have</em> a website, so you will have no trouble
serving the GoAccess HTML report.</p>
<p>However there's an important detail to be aware of: GoAccess shows all the IP
addresses of your visitors in the report. As long as the report is private it's
OK, but if ever you make your GoAccess report public, then you should
definitely invoke GoAccess with the option <code>--anonymize-ip</code>.</p>
<h3>Keep an eye on the logs</h3>
<p>In this tutorial, the reports we create, along with the GoAccess databases,
will grow bigger every day, forever. It also means that the GoAccess processing
time will grow a bit each day.</p>
<p>So maybe the first thing to do is to keep an eye on the logs, to see how long
it takes to GoAccess to do its job every day. Also, maybe you'd like to keep
an eye on the size of the GoAccess database with:</p>
<div class="highlight"><pre><span></span><code><span class="n">du</span><span class="w"> </span><span class="o">-</span><span class="n">sh</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">goaccess</span><span class="o">-</span><span class="n">db</span><span class="o">/</span><span class="n">SITE</span>
</code></pre></div>
<p>If your site has few visitors, I suspect it won't be a problem though.</p>
<p>You could also be a bit pro-active in preventing this problem in the future,
and for example you could break the reports into, say, monthly reports. Meaning
that every month, you would create a new database in a new directory, and also
start a new HTML report. This way you'd have monthly reports, and you make sure
to limit the GoAccess processing time, by limiting the database size to a
month.</p>
<p>This can be achieved very easily, by including something like <code>YEAR-MONTH</code> in
the database directory, and in the HTML report. You can handle that
automatically in the wrapper script, for example:</p>
<div class="highlight"><pre><span></span><code>sfx=$(date +'%Y-%m')
mkdir -p $DBDIR/$sfx
goaccess \
--db-path $DBDIR/$sfx \
--output "$OUTDIR/$SITE-$sfx.html" \
...
</code></pre></div>
<p>You get the idea.</p>
<h2>Further notes</h2>
<h3>Migration from older versions</h3>
<p>With the <code>--persist</code> option, GoAccess keeps all the information from the logs
in a database, so that it can re-use it later. In prior versions, GoAccess used
the <a href="https://dbdb.io/db/tokyo-cabinet">Tokyo Cabinet</a> key-value store for that. However starting
from <code>v1.4</code>, GoAccess dropped this dependency and now uses its own database format.</p>
<p>As a result, the previous database can't be used anymore, you will have to
remove it and restart from zero. At the moment there is no way to convert the
data from the old database to the new one. If you're interested, this is
discussed upstream at <a href="https://github.com/allinurl/goaccess/issues/1783">#1783</a>.</p>
<p>Another thing that changed with this new version is the name for some of the
command-line options. For example, <code>--load-from-disk</code> was dropped in favor of
<code>--restore</code>, and <code>--keep-db-files</code> became <code>--persist</code>. So you'll have to
look at the documentation a bit, and update your script(s) accordingly.</p>
<h3>Other ways to use GoAccess</h3>
<p>It's also possible to do it completely differently. You could keep GoAccess
running, pretty much like a daemon, with the <code>--real-time-html</code> option, and
have it process the logs continuously, rather than calling it on a regular
basis.</p>
<p>It's also possible to see the GoAccess report straight in the terminal, thanks
to <code>libncurses</code>, rather than creating a HTML report.</p>
<p>And much more, GoAccess is <a href="https://goaccess.io/features">packed with features</a>.</p>
<h2>Conclusion</h2>
<p>I hope that this tutorial helped some of you folks. Feel free to drop an e-mail
for comments.</p>Cute CSS hacks for your favorite sites2019-05-30T00:00:00+00:002019-05-30T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2019-05-30:/2019/05/30/cute-css-hacks-for-your-favorite-sites/<p>Do you have a wide screen? Are you annoyed by these sites that don't set any
<code>max-width</code> property on their page, hence you have to read ultra-long lines
that span all over the screen? If you're in this case, read on, you can hack
your way through!</p>
<p>Do you have a wide screen? Are you annoyed by these sites that don't set any
<code>max-width</code> property on their page, hence you have to read ultra-long lines
that span all over the screen? If you're in this case, read on, you can hack
your way through!</p>
<h2>The problem</h2>
<p>Let's start with an example. As a software engineer, I spend a fair amount of
my life reading documentation in a web browser. These days I often find myself
at <a href="https://www.freedesktop.org">freedesktop.org</a>, and here's how a typical
page looks like:</p>
<p><img alt="freedesktop orig" src="/images/css-hacks-fdo-1.png"></p>
<p>No <code>max-width</code>. The text unfolds forever, and it's a bit tedious to read.</p>
<p>It's one of these many things in life that are a bit annoying, but not enough
to call it a problem and decide to tackle it. Until now. For some mysterious
reasons, this morning I said to myself: "Enough! I won't take it anymore!"</p>
<p>Thankfully, this quest was quite short.</p>
<h2>The solution</h2>
<p>So I went looking for a way to inject some CSS into the page, so that I could
change its appearance, and set a maximum width for the text.</p>
<p>The solution came under the name of
<a href="https://addons.mozilla.org/en-US/firefox/addon/styl-us/">Stylus</a>: it's a
Firefox/Chrome/Opera extension that allows one to inject some CSS into a
particular website, based on its URL. Simple and straightforward.</p>
<p>All it takes now is to open the Web Developer Tools, inspect your favorite
page, and add some CSS trickery where needs be to make it look better.</p>
<p>For <em>freedesktop.org</em>, here's the CSS snippet I use:</p>
<div class="highlight"><pre><span></span><code><span class="nt">html</span><span class="o">,</span><span class="w"> </span><span class="nt">body</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">margin</span><span class="p">:</span><span class="w"> </span><span class="kc">auto</span><span class="p">;</span>
<span class="w"> </span><span class="k">max-width</span><span class="p">:</span><span class="w"> </span><span class="mi">56</span><span class="kt">em</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">pre</span><span class="p">.</span><span class="nc">programlisting</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="mh">#f2f2f2</span><span class="p">;</span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">16</span><span class="kt">px</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>And here's the result:</p>
<p><img alt="freedesktop with hacked css" src="/images/css-hacks-fdo-2.png"></p>
<h2>Some more ramblings to conclude</h2>
<p>So you get the idea, if you visit a website often, and if it pains your eyes,
stop suffering, and fix it. It's easy and fun, at least for those who have some
notions of HTML/CSS.</p>
<p>If you don't, you'll be delighted to learn that you can find a collection of
"CSS themes" at <a href="https://userstyles.org/">userstyles.org</a>. Who knows, maybe
someone already solved your problem.</p>
<p>I always find it fascinating to discover a new corner of the web, and a new
community of people who gathered around an unlikely topic, like theming the web
in a super hacky way that is likely to break every now and then :) So I
couldn't help but dig a bit further.</p>
<p>It turns out that there are <strong>two</strong> (at least) Firefox extensions that does the
job: <em>Stylish</em> and <em>Stylus</em>. Stylus was born as a <a href="https://add0n.com/stylus.html">fork of
Stylish</a>, and it seems that the main change was
the removal of all analytics. The Stylus extension page says it loud:</p>
<blockquote>
<p>Unlike other similar extensions, we don't find you to be all that interesting.
Your questionable browsing history should remain between you and the NSA.
Stylus collects nothing. Period.</p>
</blockquote>
<p>It's also worth noting that the Stylish extension <a href="https://www.ghacks.net/2018/07/09/stylus-sees-large-user-increase-after-stylish-removal/">was removed from Firefox and
Chrome</a>
at some point in 2018. It was found that the extension would send all your
browsing data to their server and link it to a unique identifier. The extension
seems to be back online now, but you'd better stay away from it if you care
about your privacy.</p>
<p>So don't mistake Stylus for Stylish, go for Stylus, and happy CSS hacking!</p>Building your Pelican website with schroot2019-03-16T00:00:00+00:002019-03-16T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2019-03-16:/2019/03/16/building-your-pelican-website-with-schroot/<p>Lately I moved my blog to Pelican. I really like how simple and flexible it is.
So in this post I'd like to highlight one particular aspect of my Pelican's
workflow: how to setup a Debian-based environment to build your Pelican's
website, and how to leverage Pelican's Makefile to transparently use this build
environment. Overall, this post has more to do with the Debian tooling, and
little with Pelican.</p>
<p>Lately I moved my blog to Pelican. I really like how simple and flexible it is.
So in this post I'd like to highlight one particular aspect of my Pelican's
workflow: how to setup a Debian-based environment to build your Pelican's
website, and how to leverage Pelican's Makefile to transparently use this build
environment. Overall, this post has more to do with the Debian tooling, and
little with Pelican.</p>
<h2>Introduction</h2>
<p>First thing first, why would you setup a build environment for your project?</p>
<p>Imagine that you run Debian <em>stable</em> on your machine, then you want to build
your website with a fancy theme, that requires the latest bleeding edge
features from Pelican. But hey, in Debian stable you don't have these shiny new
things, and the version of Pelican you need is only available in Debian
<em>unstable</em>. How do you handle that? Will you start messing around with apt
configuration and pinning, and try to install an unstable package in your
stable system? Wrong, please stop.</p>
<p>Another scenario, the opposite: you run Debian unstable on your system. You
have all the new shiny things, but sometimes an update of your system might
break things. What if you update, and then can't build your website because of
this or that? Will you wait a few days until another update comes and fixes
everything? How many days before you can build your blog again? Or will you
dive in the issues, and debug, which is nice and fun, but can also keep you
busy all night, and is not exactly what you wanted to do in the first place,
right?</p>
<p>So, for both of these issues, there's one simple answer: setup a build
environment for your project. The most simple way is to use a <em>chroot</em>, which
is roughly another filesystem hierarchy that you create and install somewhere,
and in which you will run your build process. A more elaborate build
environment is a <em>container</em>, which brings more isolation from the host system,
and many more features, but for something as simple as building your website on
your own machine, it's kind of overkill.</p>
<p>So that's what I want to detail here, I'll show you the way to setup and use a
chroot. There are many tools for the job, and for example Pelican's official
documentation recommends <a href="https://docs.getpelican.com/en/stable/install.html">virtualenv</a>, which is kind of the standard Python
solution for that. However, I'm not too much of a Python developer, and I'm
more familiar with the Debian tools, so I'll show you the Debian way instead.</p>
<p>Version-wise, it's 2019, we're talking about Pelican 4.x, if ever it matters.</p>
<h2>Create the chroot</h2>
<p>To create a basic, minimal Debian system, the usual command is <a href="https://manpages.debian.org/unstable/debootstrap/debootstrap.8.en.html">debootstrap</a>.
Then in order to actually use this new system, we'll use <a href="https://manpages.debian.org/unstable/schroot/schroot.1.en.html">schroot</a>. So be
sure to have these two packages installed on your machine.</p>
<div class="highlight"><pre><span></span><code>sudo apt install debootstrap schroot
</code></pre></div>
<p>It seems that the standard location for chroots is <code>/srv/chroot</code>, so let's
create our chroot there. It also seems that the traditional naming scheme for
these chroots is something like <code>SUITE-ARCH-APPLICATION</code>, at least that's what
other tools like <a href="https://manpages.debian.org/unstable/sbuild/sbuild.1.en.html">sbuild</a> do. While you're free to do whatever you want, in
this tutorial we'll try to stick to the conventions.</p>
<p>Let's go and create a <code>bookworm</code> chroot:</p>
<div class="highlight"><pre><span></span><code><span class="n">SYSROOT</span><span class="o">=/</span><span class="n">srv</span><span class="o">/</span><span class="n">chroot</span><span class="o">/</span><span class="n">bookworm</span><span class="o">-</span><span class="n">amd64</span><span class="o">-</span><span class="n">pelican</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">$</span><span class="p">{</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="p">}</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">debootstrap</span><span class="w"> </span><span class="o">--</span><span class="n">variant</span><span class="o">=</span><span class="n">minbase</span><span class="w"> </span><span class="n">bookworm</span><span class="w"> </span><span class="o">$</span><span class="p">{</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="p">}</span>
</code></pre></div>
<p>And there we are, we just installed a minimal Debian system in <code>$SYSROOT</code>, how
easy and neat is that! Just run a quick <code>ls</code> there, and see by yourself:</p>
<div class="highlight"><pre><span></span><code>ls<span class="w"> </span><span class="cp">${</span><span class="n">SYSROOT</span><span class="p">:</span><span class="err">?</span><span class="cp">}</span>
</code></pre></div>
<p>Now let's setup schroot to be able to use it. schroot will require a bit of a
configuration file that tells it how to use this chroot. This is where things
might get a bit complicated and cryptic for the newcomer.</p>
<p>So for now, stick with me, and create the schroot config file as follow:</p>
<div class="highlight"><pre><span></span><code>cat << EOF | sudo tee /etc/schroot/chroot.d/bookworm-amd64-pelican.conf
[bookworm-amd64-pelican]
users=$LOGNAME
root-users=$LOGNAME
source-users=$LOGNAME
source-root-users=$LOGNAME
type=directory
union-type=overlay
directory=/srv/chroot/bookworm-amd64-pelican
EOF
</code></pre></div>
<p>Here, we tell schroot who can use this chroot (<code>$LOGNAME</code>, it's you), as normal
user and root user. We also say where is the chroot directory located, and that
we want an <code>overlay</code>, which means that the chroot will actually be read-only,
and during operation a writable overlay will be stacked up on top of it, so
that modifications are possible, but are not saved when you exit the chroot.</p>
<p>In our case, it makes sense because we have no intention to modify the build
environment. The basic idea with a build environment is that it's identical for
every build, we don't want anything to change, we hate surprises. So we make it
read-only, but we also need a writable overlay on top of it, in case some
process might want to write things in <code>/var</code>, for example. We don't care about
these changes, so we're fine discarding this data after each build, when we
leave the chroot.</p>
<p>And now, for the last step, let's install Pelican in our chroot:</p>
<div class="highlight"><pre><span></span><code>schroot -c source:bookworm-amd64-pelican -u root -- \
bash -c "apt update && apt install --yes make pelican && apt clean"
</code></pre></div>
<p>In this command, we log into the source chroot as root, and we install the two
packages <code>make</code> and <code>pelican</code>. We also clean up after ourselves, to save a bit
of space on the disk.</p>
<p>At this point, our chroot is ready to be used. If you're new to all of this,
then read the next section, I'll try to explain a bit more how it works.</p>
<h2>A quick introduction to schroot</h2>
<p>In this part, let me try to explain a bit more how schroot works. If you're
already acquainted, you can skip this part.</p>
<p>So now that the chroot is ready, let's experiment a bit. For example, you might
want to start by listing the chroots available:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-l
chroot:bookworm-amd64-pelican
source:bookworm-amd64-pelican
</code></pre></div>
<p>Interestingly, there are two of them... So, this is due to the overlay thing
that I mentioned just above. Using the <em>regular chroot</em> (<code>chroot:</code>) gives you
the read-only version, for daily use, while the <em>source chroot</em> (<code>source:</code>)
allows you to make persistent modifications to the filesystem, for install and
maintenance basically. In effect, the <em>source chroot</em> has no overlay mounted on
top of it, and is writable.</p>
<p>So you can experiment some more. For example, to have a shell into your <em>regular
chroot</em>, run:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>chroot:bookworm-amd64-pelican
</code></pre></div>
<p>Notice that the <em>namespace</em> (eg. <code>chroot:</code> or <code>source:</code>) is optional, if you
omit it, schroot will be smart and choose the right namespace. So the command
above is equivalent to:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>bookworm-amd64-pelican
</code></pre></div>
<p>Let's try to see the overlay thing in action. For example, once inside the
chroot, you could create a file in some writable place of the filesystem.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">chroot</span><span class="p">)</span><span class="o">$</span><span class="w"> </span><span class="n">touch</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">this</span><span class="o">-</span><span class="k">is</span><span class="o">-</span><span class="n">an</span><span class="o">-</span><span class="n">empty</span><span class="o">-</span><span class="n">file</span>
<span class="p">(</span><span class="n">chroot</span><span class="p">)</span><span class="o">$</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">tmp</span>
<span class="n">this</span><span class="o">-</span><span class="k">is</span><span class="o">-</span><span class="n">an</span><span class="o">-</span><span class="n">empty</span><span class="o">-</span><span class="n">file</span>
</code></pre></div>
<p>Then log out with <code><Ctrl-D></code>, and log in again. Have a look in <code>/var/tmp</code>: the
file is gone. The overlay in action.</p>
<p>Now, there's a bit more to that. If you look into the current directory, you
will see that you're not within any isolated environment, you can still see all
your files, for example:</p>
<div class="highlight"><pre><span></span><code>(chroot)$ pwd
/home/arno/my-pelican-blog
(chroot)$ ls
content Makefile pelicanconf.py ...
</code></pre></div>
<p>Not only are all your files available in the chroot, you can also create new
files, delete existing ones, and so on. It doesn't even matter if you're inside
or outside the chroot, and the reason is simple: by default, schroot will mount
the <code>/home</code> directory inside the chroot, so that you can access all your files
transparently. For more details, just type <code>mount</code> inside the chroot, and see
what's listed.</p>
<p>So, this default of schroot is actually what makes it super convenient to use,
because you don't have to bother about bind-mounting every directory you care
about inside the chroot, which is actually quite annoying. Having <code>/home</code>
directly available saves time, because what you want to isolate are the tools
you need for the job (so basically <code>/usr</code>), but what you need is the data you
work with (which is supposedly in <code>/home</code>). And schroot gives you just that,
out of the box, without having to fiddle too much with the configuration.</p>
<p>If you're not familiar with chroots, containers, VMs, or more generally bind
mounts, maybe it's still very confusing. But you'd better get used to it, as
virtual environment are very standard in software development nowadays.</p>
<p>But anyway, let's get back to the topic. How to make use of this chroot to
build our Pelican website?</p>
<h2>Chroot usage with Pelican</h2>
<p>Pelican provides two helpers to build and manage your project: one is a
<code>Makefile</code>, and the other is a Python script called <code>fabfile.py</code>. As I said
before, I'm not really a seasoned Pythonista, but it happens that I'm quite a
fan of <a href="https://manpages.debian.org/unstable/make-guile/make.1.en.html">make</a>, hence I will focus on the Makefile for this part.</p>
<p>So, here's how your daily blogging workflow might look like, now that
everything is in place.</p>
<p>Open a first terminal, and edit your blog posts with your favorite editor:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>nano<span class="w"> </span>content/bla-bla-bla.md
</code></pre></div>
<p>Then open a second terminal, enter the chroot, build your blog and serve it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>schroot<span class="w"> </span>-c<span class="w"> </span>bookworm-amd64-pelican
<span class="o">(</span>chroot<span class="o">)</span>$<span class="w"> </span>make<span class="w"> </span>html
<span class="o">(</span>chroot<span class="o">)</span>$<span class="w"> </span>make<span class="w"> </span>serve
</code></pre></div>
<p>And finally, open your web browser at <a href="http://localhost:8000">http://localhost:8000</a> and enjoy
yourself.</p>
<p>This is easy and neat, but guess what, we can even do better. Open the Makefile
and have a look at the very first lines:</p>
<div class="highlight"><pre><span></span><code>PY?=python3
PELICAN?=pelican
</code></pre></div>
<p>It turns out that the Pelican developers know how to write Makefiles, and they
were kind enough to allow their users to easily override the default commands.
In our case, it means that we can just replace these two lines with these ones:</p>
<div class="highlight"><pre><span></span><code>PY?=schroot -c bookworm-amd64-pelican -- python3
PELICAN?=schroot -c bookworm-amd64-pelican -- pelican
</code></pre></div>
<p>And after these changes, we can now completely forget about the chroot, and
simply type <code>make html</code> and <code>make serve</code>. The chroot invocation is now handled
automatically in the Makefile. How neat!</p>
<h2>Maintenance</h2>
<p>So you might want to update your chroot from time to time, and you do that with
<a href="https://manpages.debian.org/unstable/apt/apt.8.en.html">apt</a>, like for any Debian system. Remember the distinction between <em>regular
chroot</em> and <em>source chroot</em> due to the overlay? If you want to actually modify
your chroot, what you want is the <em>source chroot</em>. And here's the one-liner:</p>
<div class="highlight"><pre><span></span><code>schroot -c source:$PROJECT -u root -- \
bash -c "apt update && apt --yes dist-upgrade && apt clean"
</code></pre></div>
<p>If one day you stop using it, just delete the chroot directory, and the schroot
configuration file:</p>
<div class="highlight"><pre><span></span><code>sudo rm /etc/schroot/chroot.d/bookworm-amd64-pelican.conf
sudo rm -fr /srv/chroot/bookworm-amd64-pelican
</code></pre></div>
<p>And that's about it.</p>
<h2>Last words</h2>
<p>The general idea of keeping your build environments separated from your host
environment is a very important one if you're a software developer, and
especially if you're doing consulting and working on several projects at the
same time. Installing all the build tools and dependencies directly on your
system can work at the beginning, but it won't get you very far.</p>
<p><code>schroot</code> is only one of the many tools that exist to address this, and I
<em>think</em> it's Debian specific. Maybe you have never heard of it, as chroots in
general are far from the container hype, even though they have some common
use-cases. schroot has been around for a while, it works great, it's simple,
flexible, what else? Just give it a try!</p>
<p>It's also well integrated with other Debian tools, for example you might use it
through <code>sbuild</code> to build Debian packages (another daily task that is better
done in a dedicated build environment), so I think it's a tool worth knowing if
you're doing some Debian work.</p>
<p>That's about it, in the end it was mostly a post about schroot, hope you liked
it.</p>The docker.io Debian package is back to life2018-07-04T00:00:00+00:002018-07-04T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2018-07-04:/2018/07/04/the-dockerio-debian-package-is-back-to-life/<p>Last week, a new version of <code>docker.io</code>, the Docker package provided by Debian,
was uploaded to <a href="https://packages.debian.org/sid/docker.io">Debian Unstable</a>.
Quickly afterward, the package moved to <a href="https://packages.debian.org/testing/docker.io">Debian Testing</a>.
And this is good news for Debian users, as before that the package was more
or less abandoned in "unstable", and the future …</p><p>Last week, a new version of <code>docker.io</code>, the Docker package provided by Debian,
was uploaded to <a href="https://packages.debian.org/sid/docker.io">Debian Unstable</a>.
Quickly afterward, the package moved to <a href="https://packages.debian.org/testing/docker.io">Debian Testing</a>.
And this is good news for Debian users, as before that the package was more
or less abandoned in "unstable", and the future was uncertain.</p>
<p>The most striking fact about this change: it's the <em>first time in two years</em> that
<em>docker.io</em> has migrated to "testing". Another interesting fact is that, version-wise,
the package is moving from <code>1.13.1</code> from early 2017 to version <code>18.03</code> from
March 2018: that's a <em>one-year leap forward</em>.</p>
<p>Let me give you a very rough summary of how things came to be. I personally
started to work on that early in 2018. I joined the <em>Debian Go Packaging Team</em>
and I started to work on the many, many Docker dependencies that needed to be
updated in order to update the Docker package itself. I could get some of this
work uploaded to Debian, but ultimately I was a bit stuck on how to solve the
circular dependencies that plague the Docker package. This is where another
Debian Developer, Dmitry Smirnov, jumped in. We discussed the current status
and issues, and then he basically did all the job, from updating the package to
tackling all the long-time opened bugs.</p>
<p>This is for the short story, let me know give you some more details.</p>
<h2>The Docker package in Debian</h2>
<p>To better understand why this update of the <code>docker.io</code> package is such a good
news, let's have quick look at the current Debian offer:</p>
<div class="highlight"><pre><span></span><code>rmadison -u debian docker.io
</code></pre></div>
<p>If you're running Debian 8 Jessie, you can install Docker 1.6.2, through
backports. This version was released on May 14, 2015. That's 3 years old, but
Debian Jessie is fairly old as well.</p>
<p>If you're running Debian 9 Stretch (ie. Debian stable), then you have no
install candidate. No-thing. The current Debian doesn't provide any package for
Docker. That's a bit sad.</p>
<p>What's even more sad is that for quite a while, looking into Debian unstable
didn't look promising either. There used to be a package there, but it had bugs
that prevented it to migrate to Debian testing. This package was stuck at the
version <code>1.13.1</code>, released on Feb 8, 2017. Looking at the git history, there
was not much happening.</p>
<p>As for the reason for this sad state of things, I can only guess. Packaging
Docker is a tedious work, mainly due to a very big dependency tree. After
handling all these dependencies, there are other issues to tackle, some related
to Go packaging itself, and others due to Docker release process and
development workflow. In the end, it's quite difficult to find the right
approach to package Docker, and it's easy to make mistakes that cost hours of
works. I did this kind of mistakes. More than once.</p>
<p>So packaging Docker is not for the faint of heart, and maybe it's too much of
a burden for one developer alone. There was a <code>docker-maint</code> mailing list that
suggests an attempt to coordinate the effort, however this list was already dead
by the time I found it. It looks like the people involved walked away.</p>
<p>Another explanation for the disinterest in the Docker package could be that
Docker itself already provides a Debian package on docker.com. One can
always fall back to this solution, so why bothering with the extra-work of
doing a Debian package proper?</p>
<p>That's what the next part is about!</p>
<h2>Docker.io vs Docker-ce</h2>
<p>You have two options to install Docker on Debian: you can get the package from
docker.com (this package is named <code>docker-ce</code>), or you can get it from the
Debian repositories (this package is named <code>docker.io</code>). You can rebuild both
of these packages from source: for <em>docker-ce</em> you can fetch the source code
with git (it includes the packaging files), and for <em>docker.io</em> you can just
get the source package with <em>apt</em>, like for every other Debian package.</p>
<p>So what's the difference between these two packages?</p>
<p>No suspense, straight answer: what differs is the build process, and mostly,
the way dependencies are handled.</p>
<p>Docker is written in Go, and Golang comes with some tooling that allows
applications to keep a local copy of their dependencies in their source tree.
In Go-talk, this is called <em>vendoring</em>. Docker makes heavy use of that (like
many other Go applications), which means that the code is more or less
self-contained. You can build Docker without having to solve external
dependencies, as everything needed is already in-tree.</p>
<p>That's how the <code>docker-ce</code> package provided by Docker is built, and that's what
makes the packaging files for this package trivial. You can look at these files
at <a href="https://github.com/docker/docker-ce/tree/master/components/packaging/deb">https://github.com/docker/docker-ce/tree/master/components/packaging/deb</a>.
So everything is in-tree, there's almost no external build dependency, and
hence it's real easy for Docker to provide a new package for <code>docker-ce</code> every
month.</p>
<p>On the other hand, the <code>docker.io</code> package provided by Debian takes a
completely different approach: Docker is built against the libraries that are
packaged in Debian, instead of using the local copies that are present in the
Docker source tree. So if Docker is using <em>libABC</em> version 1.0, then it has a
build dependency on <em>libABC</em>. You can have a look at the current build
dependencies at <a href="https://salsa.debian.org/docker-team/docker/blob/master/debian/control">https://salsa.debian.org/docker-team/docker/blob/master/debian/control</a>.</p>
<p>There are more than 100 dependencies there, and that's one reason why the
Debian package is quite time-consuming to maintain. To give you a rough
estimation, in order to get the current "stable" release of Docker to Debian
"unstable", it took up to 40 uploads of related packages to stabilize the
dependency tree.</p>
<p>It's quite an effort. And once again, why bother? For this part I'll quote
Dmitry as he puts it better than me:</p>
<blockquote>
<p>Debian cares about reusable libraries, and packaging them individually allows to
build software from tested components, as Golang runs no tests for vendored
libraries. It is a mind blowing argument given that perhaps there is more code
in "vendor" than in the source tree.</p>
<p>Private vendoring have all disadvantages of <a href="https://wiki.debian.org/StaticLinking">static linking</a>,
making it impossible to provide meaningful security support. On top of that, it
is easy to lose control of vendored tree; it is difficult to track changes in
vendored dependencies and there is no incentive to upgrade vendored components.</p>
</blockquote>
<p>That's about it, whether it matters is up to you and your use-case. But it's
definitely something you should know about if you want to make an informed
decision on which package you're about to install and use.</p>
<p>To finish with this article, I'd like to give more details on the packaging of
<em>docker.io</em>, and what was done to get this new version in Debian.</p>
<h2>Under the hood of the docker.io package</h2>
<p>Let's have a brief overview of the difficulties we had to tackle while packaging
this new version of Docker.</p>
<p>The most outstanding one is circular dependencies. It's especially present in
the top-level dependencies of Docker: <code>docker/swarmkit</code>, <code>docker/libnetwork</code>,
<code>containerd</code>... All of these are Docker build dependencies, and all of these
depend on Docker to build. Good luck with that ;)</p>
<p>To solve this issue, the new <em>docker.io</em> package leverages MUT (Multiple
Upstream Tarball) to have these different components downloaded and built all
at once, instead of being packaged separately. In this particular case it
definitely makes sense, as we're really talking about different parts of
Docker. Even if they live in different git repositories, these components are
not standalone libraries, and there's absolutely no good reason to package them
separately.</p>
<p>Another issue with Docker is "micro-packaging", ie. wasting time packaging
small git repositories that, in the end, are only used by one application
(Docker in our case). This issue is quite interesting, really. Let me try to
explain.</p>
<p>Golang makes it extremely easy to split a codebase among several git
repositories. It's so easy that some projects (Docker in our case) do it
extensively, as part of their daily workflow. And in the end, at a first glance
you can't really say if a dependency of Docker is really a standalone project
(that would require a proper packaging), or only just a part of Docker
codebase, that happens to live in a different git repository. In this second
case, there's really no reason to package it independently of Docker.</p>
<p>As a packager, if you're not a bit careful, you can easily fall in this trap,
and start packaging every single dependency without thinking: that's
"micro-packaging". It's bad in the sense that it increases the maintenance cost
on the long-run, and doesn't bring any benefit. As I said before, <em>docker.io</em> has
currently 100+ dependencies, and probably a few of them fall in this category.</p>
<p>While working on this new version of <em>docker.io</em>, we decided to stop packaging
such dependencies. The guideline is that if a dependency has no <a href="https://semver.org/">semantic
versioning</a>, and no consumer other than Docker, then it's
not a library, it's just a part of Docker codebase.</p>
<p>Even though some tools like <a href="https://people.debian.org/%7Estapelberg/2015/07/27/dh-make-golang.html">dh-make-golang</a>
make it very easy to package simple Go packages, it doesn't mean that everything
should be packaged. Understanding that, and taking a bit of time to think before
packaging, is the key to successful Go packaging!</p>
<h2>Last words</h2>
<p>I could go on for a while on the technical details, there's a lot to say, but
let's not bore you to death, so that's it. I hope by now you understand that:</p>
<ol>
<li>There's now an up-to-date <code>docker.io</code> package in Debian.</li>
<li><code>docker.io</code> and <code>docker-ce</code> both give you a Docker binary, but through a
very different build process.</li>
<li>Maintaining the <code>docker.io</code> package is not an easy task.</li>
</ol>
<p>If you care about having a Docker package in Debian, feel free to try it out,
and feel free to join the maintenance effort!</p>
<p>Let's finish with a few credits. I've been working on that topic, albeit
sparingly, for the last 4 months, thanks to the support of
<a href="https://www.collabora.com/">Collabora</a>. As for Dmitry Smirnov, the work he did
on the <em>docker.io</em> package represents a three weeks, full-time effort, which
was sponsored by <a href="https://raid6.com.au">Libre Solutions Pty Ltd</a>.</p>
<p>I'd like to thank the <a href="https://go-team.pages.debian.net/">Debian Go Packaging
Team</a> for their support, and also the
reviewers of this article, namely Dmitry Smirnov and Héctor Orón Martínez.</p>
<p>Last but not least, I will attend <a href="https://debconf18.debconf.org/">DebConf18</a>
in Taiwan, where I will give a speak on this topic. There's also a BoF on Go
Packaging planned.</p>
<p>See you there!</p>GRUB Recovery for UEFI-GPT-LUKS-LVM2018-03-26T00:00:00+00:002018-03-26T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2018-03-26:/2018/03/26/grub-recovery-for-uefi-gpt-luks-lvm/<p>Yet another GRUB recovery article... Yep, I can't deny, that's what it's about!</p>
<p>Yet another GRUB recovery article... Yep, I can't deny, that's what it's about!</p>
<h2>Introduction</h2>
<p><a href="https://www.gnu.org/software/grub/">GRUB</a> is a bootloader, ie. a piece of
software that gets loaded very early on when you boot your machine, and which
is in charge of booting the Operating System. GRUB is the bootloader of choice
for many Linux distributions.</p>
<p>From time to time, it happens that the poor GRUB is stepped over by some very
rude operating systems. In my case, I needed the infamous Windows, and as I didn't
want to install it, I googled my way to a tutorial that explained how to
create a bootable USB stick with Windows. Great, I thought, and it worked!
Except that after doing my things in Windows, I restarted and found that the
machine couldn't boot anymore. GRUB had been wiped away, silently, without
any warning, by the infamous Windows...</p>
<p>If you find yourself in a similar situation, fear not! We can always boot a
<a href="https://en.wikipedia.org/wiki/Live_CD">live system</a> from a USB stick, and
repair the boot partition (ie. re-install GRUB) from there. It's fairly easy,
it's just a matter of knowing the right commands.</p>
<p>There's already many tutorial of this kind available, each one slightly
different from each other depending on the particular hardware and software
details of the person writing the article. And same goes with this article, so
let me tell you immediately about my configuration:</p>
<ul>
<li>We're talking about recovering GRUB with a GParted Live CD.</li>
<li>This happens on a UEFI machine.</li>
<li>The disk is partitioned with GPT.</li>
<li>The disk happens to be a SSD.</li>
<li>The whole disk is encrypted with LUKS.</li>
<li>The LUKS partition is "partitioned" using LVM.</li>
</ul>
<p>If you have no idea what I'm talking about, you're probably on the wrong page.
But if you understand what I mean, and even if your configuration is a bit
different, read on. I'll do my best to explain what's going on so that you
can adapt these commands to your particular setup.</p>
<p><strong>A word of caution though</strong>: we're dealing with the disks and filesystems here,
so if you screw up you can lose data. I will just assume that you understand
what's going on, and that you're able to adapt the commands below to your own
setup. Mostly, it's about changing the device names here and there, and I can't
do that for you.</p>
<h2>At first, let's understand how things work</h2>
<p>So let's get started with some explanations. If you already know how things
work and just want to copy/paste some commands, skip this part. But if you
want to understand a bit what we're doing here, read on.</p>
<p>The commands in this part are meant to be run when your machine is up and
running, the purpose it to look at how things are setup. So of course, if right
now your machine can't boot, you can't run these commands. I mean, don't run it
from a live system, it won't give you the expected result.</p>
<h4>Overview of the disk partitions</h4>
<p>Let's have an overview of the partitions present on the disk, the best command
for that being <code>lsblk</code>. On my laptop, here's the output:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">lsblk</span>
<span class="n">NAME</span><span class="w"> </span><span class="nl">MAJ</span><span class="p">:</span><span class="nf">MIN</span><span class="w"> </span><span class="n">RM</span><span class="w"> </span><span class="k">SIZE</span><span class="w"> </span><span class="n">RO</span><span class="w"> </span><span class="n">TYPE</span><span class="w"> </span><span class="n">MOUNTPOINT</span>
<span class="n">nvme0n1</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">477</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">disk</span><span class="w"> </span>
<span class="err">├─</span><span class="n">nvme0n1p1</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">512</span><span class="n">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span><span class="o">/</span><span class="n">boot</span><span class="o">/</span><span class="n">efi</span>
<span class="err">├─</span><span class="n">nvme0n1p2</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">244</span><span class="n">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span><span class="o">/</span><span class="n">boot</span>
<span class="err">└─</span><span class="n">nvme0n1p3</span><span class="w"> </span><span class="mi">259</span><span class="err">:</span><span class="mi">3</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">476.2</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">part</span><span class="w"> </span>
<span class="w"> </span><span class="err">└─</span><span class="n">nvme0n1p3_crypt</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">476.2</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">crypt</span><span class="w"> </span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">root</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">23.3</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="nf">var</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">2</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">40</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span><span class="nf">var</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">swap_1</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">3</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mf">15.9</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">[</span><span class="n">SWAP</span><span class="o">]</span>
<span class="w"> </span><span class="err">├─</span><span class="n">debian</span><span class="o">--</span><span class="n">vg</span><span class="o">-</span><span class="n">tmp</span><span class="w"> </span><span class="mi">253</span><span class="err">:</span><span class="mi">4</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">10</span><span class="n">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">lvm</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span>
<span class="w"> </span><span class="err">└─</span><span class="n">debian</span><span class="c1">--vg-home 253:5 0 385.9G 0 lvm /home</span>
</code></pre></div>
<p>As I explained above, I have an UEFI machine, the hard drive is a SSD,
partitioned with GPT, and the disk is encrypted. You can see all of that
above, ie:</p>
<ul>
<li><code>nvme0n1</code> is what you get if you have a <a href="https://en.wikipedia.org/wiki/Solid-state_drive">SSD</a>
drive. For comparison, a more ancient <a href="https://en.wikipedia.org/wiki/Hard_disk_drive">HDD</a>
would be named something like <code>sda</code>. <code>nvme</code> stands for Non-Volatile Memory
Express (roughly), and <code>n1</code> stands for disk number one. Additionally,
the suffix <code>p1</code>, <code>p2</code> and so on is appended to give the partition number.</li>
<li><code>nvme0n1p1</code> and <code>nvme0n1p2</code> are the two partitions involved in the boot
process.</li>
<li><code>nvme0n1p3</code> is the <a href="https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup">LUKS</a>
encrypted partition that takes up the whole disk. When the partition is
unlocked, it is available unencrypted at <code>nvme0n1p3_crypt</code>, thanks to the
<a href="https://en.wikipedia.org/wiki/Device_mapper">Linux device mapper</a>.</li>
<li>Additionally, all the <code>debian--</code> devices are the different logical volumes
presents on the partition. If you're familiar with Linux distributions, you
can recognize the usual partitions <code>/</code>, <code>/home</code> and the swap partition. I
also decided to have a separate <code>/tmp</code> and <code>/var</code> partitions, but that's a
matter of taste.</li>
</ul>
<p>The logical volumes are not exactly partitions the way we know it. Ultimately,
they appear to the user as <em>block devices</em>, however they are <em>virtual block devices</em>,
you won't find them directly in <code>/dev</code>, but instead in the sub-directory
<code>/dev/mapper</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>/dev/mapper/
control
nvme0n1p3_crypt
debian--vg-home
debian--vg-root
debian--vg-swap_1
debian--vg-tmp
debian--vg-var
</code></pre></div>
<p>In case you wonder, such layout is nothing fancy. It's what you get with the
Debian Buster installer, assuming that you select the whole disk encryption
during the install process.</p>
<p>Ok, let's dive a bit more into details.</p>
<h4>The boot partition</h4>
<p>There are two partitions involved in the boot process.</p>
<p>The first partition of the disk, <code>nvme0n1p1</code> is the <a href="https://en.wikipedia.org/wiki/EFI_system_partition">EFI System Partition</a>.
At boot time, the UEFI firmware loads this partition and will boot your system
from there. In our case, this is where GRUB lives.</p>
<p>On a Debian system, this partition is usually mounted at <code>/boot/efi</code>, and you
can have a look at what's inside.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>tree<span class="w"> </span>/boot/efi
/boot/efi
└──<span class="w"> </span>EFI
<span class="w"> </span>├──<span class="w"> </span>debian
<span class="w"> </span>│<span class="w"> </span>└──<span class="w"> </span>grubx64.efi
...
</code></pre></div>
<p>Indeed, GRUB lives there, among other things.</p>
<h4>The kernel and initrd partition</h4>
<p>I mentioned that my disk is encrypted, right? It means that at boot time, I
need to enter my password in order to decrypt the operating system. The
decryption can be implemented in different places, and actually recent versions
of GRUB can handle that. However, in the setup I describe here, this is not the
case. GRUB will just boot and <em>unencrypted</em> kernel, and it's the kernel who's in
charge of decrypting the encrypted partition.</p>
<p>It means that we need an unencrypted partition somewhere to store the kernel
and the initrd: this is the purpose of the second partition: <code>nvme0n1p2</code>.</p>
<p>On a Debian system, this partition is usually mounted at <code>/boot</code>. Once again,
the best is to have a look and convince yourself.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>/boot
config-4.13.0-1-amd64
config-4.14.0-3-amd64
efi
grub
initrd.img-4.13.0-1-amd64
initrd.img-4.14.0-3-amd64
lost+found
System.map-4.13.0-1-amd64
System.map-4.14.0-3-amd64
vmlinuz-4.13.0-1-amd64
vmlinuz-4.14.0-3-amd64
</code></pre></div>
<p><code>vmlinuz</code> is the usual filename for the Linux kernel, while <code>initrd.img</code> is the
initial ramdisk, ie. a temporary, minimal root filesystem that gets loaded by
the kernel at first, and perform some initial setup before mounting your "real"
root filesystem.</p>
<h4>Some more commands</h4>
<p>Another good command to have a hierachical view of the mount points is <code>findmnt</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>findmnt
...
├─/boot<span class="w"> </span>/dev/nvme0n1p2<span class="w"> </span>ext2<span class="w"> </span>rw,relatime,...
│<span class="w"> </span>└─/boot/efi<span class="w"> </span>/dev/nvme0n1p1<span class="w"> </span>vfat<span class="w"> </span>rw,relatime,...
...
</code></pre></div>
<p>The output is interesting, because it shows clearly how the <code>efi</code> mount point
is nested inside the <code>/boot</code> mount point. This is interesting because during
the recovery process we will have to reproduce this layout.</p>
<p>At last, you can also have some useful information with <code>fdisk</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>fdisk<span class="w"> </span>-l
Disk<span class="w"> </span>/dev/nvme0n1:<span class="w"> </span><span class="m">477</span><span class="w"> </span>GiB,<span class="w"> </span><span class="m">512110190592</span><span class="w"> </span>bytes,<span class="w"> </span><span class="m">1000215216</span><span class="w"> </span>sectors
Units:<span class="w"> </span>...
Sector<span class="w"> </span>size<span class="w"> </span>...
I/O<span class="w"> </span>size<span class="w"> </span>...
Disklabel<span class="w"> </span>type:<span class="w"> </span>gpt
Disk<span class="w"> </span>identifier:<span class="w"> </span>...
Device<span class="w"> </span>Start<span class="w"> </span>End<span class="w"> </span>Sectors<span class="w"> </span>Size<span class="w"> </span>Type
/dev/nvme0n1p1<span class="w"> </span><span class="m">2048</span><span class="w"> </span><span class="m">1050623</span><span class="w"> </span><span class="m">1048576</span><span class="w"> </span>512M<span class="w"> </span>EFI<span class="w"> </span>System
/dev/nvme0n1p2<span class="w"> </span><span class="m">1050624</span><span class="w"> </span><span class="m">1550335</span><span class="w"> </span><span class="m">499712</span><span class="w"> </span>244M<span class="w"> </span>Linux<span class="w"> </span>filesystem
/dev/nvme0n1p3<span class="w"> </span><span class="m">1550336</span><span class="w"> </span><span class="m">1000214527</span><span class="w"> </span><span class="m">998664192</span><span class="w"> </span><span class="m">476</span>.2G<span class="w"> </span>Linux<span class="w"> </span>filesystem
...
</code></pre></div>
<p>One interesting detail here is the <code>Disklabel type</code>, which should be <code>gpt</code> if your
disk is partitioned with <a href="https://en.wikipedia.org/wiki/GUID_Partition_Table">GPT</a>,
or <code>dos</code> otherwise.</p>
<p>Ok, I hope you get a good overview of things now, time to get started with the
real thing: recovering GRUB.</p>
<h2>Get yourself a GParted Live CD</h2>
<p>Let's get started with the recovery process!</p>
<p>The first thing to do is to prepare a USB stick (or a CD-ROM if you're a bit
old-fashioned) with a live system of some kind. I usually go with a
<a href="https://gparted.org/livecd.php">GParted Live CD/USB</a>
for this task, as it's been around forever and always did the job. But there
are other alternatives, and you might want to give a try to
<a href="http://www.rodsbooks.com/refind">ReFind</a> if you want to try out new stuff.</p>
<p>So, let's visit <a href="https://gparted.org/download.php">https://gparted.org/download.php</a>, download an iso, and install
that on a USB stick. I won't cover these details, it's nothing complicated and
there's plenty of explanation available on the GParted website already.</p>
<p>When you're done, plug the USB stick, reboot your machine, get a boot prompt,
and then choose to boot from the USB stick.</p>
<p><strong>Be sure that your machine is configured to boot in EFI mode</strong>, and not in
legacy BIOS mode. If I'm not mistaken, GRUB will detect that later on, and will
install itself accordingly. So if you boot in legacy BIOS mode, then grub will
install itself the legacy way, and you will not be able to boot it through UEFI.</p>
<h2>Your first steps in GParted</h2>
<p>The desktop looks a bit old fashioned, it's not super pretty, but that's not
the point. The point is that GParted comes furnished with all the tools you
need to deal with disks and filesystems, all of that up-to-date and well
maintained.</p>
<p>There are a few icons on the desktop, one is about changing the display
resolution, and it can be very useful. Other than that, just right click
somewhere on the desktop, and choose to open a root terminal.</p>
<p>And that's pretty much all we need to know.</p>
<h2>The recovery process</h2>
<p>Now, it's just a matter of <em>chrooting</em> properly.
<a href="https://en.wikipedia.org/wiki/Chroot">chroot</a> (literally "change root") is the
magic of executing a process in another root directory. Right now, if you
type <code>ls /</code> in your terminal, you will see the current root directory of the
system. And which system? The live system, the one you booted from the USB
stick.</p>
<p>However, in order to recover GRUB, we will need to run some commands in the root
directory of your machine, the one you can't boot at the moment. The reason is
that GRUB needs to access resources in several places of this hierarchy. So the
right way to achieve that is to chroot. And we need a bit of work to prepare
the environment in which we will chroot, it's a bit more complicated than just
doing <code>cd</code> somewhere, or just remounting a directory.</p>
<p>So, yep, that's where all the difficulty is, simply because it's the kind of
commands we never have to run in the daily life. And there are different
elements in the picture, we will have to deal with the encryption, and the
logical volumes, and all the rest.</p>
<p>So, let me guide you in the process.</p>
<h4>Mapping every devices</h4>
<p>Please remember that I told you to open a <em>root</em> terminal!</p>
<div class="highlight"><pre><span></span><code># whoami
root
</code></pre></div>
<p>A lot of magic will happen in <code>/dev/mapper</code>, so you might as well have a look
right now.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>Yep, right now it's more or less empty, but we will populate it.</p>
<p>The first thing we need to handle is the encryption: we need to unlock our
encrypted partition before we can work with it.</p>
<p>This is actually super easy, as long as you know the command :)</p>
<div class="highlight"><pre><span></span><code># cryptsetup luksOpen /dev/nvme0n1p3 cryptdisk
</code></pre></div>
<p>And have a look at <code>/dev/mapper</code> immediately to see what happened.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>Done, so from now on your encrypted partition is unlocked, let's move on to the
next step, which is to activate the LVM partitions. I'm not sure this is always
needed, but it doesn't hurt either.</p>
<p>Just enter these LVM commands.</p>
<div class="highlight"><pre><span></span><code># vgscan
# vgchange -ay
</code></pre></div>
<p>And once again, be curious, check what's up in <code>/dev/mapper</code>.</p>
<div class="highlight"><pre><span></span><code># ls /dev/mapper
</code></pre></div>
<p>As you can see, all your partitions are now visible there. That's great, we can
start to mount them now.</p>
<h4>Preparing the chroot</h4>
<p>We will prepare the chroot in <code>/mnt</code>. Right now, it's empty.</p>
<div class="highlight"><pre><span></span><code># ls /mnt
</code></pre></div>
<p>We need to mount the root partition first.</p>
<div class="highlight"><pre><span></span><code># mount /dev/mapper/debian--vg-root /mnt
# ls /mnt
</code></pre></div>
<p>If you read the explanations above, then you know that the boot partitions
are supposed to be mounted in <code>/boot</code>. Since one mount point is nested into
the other, the order for these commands matter.</p>
<div class="highlight"><pre><span></span><code># mount /dev/nvme0n1p2 /mnt/boot
# mount /dev/nvme0n1p1 /mnt/boot/efi
</code></pre></div>
<p>Additionally, other system partitions need to be mounted, if any. So if you
have a separate <code>/var</code> partition, it's time to mount it.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mount /dev/mapper/debian--vg-var /mnt/var</span>
</code></pre></div>
<p>We also need to bind mount all the system things, like the /dev hierarchy and
other pseudo filesystems. These are some special directories that contain some
runtime things that the system needs to operate properly.</p>
<div class="highlight"><pre><span></span><code># mount --bind /dev /mnt/dev
# mount --bind /dev/pts /mnt/dev/pts
# mount --bind /proc /mnt/proc
# mount --bind /run /mnt/run
# mount --bind /sys /mnt/sys
</code></pre></div>
<p>At last, we need to mount the EFI configurations variables, that are exposed by
the kernel via <em>efivarfs</em> (assuming that your systemd was booted in EFI mode).</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars</span>
</code></pre></div>
<p>If ever this line did not work, try:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># modprobe efivarfs</span>
<span class="c1"># mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars</span>
</code></pre></div>
<p>If it still doesn't work, maybe you simply didn't boot in EFI mode?</p>
<h4>Entering the chroot and recovering GRUB</h4>
<p>At this point, everything is ready! Just enter the chroot, ie. "change root
directory":</p>
<div class="highlight"><pre><span></span><code># chroot /mnt
</code></pre></div>
<p>As you noticed, your prompt changed. That's the indication that you are now in
a different environment.</p>
<p>It's now super easy. We want to re-install GRUB, and we just need two commands
for that.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">grub</span><span class="o">-</span><span class="nv">install</span><span class="w"> </span><span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">nvme0n1</span>
<span class="nv">Installing</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">x86_64</span><span class="o">-</span><span class="nv">efi</span><span class="w"> </span><span class="nv">platform</span>.
<span class="nv">Installation</span><span class="w"> </span><span class="nv">finished</span>.<span class="w"> </span><span class="nv">No</span><span class="w"> </span><span class="nv">error</span><span class="w"> </span><span class="nv">reported</span>.
#<span class="w"> </span><span class="nv">update</span><span class="o">-</span><span class="nv">grub</span>
<span class="nv">Generating</span><span class="w"> </span><span class="nv">grub</span><span class="w"> </span><span class="nv">configuration</span><span class="w"> </span><span class="nv">file</span>...
<span class="nv">Found</span><span class="w"> </span><span class="nv">background</span><span class="w"> </span><span class="nv">image</span>:<span class="w"> </span><span class="nv">blablabla</span>
<span class="nv">Found</span><span class="w"> </span><span class="nv">linux</span><span class="w"> </span><span class="nv">image</span>:<span class="w"> </span><span class="o">/</span><span class="nv">boot</span><span class="o">/</span><span class="nv">vmlinuz</span><span class="o">-</span><span class="mi">4</span>.<span class="mi">14</span>.<span class="mi">0</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="nv">amd64</span>
...
<span class="nv">Adding</span><span class="w"> </span><span class="nv">boot</span><span class="w"> </span><span class="nv">menu</span><span class="w"> </span><span class="nv">entry</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">EFI</span><span class="w"> </span><span class="nv">firmware</span><span class="w"> </span><span class="nv">configuration</span>
<span class="nv">done</span>
</code></pre></div>
<p>If you see this warning along the way, fear not, it's harmless.</p>
<div class="highlight"><pre><span></span><code><span class="n">WARNING</span><span class="o">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">connect</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">lvmetad</span><span class="o">.</span><span class="w"> </span><span class="n">Falling</span><span class="w"> </span><span class="n">back</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">scanning</span><span class="o">.</span>
</code></pre></div>
<p>Done! Now you want to exit the chroot.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="k">exit</span>
</code></pre></div>
<p>See how your prompt changed? You're out of the chroot, back to the real world!
Time to reboot and see if GRUB is happy again.</p>
<div class="highlight"><pre><span></span><code># reboot
</code></pre></div>
<p>After that, I'm not even sure your system will boot ;) You might need to enter
your UEFI manager, and then manually add a boot entry: select the file
<code>bootx64.efi</code>. On my laptop, the UEFI manager is a bit dumb, I also need to
give a name when I create the entry, otherwise it fails with some unhelpful
error message.</p>
<h2>Thanks</h2>
<p>Here are some readings that made this article possible.</p>
<ul>
<li><a href="https://ubuntuforums.org/showthread.php?t=2266650">https://ubuntuforums.org/showthread.php?t=2266650</a></li>
<li><a href="https://wiki.debian.org/GrubEFIReinstall">https://wiki.debian.org/GrubEFIReinstall</a></li>
</ul>
<p>Edited in 2021, May to improve the chroot setup procedure, thanks to Oliver
Kollenberg.</p>Using GMail to send system mails on Debian2017-07-04T00:00:00+00:002017-07-04T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2017-07-04:/2017/07/04/using-gmail-to-send-system-mails-on-debian/<p>If you're a Debian user like me, chances are that, at some point, you will
need to send emails from the command-line. It could be on a Debian server,
where you want to send notifications (login, logout, boot, daily report, ...).
Or it could be on your own machine, where you use some tools like <code>reportbug</code>
or <code>git send-email</code>.</p>
<p>If you're a Debian user like me, chances are that, at some point, you will
need to send emails from the command-line. It could be on a Debian server,
where you want to send notifications (login, logout, boot, daily report, ...).
Or it could be on your own machine, where you use some tools like <code>reportbug</code>
or <code>git send-email</code>.</p>
<p>On this page we will go through the configuration needed to make it happen.
I assume that we will send the mails through <a href="https://mail.google.com">GMail</a>.
Then we'll go through the config steps of <a href="http://www.exim.org/">Exim4</a>, which
is the MTA (Mail Transfer Agent) usually installed on Debian.</p>
<p>At last, we will look at the config steps for <a href="http://msmtp.sourceforge.net/">msmtp</a>,
a SMTP client, which can be a lightweight alternative to Exim, if for some
reasons you want an alternative.</p>
<p><strong>UPDATE</strong>: This tutorial is outdated, please refer to the new tutorial:
<a href="https://arnaudr.io/2020/08/24/send-emails-from-your-terminal-with-msmtp/">https://arnaudr.io/2020/08/24/send-emails-from-your-terminal-with-msmtp/</a></p>
<h2>GMail setup</h2>
<p>On the GMail-side, there's a little bit to do.</p>
<p>First, you want to ensure that the <em>2-Step Verification</em> is enabled. For that
to happen, visit <a href="https://myaccount.google.com/security">https://myaccount.google.com/security</a> and enable the 2-Step
Verification feature.</p>
<p>This will require that you authorize all your devices once (mail client, phone
and so on), which can be a bit tedious, but it's only for once. And after it's
done, you're left with a more secured account, so it's not that bad, right?</p>
<p>Enabling the 2-Step Verification will unlock the feature we want: <em>App Passwords</em>.
Visit <a href="https://myaccount.google.com/apppasswords">https://myaccount.google.com/apppasswords</a>, and go generate a password
for your application. You should use it only for <em>one</em> application, and never use
it anywhere else. The idea is that for each application that will send emails
through your GMail account, you create a password. Simple, right?</p>
<p>There are real benefits with app passwords:</p>
<ul>
<li>you won't have to write down your real password in a configuration file.</li>
<li>gmail won't block your emails for security reasons (happens if you
send emails from different servers in different locations, for example).</li>
<li>you can change your GMail password without impacting applications.</li>
<li>you can revoke an app password anytime without impacting anything else.</li>
</ul>
<p>OK, once you've done the GMail setup, keep going.</p>
<h2>Exim</h2>
<p>If you're on a Debian system, it's likely that <code>exim</code> is already installed.
You can easily check that with <code>dpkg -l | grep exim</code>. If it's not there, then
install!</p>
<div class="highlight"><pre><span></span><code>apt-get install exim4 mailutils
</code></pre></div>
<p>Then, all you need is a little conf.</p>
<div class="highlight"><pre><span></span><code>dpkg-reconfigure exim4-config
</code></pre></div>
<p>And here's the answers I used (most of it is the default):</p>
<ul>
<li>General type of mail configuration: <code>mail sent by smarthost; no local mail</code></li>
<li>System mail name: <code>localhost</code></li>
<li>IP-addresses to listen on for incoming SMTP connections: <code>127.0.0.1 ; ::1</code></li>
<li>Other destinations for which mail is accepted: <em>leave blank</em></li>
<li>Visible domain name for local users: <code>localhost</code> </li>
<li><strong>IP address or host name of the outgoing smarthost: <code>smtp.gmail.com::587</code></strong></li>
<li>Keep number of DNS-queries minimal (Dial-on-Demand)? <code>No</code></li>
<li>Split configuration into small files? <code>No</code></li>
</ul>
<p>Now, it's time to store your password in plain text. This is where you're happy
to use an app password instead of your account password.</p>
<p>In case you're a bit lost, I'm talking here about an app password that you
have to generate from your Google account page, mentioned above.</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'*.google.com:YourAddress@gmail.com:YourAppPassword'</span><span class="w"> </span><span class="err">\</span>
<span class="o">>></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">exim4</span><span class="o">/</span><span class="n">passwd</span><span class="p">.</span><span class="n">client</span>
</code></pre></div>
<p>Then ensure the file has the right ownership and permissions:</p>
<div class="highlight"><pre><span></span><code>chown root:Debian-exim /etc/exim4/passwd.client
chmod 640 /etc/exim4/passwd.client
</code></pre></div>
<p>At last, restart exim:</p>
<div class="highlight"><pre><span></span><code>update-exim4.conf
systemctl restart exim4
</code></pre></div>
<p>If you made it up to here, it's time for a reward! Send yourself a little email
to see if everything is OK.</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'Time for coffee buddy'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">mail</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="ss">"Coffee break"</span><span class="w"> </span><span class="n">AnyAddress</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span>
</code></pre></div>
<h4>Aliases</h4>
<p>Aliases are used to redirect mails for local recipients. For example, if you wish
to associate an email address with <code>root</code>, add this kind of line to the file
<code>/etc/aliases</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="o">:</span><span class="w"> </span><span class="n">RootAddress</span><span class="err">@</span><span class="n">Wherever</span><span class="o">.</span><span class="na">com</span>
</code></pre></div>
<p>Now, sending an email to root is easier.</p>
<div class="highlight"><pre><span></span><code>echo 'Root are you there?' | mail -s "Root message" root
</code></pre></div>
<h4>Tips and Tricks</h4>
<p>On the GMail App Passwords page, you can see the last time a password was used.
If you don't see this date being bumped while you send an email, you know that
your mail didn't even reach this step.</p>
<p>You can also look at the "Send Mails" section of your GMail mailbox, your mails
should leave a trace there.</p>
<p>If the mail is send but doesn't seem to arrive, always check your Spam.</p>
<h4>References</h4>
<ul>
<li><code>man exim4-config_files</code></li>
<li><a href="https://wiki.debian.org/GmailAndExim4">Debian Wiki - Gmail and Exim4</a></li>
<li><a href="https://www.talk-about-it.ca/setting-up-exim4-with-gmail-and-2-factor-authentication">Setting up Exim4 with GMail and 2-factor Authentication</a></li>
</ul>
<h2>MSMTP</h2>
<p>Msmtp is lightweigth, very simple to install and configure. If you just want
to be able to send emails, it's a great alternative to Exim. Here's the way
to go.</p>
<p>Install is a one-liner as usual.</p>
<div class="highlight"><pre><span></span><code>apt-get install msmtp
</code></pre></div>
<p>Create the configuration for the current user.</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtprc</span>
<span class="err">#</span><span class="w"> </span><span class="k">Set</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="k">values</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="ow">all</span><span class="w"> </span><span class="n">following</span><span class="w"> </span><span class="n">accounts</span><span class="p">.</span>
<span class="n">defaults</span>
<span class="n">auth</span><span class="w"> </span><span class="k">on</span>
<span class="n">tls</span><span class="w"> </span><span class="k">on</span>
<span class="n">tls_trust_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">certs</span><span class="o">/</span><span class="n">ca</span><span class="o">-</span><span class="n">certificates</span><span class="p">.</span><span class="n">crt</span>
<span class="n">logfile</span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">msmtp</span><span class="p">.</span><span class="nf">log</span>
<span class="err">#</span><span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="n">gmail</span><span class="w"> </span><span class="n">account</span>
<span class="n">account</span><span class="w"> </span><span class="n">gmail</span>
<span class="k">host</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span>
<span class="n">port</span><span class="w"> </span><span class="mi">587</span>
<span class="k">from</span><span class="w"> </span><span class="n">YourAddress</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="k">user</span><span class="w"> </span><span class="n">YourAddress</span>
<span class="n">password</span><span class="w"> </span><span class="n">YourAppPassword</span>
<span class="err">#</span><span class="w"> </span><span class="k">Set</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="n">account</span>
<span class="n">account</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">gmail</span>
<span class="n">EOF</span>
</code></pre></div>
<p>Explicitly set the right permissions on the configuration file.</p>
<div class="highlight"><pre><span></span><code>chmod 600 ~/.msmtprc
</code></pre></div>
<p>Now is the time to test!</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">msmtp</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="k">default</span>
<span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="n">Recipient</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span>
<span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="n">YourAddress</span><span class="nv">@gmail</span><span class="p">.</span><span class="n">com</span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Cafe</span><span class="w"> </span><span class="n">Sua</span><span class="w"> </span><span class="n">Da</span>
<span class="n">Iced</span><span class="o">-</span><span class="n">coffee</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">condensed</span><span class="w"> </span><span class="n">milk</span>
<span class="n">EOF</span>
</code></pre></div>
<h4>MTA</h4>
<p>Maybe you don't want to explicitly use msmtp to send emails. Maybe you have
some applications that expect a regular MTA program to be installed, and
will try to use the standard command <code>sendmail</code> to send emails.</p>
<p>Good news, it's just a matter of symlinking msmtp, and there is a package
that does just that for you.</p>
<div class="highlight"><pre><span></span><code>apt-get install msmstp-mta
</code></pre></div>
<p>Now you can test one last time.</p>
<div class="highlight"><pre><span></span><code><span class="n">cat</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sendmail</span><span class="w"> </span><span class="n">Recipient</span><span class="nv">@Wherever</span><span class="p">.</span><span class="n">com</span><span class="w"> </span>
<span class="nl">Subject</span><span class="p">:</span><span class="w"> </span><span class="n">Flat</span><span class="w"> </span><span class="n">White</span>
<span class="n">The</span><span class="w"> </span><span class="n">milky</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">making</span><span class="w"> </span><span class="n">coffee</span>
<span class="n">EOF</span>
</code></pre></div>
<h4>References</h4>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/Msmtp">https://wiki.archlinux.org/index.php/Msmtp</a></li>
</ul>VirtualBox In Five Minutes2017-01-26T00:00:00+00:002017-01-26T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2017-01-26:/2017/01/26/virtualbox-in-five-minutes/<p>Getting started with <a href="https://www.virtualbox.org/">VirtualBox</a> in a few words...</p>
<p>Getting started with <a href="https://www.virtualbox.org/">VirtualBox</a> in a few words...</p>
<h2>Install</h2>
<p>It all starts here.</p>
<div class="highlight"><pre><span></span><code>apt-get install virtualbox
</code></pre></div>
<p>Then, creating image just boils down to clicking on "Next" a couple of times.
Soon enough you will create image after image without even giving it a look.</p>
<h2>Sharing The Clipboard</h2>
<p>Soon you will want to copy/paste like crazy from host to guest, and for that you will need the guest additions.</p>
<p>On the guest, install, then reboot.</p>
<div class="highlight"><pre><span></span><code>apt-get install virtualbox-guest-dkms
reboot
</code></pre></div>
<p>Then just enable this wonderful feature in the machine settings: "Devices -> Shared Clipboard". </p>
<p>If for some reason it doesn't work, check this page for a complete procedure:<br>
<a href="http://askubuntu.com/q/22743/424636">http://askubuntu.com/q/22743/424636</a></p>
<h2>Sharing Directories</h2>
<h4>Automatically With VirtualBox "Auto-mount"</h4>
<p>Now, you might want to share some data: "Shared Folders" is up to the task.
"Devices -> Shared Folders -> Shared Folders Settings", click the button "Add",
then don't forget to check "Auto-mount" and "Make Permanent".</p>
<p>The folders show up under <code>/media/</code>. In order to access it, you must add yourself to a group:</p>
<div class="highlight"><pre><span></span><code>sudo adduser <user> vboxsf
</code></pre></div>
<p>Don't forget to log out and in for the change to take effect.</p>
<p>Now, that was very straightforward, but it's not completely satisfactory.
At first, if you're like me and you don't like to leave your home directory,
then you won't like to have to navigate to <code>/media/</code> each time.
Second (and more important), the files belong to root, and if you care a bit
about ownership, and if you're a clean person, then you won't like it.</p>
<p>To solve the first problem, you can use a bind mount, like that:</p>
<div class="highlight"><pre><span></span><code><span class="o">/</span><span class="nv">media</span><span class="o">/<</span><span class="nv">sf_shared_dir</span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="nv">home</span><span class="o">/<</span><span class="nv">user</span><span class="o">>/<</span><span class="k">dirname</span><span class="o">></span><span class="w"> </span><span class="nv">none</span><span class="w"> </span><span class="nv">bind</span>
</code></pre></div>
<p>But that won't solve the ownership issue.</p>
<h4>Automatically, But Smarter</h4>
<p>So, the cleanest way is actually to disable the "Auto-mount" setting for your shared folder,
and setup the automation yourself. It's fairly easy.</p>
<p>The most obvious solution it to add a line to <code>/etc/fstab</code>:</p>
<div class="highlight"><pre><span></span><code><span class="o"><</span><span class="nv">shared_dir</span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="nv">home</span><span class="o">/<</span><span class="nv">user</span><span class="o">>/<</span><span class="k">dirname</span><span class="o">></span><span class="w"> </span><span class="nv">vboxsf</span><span class="w"> </span><span class="nv">uid</span><span class="o">=</span><span class="mi">1000</span>,<span class="nv">gid</span><span class="o">=</span><span class="mi">1000</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>However in my case it failed the boot, probably because it was exectued too early or something.
So instead, I add to put the mount command in <code>/etc/rc.local</code>, and to delay it a bit.</p>
<p>The cleanest way for me to do that: keep the line in <code>fstab</code>, make it <code>noauto</code>,
then mount from <code>rc.local</code> with a delay.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">That</span><span class="err">'s /etc/fstab</span>
<span class="err"><shared_dir> /home/<user>/<dirname> vboxsf noauto,uid=1000,gid=1000 0 0</span>
#<span class="w"> </span><span class="nv">That</span><span class="err">'s /etc/rc.local</span>
<span class="err">(sleep 5s && mount <shared_dir>) &</span>
</code></pre></div>
<p>Notice that <code><shared_dir></code> is the folder name you gave to VirtualBox when you
created the shared folder. It acts as an uid here.</p>
<h2>In a nutshell</h2>
<p>So, to finish, here are the commands I run on every guest after a fresh install.
I know you don't give a damn, but I do, because this blog is my second memory.
That's what it's called a memento :)</p>
<div class="highlight"><pre><span></span><code>sudo dpkg-reconfigure tzdata
sudo apt-get update && sudo apt-get -y dist-upgrade
sudo apt-get -y install virtualbox-guest-dkms
sudo apt-get -y install qdbus git gitk vim
# And so on, add your favorite packages here...
# When it's over, save a bit of space?
sudo apt-get clean
</code></pre></div>Docker + Shippable for Painless Continuous Integration2017-01-13T00:00:00+00:002017-01-13T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2017-01-13:/2017/01/13/docker-shippable-for-painless-continuous-integration/<p>Here's my situation in a glimpse:</p>
<ul>
<li>I've been writing an application in C</li>
<li>I target the GNU/Linux OS</li>
<li>I want to ensure it builds on every major distros</li>
<li>I want an automated solution</li>
</ul>
<p>A very normal situation for a developer, isn't it?</p>
<p>Here's my situation in a glimpse:</p>
<ul>
<li>I've been writing an application in C</li>
<li>I target the GNU/Linux OS</li>
<li>I want to ensure it builds on every major distros</li>
<li>I want an automated solution</li>
</ul>
<p>A very normal situation for a developer, isn't it?</p>
<p>Automating the build is a part of what is called <a href="https://en.wikipedia.org/wiki/Continuous_integration">Continuous Integration</a>.
You probably heard the term already, it's very trendy lately. It's trendy,
and also fairly easy to setup if you use a major hosting service for your
project, such as <a href="https://github.com/">GitHub</a>, <a href="https://bitbucket.org/">Bitbucket</a> or <a href="https://gitlab.com/">GitLab</a>.</p>
<p>In this article, I'll describe the steps I followed to setup automatic
builds for my application <a href="https://github.com/elboulangero/goodvibes">Goodvibes</a>, an internet radio player for the
GNU/Linux OS.</p>
<p>Goodvibes is hosted on Github.</p>
<p>Builds are automatically triggered thanks to <a href="https://app.shippable.com/">Shippable</a>, one of the numerous
CI services available out there. Builds are performed on different environments:
Debian, Ubuntu, Fedora at the moment.</p>
<p>These build environments are provided under the shape of docker images.
I had to create these images myself, and I stored them on <a href="https://hub.docker.com/">Docker Hub</a>,
so that Shippable can easily retrieve it.</p>
<p>One last word: the workflow described here can be reproduced without the services
mentioned here. If you have your own GNU/Linux server, and are keen on git hooks
and shell scripting, you can probably reproduce such an automated process fairly
easily.</p>
<h2>Introducing the Players</h2>
<p><strong>Player A: GitHub</strong></p>
<p>GitHub is used to host the source code of my project. But it's also the key player
in the automation process, thanks to the magic of git & web hooks. Changes in git
repositories will trigger actions of the two other players: Shippable and Docker Hub.</p>
<p><strong>Player B: Shippable</strong></p>
<p>Shippable is in charge of automating the build. It supports C language, which means
that there will be very little scripting to do.</p>
<p>Shippable already provides a base environment to build C applications. But you're
not obliged to use it. The good thing is that Shippable uses docker images for
its build environment. It means that using another environment is extremely simple:
you just need to provide a different docker image.</p>
<p><strong>Player C: Docker Hub</strong></p>
<p>Docker Hub will host my build environments, aka docker images. That's all it does,
and that's all I need. A perfect match.</p>
<h2>The Big Picture</h2>
<p>Let me know describe briefly the process as it will happen once everyting is setup.</p>
<ul>
<li>my C project is hosted on GitHub, remember?</li>
<li>when I push a change, Shippable is notified (thanks to the hooks magic)</li>
<li>Shippable wants to build. For that:</li>
<li>it pulls the docker images from Docker Hub</li>
<li>then it builds in these contained environments</li>
</ul>
<p>Is it more or less clear? If you don't understand a word, it's time to quit reading.
Otherwise, hold on!</p>
<p>To achieve that, we will go through the following steps:</p>
<ul>
<li>prepare docker images that allow building our app</li>
<li>store it on Docker Hub</li>
<li>link our Github project to Shippable to automate the build</li>
</ul>
<h2>Getting Started with Docker</h2>
<p>I won't go into much details here, since the official documentation is just PER-FECT.
I advise you to read it, and follow the tutorials.</p>
<p>First thing first, installation. It's very easy. If you're using Debian, the
documentation is available here: <a href="https://docs.docker.com/engine/installation/linux/debian/">https://docs.docker.com/engine/installation/linux/debian/</a></p>
<p>In a nutshell, it looks like that:</p>
<div class="highlight"><pre><span></span><code><span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">apt</span><span class="o">-</span><span class="n">transport</span><span class="o">-</span><span class="n">https</span><span class="w"> </span><span class="n">ca</span><span class="o">-</span><span class="n">certificates</span><span class="w"> </span><span class="n">gnupg2</span>
<span class="n">apt</span><span class="o">-</span><span class="n">key</span><span class="w"> </span><span class="n">adv</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">keyserver</span><span class="w"> </span><span class="n">hkp</span><span class="p">:</span><span class="o">//</span><span class="n">ha</span><span class="o">.</span><span class="n">pool</span><span class="o">.</span><span class="n">sks</span><span class="o">-</span><span class="n">keyservers</span><span class="o">.</span><span class="n">net</span><span class="p">:</span><span class="mi">80</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">recv</span><span class="o">-</span><span class="n">keys</span><span class="w"> </span><span class="mf">58118E89</span><span class="n">F3A912897C070ADBF76221572C52609D</span>
<span class="n">echo</span><span class="w"> </span><span class="s1">'deb https://apt.dockerproject.org/repo debian-stretch main'</span><span class="w"> </span>\
<span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">apt</span><span class="o">/</span><span class="n">sources</span><span class="o">.</span><span class="n">list</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">list</span>
<span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">update</span>
<span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">engine</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">daemon</span><span class="o">-</span><span class="n">reload</span>
<span class="n">systemctl</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="n">docker</span>
</code></pre></div>
<p>A few words of caution:</p>
<ul>
<li>the <code>apt-key</code> command might be outdated, please check the official documentation</li>
<li>make sure to change <code>debian-stretch</code> to match your debian version</li>
</ul>
<p>To avoid running Docker with <code>sudo</code>, you might want to create a group:</p>
<div class="highlight"><pre><span></span><code>groupadd docker
gpasswd -a user docker
systemctl restart docker
</code></pre></div>
<p>Please note that:</p>
<ul>
<li>you should replace <code>user</code> with your username</li>
<li>log out and log in is required after adding a group</li>
</ul>
<p>Once Docker is up and running, it's time to start to play with it. Please follow
the documentation here:</p>
<ul>
<li><a href="https://docs.docker.com/engine/">https://docs.docker.com/engine/</a></li>
<li><a href="https://docs.docker.com/engine/userguide/#/learn-by-example">https://docs.docker.com/engine/userguide/#/learn-by-example</a></li>
</ul>
<p>After you finish playing with Docker, you might want to cleanup behind you.
There's a good summary of the commands you might need here:<br>
<a href="https://www.calazan.com/docker-cleanup-commands/">https://www.calazan.com/docker-cleanup-commands/</a></p>
<h2>Creating Docker Images for Your Build</h2>
<p>Basically, what I need is a minimal environment that allows to build my C application.</p>
<p>Docker Hub provides bare, minimal images of some major distros, like Debian,
Ubuntu, Fedora and such. These images are a base layer, I just need to top it up
with the packages I need:</p>
<ul>
<li>the C compilation toolchain (gcc/clang, make, autotools and so on)</li>
<li>the libraries required by my application (glib, gstreamer, gtk+, ...)</li>
</ul>
<p>Once my images are ready, tested, validated, I must create a <strong>Dockerfile</strong>:
this file contain the steps needed to build the image.</p>
<p>Dockerfiles are easy to create, easy to understand, but that shouldn't prevent
you frome reading the "best practices" section from the Docker documentation:
<a href="https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/">https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/</a></p>
<p>Dockerfiles are needed since we will later on create "automated build" on Docker Hub.
Please focus here, otherwise it might be confusing. I'm talking about automated
build of the docker images, in which I will later on run automated builds of my
application. You get it ;)?</p>
<p>Please don't get lost in all this automation and read on.</p>
<h2>Storing Dockerfiles on GitHub</h2>
<p>Now that you have your Dockerfiles ready, you need to store them in a git
repository, on GitHub or Bitbucket (both are supported by Docker Hub).</p>
<p>Please be logical in your naming. My application repository is <code>goodvibes</code>,
so I created a repository named <code>goodvibes-docker</code>. As simple as that.</p>
<p>As an example, I invite you to have a look there:<br>
<a href="https://github.com/elboulangero/goodvibes-docker">https://github.com/elboulangero/goodvibes-docker</a></p>
<p>As you can see, I just created a sub-directory for each docker image I want
to build, containing a single Dockerfile.</p>
<p>If you're a bit lost, don't worry, everything will become clear in the next
step, where Docker Hub kicks in.</p>
<h2>Creating a Docker Hub Account</h2>
<p>Now that your Dockerfiles are ready and hosted on the net, the next step is
to have a Docker Hub account, where you will store your docker images.</p>
<p>So do that. Create your account on Docker Hub, then link it with your GitHub
account. Once it's done, you now want to create <em>Automated Builds</em>. That is,
you want your images to be automatically generated according to your Dockerfiles.</p>
<p>It might be time for you to read some more documentation now:<br>
<a href="https://docs.docker.com/docker-hub/builds/">https://docs.docker.com/docker-hub/builds/</a></p>
<p>The procedure is quite straightforward, there's little to explain. By default,
the automatic build settings expect your Dockerfile to be at the root of your
git repository. If it's not the case (and it's not in my case), just click on
the <em>Customize</em> link when you can, and set the location of your Dockerfile.</p>
<p>Then repeat the procedure for each docker image you intend to build, so that in
the end you have an automated build for each Dockerfile.</p>
<p>It seemed to me that the first build had to be triggered manually. Just check that
in the <em>Build Details</em> section. If nothing's happening, just go to <em>Build Settings</em>
and trigger a build manually.</p>
<p>Now, you want Docker Hub to be smart, and automatically rebuild your images when
it's needed. When is it needed by the way?</p>
<ul>
<li>when the base image (Debian, Ubuntu and so on) is updated</li>
<li>when you modify your Dockerfiles</li>
</ul>
<p>To handle the first situation, it all happens on Docker Hub. Just enter the
<em>Build Settings</em> section of your automated build, then the <em>Repository Links</em>,
and add a link to the base image. Done.</p>
<p>For the second situation, it happens on GitHub. You will have to manually link the
Docker service to your Github project. In order to do that, open GitHub, navigate
to the <em>Settings</em> part of the repository containing your Dockerfiles, then choose
<em>Integration & services</em>, and there you add Docker.</p>
<p>When it's done, Docker should automatically be notified when your Dockerfiles
are modified, and rebuild your images accordingly.</p>
<p>As an example, you can have a look at my docker images here:<br>
<a href="https://hub.docker.com/u/elboulangero/">https://hub.docker.com/u/elboulangero/</a></p>
<h2>Create a Shippable Account</h2>
<p>We're almost there. We have our docker images hosted and ready, automatically
built, easy to update, it's all wonderful. What's next? Setup the continuous
integration service, which is Shippable in this tutorial.</p>
<p>Creating an account there is straightforward, nothing special.</p>
<p>Then, link it with your GitHub account, and then enable the projects you want to
monitor. You might need a few clicks on the GitHub side as well, or maybe not, I
don't remember. Anyway, this is simple, you won't get lost.</p>
<p>The very last step of this tutorial is coming: you now need to add a <code>.shippable.yml</code>
file to your project, where you describe the steps for the build. I strongly advise
you, one more time, to check the doc:<br>
<a href="http://docs.shippable.com/ci/overview/">http://docs.shippable.com/ci/overview/</a></p>
<p>Shippable provides a default build image, Ubuntu something. If you don't want
to use it, no problem. You can instead indicate the custom images you want to run.
If these images are hosted on Docker Hub, it's very straightforward.</p>
<p>As an example, you can have a look at my <code>.shippable.yml</code> file:<br>
<a href="https://github.com/elboulangero/goodvibes/blob/master/.shippable.yml">https://github.com/elboulangero/goodvibes/blob/master/.shippable.yml</a></p>
<h2>Finished</h2>
<p>So let me draw you the whole picture again. I've you've been reading seriously
up to now, you already have the picture in your mind:</p>
<div class="highlight"><pre><span></span><code><span class="c"> </span><span class="nb">----------------------</span>
<span class="c"> | dockerfiles @ github |</span>
<span class="c"> </span><span class="nb">----------------------</span>
<span class="c"> \</span>
<span class="c"> source for</span>
<span class="c"> \</span>
<span class="c"> </span><span class="nb">---------------------</span>
<span class="c"> | images @ docker hub |</span>
<span class="c"> </span><span class="nb">---------------------</span>
<span class="c"> /</span>
<span class="c"> pull from</span>
<span class="c"> /</span>
<span class="c"> </span><span class="nb">-------------------</span>
<span class="c"> | build @ shippable |</span>
<span class="c"> </span><span class="nb">-------------------</span>
<span class="c"> /</span>
<span class="c"> trigger</span>
<span class="c"> /</span>
<span class="c"> </span><span class="nb">------------------</span>
<span class="c">| project @ github |</span>
<span class="c"> </span><span class="nb">------------------</span>
</code></pre></div>
<p>Easy to understand once all the pieces of the puzzle fit.</p>
<p>I hope you enjoyed the reading, now it's time to dock and ship!</p>Learning GObject2016-11-28T00:00:00+00:002016-11-28T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-11-28:/2016/11/28/learning-gobject/<p>I worked on a C/GObject project for a while now, and had to learn GObject from
scratch. Learning GObject is definitely not the easiest thing to do, it's much
more than just learning to use a new library. Hence this "bookmark article",
that list the best links I found so far. Hope this helps other learners like me.</p>
<p>I worked on a C/GObject project for a while now, and had to learn GObject from
scratch. Learning GObject is definitely not the easiest thing to do, it's much
more than just learning to use a new library. Hence this "bookmark article",
that list the best links I found so far. Hope this helps other learners like me.</p>
<h2>How Do I Learn GObject?</h2>
<p>First, you need to bookmark the official documentation. It's the best place to
start, but also a good place to go back afterward. Things that don't make sense
at the first reading will become clear as you go further.</p>
<ul>
<li><a href="https://developer.gnome.org/gobject/stable/">GObject Reference Manual</a></li>
</ul>
<p>Second, there's one essential article that nicely sums up the GObject
construction process, and raise some important points around it. This article
you will have to read again and again.</p>
<ul>
<li><a href="https://blogs.gnome.org/desrt/2012/02/26/a-gentle-introduction-to-gobject-construction/">A gentle introduction to gobject construction</a></li>
</ul>
<p>At last, you need code example, although the code you find is not always
up-to-date with latest/best way to do things. Anyway. The main GObject-based
libraries are probably the best places to go to learn GObject. Grab the source
code, sharpen your grep, and there you go.</p>
<ul>
<li><a href="https://developer.gnome.org/gtk3/stable/">GTK+</a></li>
<li><a href="https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/">GStreamer</a></li>
</ul>
<h2>Good Practices</h2>
<h3>Follow The Conventions</h3>
<p>The first place to learn about good practices is the GObject conventions page.
It's more than good practice actually. GObject expects you to follow a few
conventions, so that code generation tools (<code>glib-mkenum</code>) and other stuff can
actually work. Don't try to avoid that.</p>
<ul>
<li><a href="https://developer.gnome.org/gobject/stable/gtype-conventions.html">GObject Conventions</a></li>
</ul>
<h3>Use <code>G_DECLARE_*</code> And <code>G_DEFINE_*</code></h3>
<p>Do not clutter the header file with too much boilerplate, as it's often seen in
some old GObject-based code.</p>
<ul>
<li><a href="https://blogs.gnome.org/desrt/2015/01/27/g_declare_finalderivable_type/">G_DECLARE_FINAL_DERIVABLE_TYPE</a></li>
</ul>
<h3>Do The Minimum In <code>_init()</code></h3>
<p>There should be almost nothing in the <code>init()</code> function. Most of the time,
it just boils down to setting up the private pointer, if any.</p>
<p>For all the rest, it's better to implement it in the <code>constructed()</code> method.</p>
<ul>
<li><a href="https://blogs.gnome.org/desrt/2012/02/26/a-gentle-introduction-to-gobject-construction/">A gentle introduction to gobject construction</a></li>
</ul>
<h3>Always Use A Private Pointer</h3>
<p>Using a private structure is particularly useful for derivable objects, since
for these objects, the structure MUST BE defined in the header (public) to allow
inheritance. But when the object is not derivable, and therefore the structure
is defined in the code (private), what the point of having a private struct?
Answer: consistency, and less hassle when the objects change from final to
derivable, or the other way around.</p>
<ul>
<li><a href="https://blogs.gnome.org/swilmet/2015/10/10/changing-quickly-between-a-final-and-derivable-gobject-class/">Changing quickly between a final and derivable GObject class</a></li>
</ul>
<h3>Best Way To Set Properties</h3>
<p>The correct way to set an object pointer. It's not as trivial as it seems.</p>
<ul>
<li><a href="https://tecnocode.co.uk/2014/12/19/use-g_set_object-to-simplify-and-safetyify-gobject-property-setters/">Use g_set_object() to simplify (and safetyify) GObject property setters</a></li>
</ul>
<h2>Quick FAQ</h2>
<h3>How To Change A Construct-Only Property Via Inheritance</h3>
<p>This questions is asked on StackOverflow:</p>
<p><a href="http://stackoverflow.com/q/16557905/776208">http://stackoverflow.com/q/16557905/776208</a></p>
<p>Unfortunately, the answer given by Emmanuele Bassi is wrong, and it seems that
the only other solution is to hack around, as did the guy who asked the question.</p>
<p>There's another solution though: don't use construct-only properties if they are
to be modified by the inherited objects. Make them constuct.</p>
<h3>How To Implement A Singleton</h3>
<p>This is discussed here and there on the Net.</p>
<ul>
<li><a href="https://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#GObjectClass">GObject API Reference</a></li>
<li><a href="https://blogs.gnome.org/xclaesse/2010/02/11/how-to-make-a-gobject-singleton/">How to make a GObject singleton</a></li>
</ul>
<h3>How To Have Protected Fields</h3>
<p>GObject doesn't provide any solution to have protected fields. I managed to do
without up to now, but just in case, here's a workaround. Basically, it's just
about defining the private structure in the header (therefore making it public),
and naming it 'protected' instead of 'private'.</p>
<ul>
<li><a href="http://codica.pl/2008/12/21/gobject-and-protected-fields-simple-hack/">GObject and protected fields – simple hack</a></li>
</ul>Publishing A Debian Package (Mentors & Sponsorship)2016-10-01T00:00:00+00:002016-10-01T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-10-01:/2016/10/01/publishing-a-debian-package-mentors-sponsorship/<p>Here are the steps to follow to publish a Debian package,
so that it becomes part of the official Debian repositories.</p>
<p>Here are the steps to follow to publish a Debian package,
so that it becomes part of the official Debian repositories.</p>
<h2>Introduction</h2>
<p>Here comes the second part of our <a href="https://www.debian.org/">Debian</a> packaging story!</p>
<p>This is a follow-up of the post <a href="{% post_url 2016-01-14-debian-package-starting %}">First Debian package</a>.</p>
<p>Alright, let's assume that you have a Debian package in your hands. It's nice and all,
and now you wish to make it available to the world. That is, you wish
that your package make it to the official Debian repositories. Here are the steps to do that.</p>
<p>Of course, I assume that you're not a DD (Debian Developer), otherwise you
wouldn't be reading that. No no no, you're just a random guy like me.
How can you get your package to be accepted by the Debian team?</p>
<p>Well, basically, you will have to find a Debian Developer who is willing to
help you. He will review your package, tell you what needs to be fixed,
and in the end accept your package. This guy is called a <strong>mentor</strong>,
and what you'll be doing is a <strong>request for sponsorship</strong>.</p>
<p>For this example, we use the <a href="https://github.com/nicklan/pnmixer">PNMixer</a> package,
version 0.6.1.</p>
<h2>Installing</h2>
<p>There's one MUST HAVE tool in this story: <code>reportbug</code>.</p>
<div class="highlight"><pre><span></span><code>apt-get install reportbug
</code></pre></div>
<p>There's a bit of configuration the first time you run it, don't be scared :)</p>
<p>You will also need <code>debsign</code> to sign the package:</p>
<div class="highlight"><pre><span></span><code>apt-get install devscripts
</code></pre></div>
<p>And at last, the package upload tool:</p>
<div class="highlight"><pre><span></span><code>apt-get install dput
</code></pre></div>
<h2>References</h2>
<p>There is plenty of documentation available on the Net already.
Our main source of information here is Debian Mentors pages.
Please ensure you read that all.<br>
- <a href="http://mentors.debian.net">http://mentors.debian.net</a>
- <a href="http://mentors.debian.net/intro-maintainers">http://mentors.debian.net/intro-maintainers</a></p>
<p>There is also the Debian Mentors FAQ, in the Debian Wiki.
The best "in a nutshell" guide is provided in the last link.
The 6 steps described here sum it all up.<br>
- <a href="https://wiki.debian.org/DebianMentorsFaq">https://wiki.debian.org/DebianMentorsFaq</a>
- <a href="https://wiki.debian.org/DebianMentorsFaq#How_do_I_make_my_first_package.3F">https://wiki.debian.org/DebianMentorsFaq#How_do_I_make_my_first_package.3F</a></p>
<h2>Get to know the tools</h2>
<p>First of all, Debian maintains a list of "work in progress" packages,
called the <strong>WNPP</strong> (Work-Needing and Prospective Packages).
Your package will make it to this list at first.</p>
<p>To get started, please visit: <a href="https://www.debian.org/devel/wnpp/">https://www.debian.org/devel/wnpp/</a>.</p>
<p>Second, you will have to deal with the <strong>Debian BTS</strong> (Bug Tracking System).
You will basically send mails to this robot in order to add your package
to the WNPP list, and change the status of your package if needed.</p>
<p>The Debian BTS starting point: <a href="https://bugs.debian.org/">https://bugs.debian.org/</a>. From there,
there are two pages that will teach you how to communicate with the server:
- <a href="https://www.debian.org/Bugs/server-request">https://www.debian.org/Bugs/server-request</a>
- <a href="https://www.debian.org/Bugs/server-control">https://www.debian.org/Bugs/server-control</a></p>
<p>There's one thing to understand here: <code>wnpp</code> is a pseudo-package.
And like every package, you file bug reports against it. And this is exactly
how it works: when you want to add your pacakge to this list, you file a
bug report. When you want to remove it, or change the status, you file a
bug report. But these bug reports have special format and keywords, so
that they can be parsed and handled by the Debian BTS.</p>
<p>To ensure that you don't mess up with the content of these bug reports,
you use <code>reportbug</code> to create your bug reports. This tool takes care
of the formatting, and in the end you just have to fill it a formatted
mail. Pretty cool, I tell you.</p>
<h2>Deal with the WNPP</h2>
<p>OK, so before everything else, the first thing to do is to inform the Debian
community that you're working on a package. You do that by filing a bug against
the <code>wnpp</code> pseudo-package on the Debian BTS.</p>
<p>Have a look at the WNPP list by visiting this page: <a href="https://bugs.debian.org/wnpp">https://bugs.debian.org/wnpp</a></p>
<p>Be patient, the page is a bit long to load. There are a lot of packages there.</p>
<p>When the page is ready, do a quick search to see if the software you're packaging is
mentioned in this list.</p>
<h3>Retitle an RFP to ITP</h3>
<p>When I packaged PNMixer 0.6, PNMixer was already mentioned in the list as an
<strong>RFP</strong> (Request For Package). In such case, all you have is to do is to ask
the BTS to retitle the package into an <strong>ITP</strong> (Intend To Package).</p>
<p>I did that simply by sending the following email to <code>control@bugs.debian.org</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nv">retitle</span><span class="w"> </span><span class="mi">745669</span><span class="w"> </span><span class="nv">ITP</span>:<span class="w"> </span><span class="nv">pnmixer</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="nv">Volume</span><span class="w"> </span><span class="nv">mixer</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">system</span><span class="w"> </span><span class="nv">tray</span>
</code></pre></div>
<h3>File an ITP</h3>
<p>Otherwise, if your package is not mentioned in the WNPP list,
and doesn't already exist in Debian, then you have to
file an ITP bug report against the <code>wnpp</code> pseudo-package.</p>
<p>The right tool to do that is <code>reportbug</code>. </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>reportbug<span class="w"> </span>wnpp
...
Choose<span class="w"> </span>the<span class="w"> </span>request<span class="w"> </span>type:<span class="w"> </span><span class="m">1</span>
Please<span class="w"> </span>enter<span class="w"> </span>the<span class="w"> </span>proposed<span class="w"> </span>package<span class="w"> </span>name:<span class="w"> </span>pnmixer
Checking<span class="w"> </span>status<span class="w"> </span>database...
A<span class="w"> </span>package<span class="w"> </span>called<span class="w"> </span>pnmixer<span class="w"> </span>already<span class="w"> </span>appears<span class="w"> </span>to<span class="w"> </span>exist<span class="w"> </span><span class="o">(</span>at<span class="w"> </span>least<span class="w"> </span>on<span class="w"> </span>your<span class="w"> </span>system<span class="o">)</span><span class="p">;</span><span class="w"> </span><span class="k">continue</span>?<span class="w"> </span><span class="o">[</span>y<span class="p">|</span>N<span class="p">|</span>q<span class="p">|</span>?<span class="o">]</span>?<span class="w"> </span>y
Please<span class="w"> </span>briefly<span class="w"> </span>describe<span class="w"> </span>this<span class="w"> </span>package...<span class="w"> </span>:<span class="w"> </span>Simple<span class="w"> </span>mixer<span class="w"> </span>application<span class="w"> </span><span class="k">for</span><span class="w"> </span>system<span class="w"> </span>tray
Your<span class="w"> </span>report<span class="w"> </span>will<span class="w"> </span>be<span class="w"> </span>carbon-copied<span class="w"> </span>to<span class="w"> </span>debian-devel,<span class="w"> </span>per<span class="w"> </span>Debian<span class="w"> </span>policy.
Querying<span class="w"> </span>Debian<span class="w"> </span>BTS<span class="w"> </span><span class="k">for</span><span class="w"> </span>reports<span class="w"> </span>on<span class="w"> </span>wnpp<span class="w"> </span><span class="o">(</span><span class="nb">source</span><span class="o">)</span>...
</code></pre></div>
<p>From then, just press <em>Enter</em> to discard the thousands of bug reports displayed.</p>
<p>Then, you're taken to your usual console text editor, and it's time to fill the blanks
(version, upstream author and so on). In the end, my email would be something like that:</p>
<div class="highlight"><pre><span></span><code><span class="n">Subject</span><span class="o">:</span><span class="w"> </span><span class="n">ITP</span><span class="o">:</span><span class="w"> </span><span class="n">pnmixer</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="n">Simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span>
<span class="n">Package</span><span class="o">:</span><span class="w"> </span><span class="n">wnpp</span>
<span class="n">Owner</span><span class="o">:</span><span class="w"> </span><span class="s2">"Your Name"</span><span class="w"> </span><span class="o"><</span><span class="n">your</span><span class="err">@</span><span class="n">name</span><span class="o">.</span><span class="na">com</span><span class="o">></span>
<span class="n">Severity</span><span class="o">:</span><span class="w"> </span><span class="n">wishlist</span>
<span class="o">*</span><span class="w"> </span><span class="n">Package</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">pnmixer</span>
<span class="w"> </span><span class="n">Version</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mf">0.7</span>
<span class="w"> </span><span class="n">Upstream</span><span class="w"> </span><span class="n">Author</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">Nick</span><span class="w"> </span><span class="n">Lanham</span><span class="w"> </span><span class="o"><</span><span class="n">https</span><span class="o">://</span><span class="n">github</span><span class="o">.</span><span class="na">com</span><span class="o">/</span><span class="n">nicklan</span><span class="o">></span>
<span class="o">*</span><span class="w"> </span><span class="n">URL</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">https</span><span class="o">://</span><span class="n">github</span><span class="o">.</span><span class="na">com</span><span class="sr">/nicklan/</span><span class="n">pnmixer</span>
<span class="o">*</span><span class="w"> </span><span class="n">License</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">GPL</span>
<span class="w"> </span><span class="n">Programming</span><span class="w"> </span><span class="n">Lang</span><span class="o">:</span><span class="w"> </span><span class="n">C</span>
<span class="w"> </span><span class="n">Description</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">Simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span>
<span class="n">PNMixer</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="n">designed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">system</span>
<span class="n">tray</span><span class="o">.</span><span class="w"> </span><span class="n">It</span><span class="w"> </span><span class="n">integrates</span><span class="w"> </span><span class="n">nicely</span><span class="w"> </span><span class="n">into</span><span class="w"> </span><span class="n">desktop</span><span class="w"> </span><span class="n">environments</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">don</span><span class="s1">'t have</span>
<span class="s1">a panel that supports applets and therefore can'</span><span class="n">t</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">applet</span><span class="o">.</span>
<span class="n">In</span><span class="w"> </span><span class="n">particular</span><span class="w"> </span><span class="n">it</span><span class="err">'</span><span class="n">s</span><span class="w"> </span><span class="n">been</span><span class="w"> </span><span class="n">used</span><span class="w"> </span><span class="n">quite</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">lot</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">fbpanel</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="n">tint2</span><span class="o">,</span><span class="w"> </span><span class="n">but</span>
<span class="n">should</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">fine</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">any</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span><span class="o">.</span>
<span class="n">PNMixer</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">designed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">work</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">systems</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="n">ALSA</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">sound</span>
<span class="n">management</span><span class="o">.</span><span class="w"> </span><span class="n">Any</span><span class="w"> </span><span class="n">other</span><span class="w"> </span><span class="n">sound</span><span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="n">like</span><span class="w"> </span><span class="n">OSS</span><span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">FFADO</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">currently</span><span class="w"> </span><span class="n">not</span>
<span class="n">sup</span><span class="w"> </span><span class="n">supported</span><span class="w"> </span><span class="o">(</span><span class="n">patches</span><span class="w"> </span><span class="n">welcome</span><span class="o">).</span><span class="w"> </span><span class="n">There</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">no</span><span class="w"> </span><span class="o">*</span><span class="n">official</span><span class="o">*</span><span class="w"> </span><span class="n">PulseAudio</span><span class="w"> </span>
<span class="n">support</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">moment</span><span class="o">,</span><span class="w"> </span><span class="n">but</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="n">seems</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">PNMixer</span><span class="w"> </span><span class="n">behaves</span><span class="w"> </span><span class="n">quite</span><span class="w"> </span><span class="n">well</span>
<span class="n">anyway</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">PA</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">running</span><span class="o">.</span><span class="w"> </span><span class="n">Feel</span><span class="w"> </span><span class="n">free</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">give</span><span class="w"> </span><span class="n">some</span><span class="w"> </span><span class="n">feedback</span><span class="o">.</span>
</code></pre></div>
<p>Save and quit, then one last confirmation is required...</p>
<div class="highlight"><pre><span></span><code><span class="n">Report</span><span class="w"> </span><span class="n">will</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">sent</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="ss">"Debian Bug Tracking System"</span><span class="w"> </span><span class="o"><</span><span class="n">submit</span><span class="nv">@bugs</span><span class="p">.</span><span class="n">debian</span><span class="p">.</span><span class="n">org</span><span class="o">></span>
<span class="n">Submit</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">report</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">wnpp</span><span class="w"> </span><span class="p">(</span><span class="n">e</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">edit</span><span class="p">)</span><span class="w"> </span><span class="o">[</span><span class="n">Y|n|a|c|e|i|l|m|p|q|d|t|s|?</span><span class="o">]</span><span class="vm">?</span><span class="w"> </span><span class="n">y</span>
<span class="n">Saving</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">backup</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">report</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">reportbug</span><span class="o">-</span><span class="n">wnpp</span><span class="o">-</span><span class="k">backup</span><span class="o">-</span><span class="mi">20161001</span><span class="o">-</span><span class="mi">5679</span><span class="o">-</span><span class="n">xi8Z2N</span>
<span class="n">Connecting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span><span class="w"> </span><span class="n">via</span><span class="w"> </span><span class="n">SMTP</span><span class="p">...</span>
</code></pre></div>
<p>After a few minutes, you should receive a confirmation from the server.
This is precious, because it containt the ITP bug number, and we need that in the next step.</p>
<h2>Ensure your packages closes the ITP</h2>
<p>In the case you submitted an ITP bug report, then you must ensure that your package
closes this bug report. This is something you document in the Changelog.</p>
<p>So, back in your package, edit the file <code>debian/changelog</code>. If it's the first
version of the package you should have a line like that:</p>
<div class="highlight"><pre><span></span><code> * Initial release. Closes: #nnnn.
</code></pre></div>
<p>Replace <code>#nnnn</code> with the bug number of your ITP.</p>
<p>Otherwise, add such a line (replace <code>#nnnn</code> of course):</p>
<div class="highlight"><pre><span></span><code> * Closes: #nnnn (ITP).
</code></pre></div>
<p>Then rebuild your package.</p>
<div class="highlight"><pre><span></span><code>dpkg-buildpackage -us -uc
</code></pre></div>
<h2>Create a GPG key</h2>
<p>If you don't have a GPG key, it's time to create one.</p>
<div class="highlight"><pre><span></span><code>gpg --gen-key
</code></pre></div>
<p>GPG puts its stuff in your home directory:</p>
<div class="highlight"><pre><span></span><code>ls ~/.gnupg/
</code></pre></div>
<p>You can list keys:</p>
<div class="highlight"><pre><span></span><code>gpg --list-keys
</code></pre></div>
<p>And that's enough for the moment...</p>
<h2>Sign the package</h2>
<p>Now, that you have a GPG key, signing the package is simple.</p>
<div class="highlight"><pre><span></span><code>debsign pnmixer_0.6.1-1_amd64.changes
</code></pre></div>
<p>What's important here is that the identity you used to make the package
(ie your name and email) must be exactly the same as the one you used
to create the GPG key. Otherwise you will get an error like that:</p>
<div class="highlight"><pre><span></span><code><span class="n">gpg</span><span class="o">:</span><span class="w"> </span><span class="n">skipped</span><span class="w"> </span><span class="s2">"Your Name <your@mail.com>"</span><span class="o">:</span><span class="w"> </span><span class="n">secret</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">available</span>
</code></pre></div>
<p>If you're in this situation, you can recreate a new GPG key with the right name,
or you can also use the <code>-m</code> option of <code>debsign</code> to give another identity.</p>
<h2>Upload to Debian Mentors</h2>
<p>Create an account on <a href="https://mentors.debian.net">https://mentors.debian.net</a> (account type: maintainer).</p>
<p>Validate your email address. Login to your account. Add your GPG key.
To have a public version of your GPG key, use the command suggested by the website.
At the moment, it is:</p>
<div class="highlight"><pre><span></span><code><span class="n">gpg</span><span class="w"> </span><span class="o">--</span><span class="k">export</span><span class="w"> </span><span class="o">--</span><span class="k">export</span><span class="o">-</span><span class="n">options</span><span class="w"> </span><span class="k">export</span><span class="o">-</span><span class="n">minimal</span><span class="w"> </span><span class="o">--</span><span class="n">armor</span><span class="w"> </span>\
<span class="w"> </span><span class="s2">"Your Name <your@mail.com>"</span><span class="w"> </span>\
<span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/.</span><span class="n">gnupg</span><span class="o">/</span><span class="n">yourname</span><span class="o">.</span><span class="n">pub</span>
</code></pre></div>
<p>And now, you should upload your signed package to the website using <code>dput</code>.</p>
<p>If you just installed <code>dput</code> at the beginning of this tutorial, and never run it,
then there's a bit of configuration to be done. You must create a configuration file.
Check <a href="http://mentors.debian.net/intro-maintainers">http://mentors.debian.net/intro-maintainers</a> for an up-to-date version of that.</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'</span>
<span class="s1">[mentors]</span>
<span class="s1">fqdn = mentors.debian.net</span>
<span class="s1">incoming = /upload</span>
<span class="s1">method = http</span>
<span class="s1">allow_unsigned_uploads = 0</span>
<span class="s1">progress_indicator = 2</span>
<span class="s1"># Allow uploads for UNRELEASED packages</span>
<span class="s1">allowed_distributions = .*</span>
<span class="s1">'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">dput</span><span class="p">.</span><span class="n">cf</span>
</code></pre></div>
<p>Then, a simple command will do:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">dput</span><span class="w"> </span><span class="n">mentors</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">-</span><span class="mi">1</span><span class="n">_amd64</span><span class="o">.</span><span class="n">changes</span>
<span class="o">...</span>
<span class="n">Uploading</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">mentors</span><span class="w"> </span><span class="p">(</span><span class="n">via</span><span class="w"> </span><span class="n">http</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">mentors</span><span class="o">.</span><span class="n">debian</span><span class="o">.</span><span class="n">net</span><span class="p">):</span>
<span class="w"> </span><span class="n">Uploading</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">-</span><span class="mf">1.</span><span class="n">dsc</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="o">.</span>
<span class="w"> </span><span class="n">Uploading</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">.</span><span class="n">orig</span><span class="o">.</span><span class="n">tar</span><span class="o">.</span><span class="n">gz</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="o">.</span><span class="w"> </span>
<span class="w"> </span><span class="n">Uploading</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">-</span><span class="mf">1.</span><span class="n">debian</span><span class="o">.</span><span class="n">tar</span><span class="o">.</span><span class="n">xz</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="o">.</span><span class="w"> </span>
<span class="w"> </span><span class="n">Uploading</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">-</span><span class="mi">1</span><span class="n">_amd64</span><span class="o">.</span><span class="n">deb</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="o">.</span><span class="w"> </span>
<span class="w"> </span><span class="n">Uploading</span><span class="w"> </span><span class="n">pnmixer_0</span><span class="o">.</span><span class="mf">6.1</span><span class="o">-</span><span class="mi">1</span><span class="n">_amd64</span><span class="o">.</span><span class="n">changes</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="o">.</span>
<span class="n">Successfully</span><span class="w"> </span><span class="n">uploaded</span><span class="w"> </span><span class="n">packages</span><span class="o">.</span>
</code></pre></div>
<p>You should receive a confirmation email after a few minutes.</p>
<h2>Check your package on Debian Mentors</h2>
<p>Just connect to your personal page on Debian Mentors:
<a href="https://mentors.debian.net/packages/my">https://mentors.debian.net/packages/my</a>.</p>
<p>Your package will show up after you receive a confirmation email
for your upload. It takes a bit of time, so just be patient,
relax and have a drink in the meantime.</p>
<p>This page will display everything that is right or wrong with your package.
If you've been serious and checked it properly with <code>lintian</code> beforehand,
you should have no bad surprise. Otherwise, you might discover that your
package is not as good as you thought...</p>
<h2>Ask for a sponsorship</h2>
<p>This is the last step :)</p>
<p>It happens on the Debian Mentor page. Have a look at the details for your package.
For me, it happens here: <a href="https://mentors.debian.net/package/pnmixer">https://mentors.debian.net/package/pnmixer</a> (this page
is probably unavailable at the moment you read these lines).</p>
<p>Among the several details, the one that matters to us is <strong>Needs a sponsor</strong>.
Enable this, then have a look at the home page of Debian Mentors:
<a href="https://mentors.debian.net/">https://mentors.debian.net/</a>. Your package should show up here. Great, isn't it?</p>
<p>And now, the very very last step. You should explictly utter a call for sponsorship.
The procedure is described on your package page in Debian Mentors. Just click
the <strong>View RFS template</strong> link (RFS stands for Request For Sponsorship).</p>
<p>Everything is explained here. In a nutshell, it's just filing a new bug report
against the <code>sponsorship-requests</code> pseudo-package.</p>
<p>You can see all the pending sponsorship-requests by visiting the page:
<a href="https://bugs.debian.org/sponsorship-requests">https://bugs.debian.org/sponsorship-requests</a>. It will provide you tons
of examples of RFS mails.</p>
<p>To file the bug report, you can use the usual tool <code>reportbug</code>.
Although at the moment of this writing, there's no reportbug template
for the <code>sponsorship-requests</code> pseudo-package.</p>
<p>So it might be easier to just use your usual mail client, and copy/paste
the template given by Debian Mentors.</p>
<p>I personally did that with <code>reportbug</code>, so here comes the procedure.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>reportbug<span class="w"> </span>sponsorship-requests
...
A<span class="w"> </span>package<span class="w"> </span>named<span class="w"> </span><span class="s2">"sponsorship-requests"</span><span class="w"> </span>does<span class="w"> </span>not<span class="w"> </span>appear<span class="w"> </span>to<span class="w"> </span>be<span class="w"> </span>installed...<span class="w"> </span>n
Getting<span class="w"> </span>available<span class="w"> </span>info<span class="w"> </span><span class="k">for</span><span class="w"> </span>sponsorship-requests...
Please<span class="w"> </span>enter<span class="w"> </span>the<span class="w"> </span>version<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>package<span class="w"> </span>this<span class="w"> </span>report<span class="w"> </span>applies<span class="w"> </span>to<span class="w"> </span><span class="o">(</span>blank<span class="w"> </span>OK<span class="o">)</span>
><span class="w"> </span>
Will<span class="w"> </span>send<span class="w"> </span>report<span class="w"> </span>to<span class="w"> </span>Debian<span class="w"> </span><span class="o">(</span>per<span class="w"> </span>lsb_release<span class="o">)</span>.
Querying<span class="w"> </span>Debian<span class="w"> </span>BTS<span class="w"> </span><span class="k">for</span><span class="w"> </span>reports<span class="w"> </span>on<span class="w"> </span>sponsorship-requests<span class="w"> </span><span class="o">(</span><span class="nb">source</span><span class="o">)</span>...
...
</code></pre></div>
<p>Just discard the bug reports found by hitting <em>Enter</em>. Then it starts to be interesting.
From now on, you should use the template provided by the Debian Mentors, and mentioned
above.</p>
<div class="highlight"><pre><span></span><code><span class="n">Briefly</span><span class="w"> </span><span class="k">describe</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">problem</span><span class="w"> </span><span class="p">(</span><span class="nf">max</span><span class="p">.</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="n">characters</span><span class="w"> </span><span class="n">allowed</span><span class="p">).</span>
<span class="o">></span><span class="w"> </span><span class="nl">RFS</span><span class="p">:</span><span class="w"> </span><span class="n">pnmixer</span><span class="o">/</span><span class="mf">0.7</span><span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="o">[</span><span class="n">ITP</span><span class="o">]</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="n">Simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="n">tray</span>
<span class="p">...</span>
<span class="n">Please</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">severity</span><span class="w"> </span><span class="k">level</span><span class="err">:</span><span class="w"> </span><span class="o">[</span><span class="n">normal</span><span class="o">]</span><span class="w"> </span><span class="mi">6</span>
<span class="p">...</span>
<span class="n">Please</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="nl">tags</span><span class="p">:</span><span class="w"> </span><span class="p">(</span><span class="n">one</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nc">time</span><span class="p">)</span><span class="w"> </span><span class="o">[</span><span class="n">none</span><span class="o">]</span>
</code></pre></div>
<p>Then it's time to edit the mail with your favorite console text editor.
This is when you basically copy/paste the template provided by Debian Mentors,
and discard everything reportbug prepared for you :)</p>
<p>Save and quit, then one last confirmation is required...</p>
<div class="highlight"><pre><span></span><code><span class="n">Report</span><span class="w"> </span><span class="n">will</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">sent</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="ss">"Debian Bug Tracking System"</span><span class="w"> </span><span class="o"><</span><span class="n">submit</span><span class="nv">@bugs</span><span class="p">.</span><span class="n">debian</span><span class="p">.</span><span class="n">org</span><span class="o">></span>
<span class="n">Submit</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">report</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">sponsorship</span><span class="o">-</span><span class="n">requests</span><span class="w"> </span><span class="p">(</span><span class="n">e</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">edit</span><span class="p">)</span><span class="w"> </span><span class="o">[</span><span class="n">Y|n|a|c|e|i|l|m|p|q|d|t|s|?</span><span class="o">]</span><span class="vm">?</span><span class="w"> </span><span class="n">y</span>
<span class="n">Saving</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">backup</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">report</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">reportbug</span><span class="o">-</span><span class="n">sponsorship</span><span class="o">-</span><span class="n">requests</span><span class="o">-</span><span class="k">backup</span><span class="o">-</span><span class="mi">20161001</span><span class="o">-</span><span class="mi">10960</span><span class="o">-</span><span class="n">bJE9KK</span>
<span class="n">Connecting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">smtp</span><span class="p">.</span><span class="n">gmail</span><span class="p">.</span><span class="n">com</span><span class="w"> </span><span class="n">via</span><span class="w"> </span><span class="n">SMTP</span><span class="p">...</span>
</code></pre></div>
<p>Done!</p>
<p>Now you just wait and see!</p>
<h2>Follow-up</h2>
<p>When your package has been accepted by Debian, there are several places that you can visit
to have a look at your package. Here they are (for the PNMixer package of course).</p>
<ul>
<li><a href="https://tracker.debian.org/pkg/pnmixer">https://tracker.debian.org/pkg/pnmixer</a></li>
<li><a href="https://packages.debian.org/pnmixer">https://packages.debian.org/pnmixer</a></li>
<li><a href="https://bugs.debian.org/pnmixer">https://bugs.debian.org/pnmixer</a></li>
</ul>Setting Up GoAccess To Analyse Apache Logs2016-09-16T00:00:00+00:002016-09-16T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-09-16:/2016/09/16/setting-up-goaccess-to-analyse-apache-logs/<p>On a Debian machine serving websites with the <a href="https://httpd.apache.org">Apache</a> server,
we install and configure <a href="https://goaccess.io/">GoAccess</a> to monitor the websites activity.</p>
<p>On a Debian machine serving websites with the <a href="https://httpd.apache.org">Apache</a> server,
we install and configure <a href="https://goaccess.io/">GoAccess</a> to monitor the websites activity.</p>
<p><strong>UPDATE</strong>: This tutorial is outdated, please refer to the new tutorial:
<a href="https://arnaudr.io/2020/08/10/goaccess-14-a-detailed-tutorial/">https://arnaudr.io/2020/08/10/goaccess-14-a-detailed-tutorial/</a></p>
<h2>Introduction</h2>
<p>Time to monitor a little bit my VPS and see if someone's reading my blogs!</p>
<p>I started self-hosting my blogs almost a year ago, and today I still have no idea
if someone is actually reading it! Of course I'm a bit curious, and I wish I knew.
But I never found an easy way to do that.</p>
<p>I never found, until lately, when I met a fellow blogger who wrote some articles
on the matter. Among different solutions, I settled for the very easy one: <strong>GoAccess</strong>.</p>
<p>GoAccess works by analysing the HTTP server logs, building some statistics,
and spitting the result out in a nice HTML page. It's a zero-hassle solution.
Exactly what I was looking for!</p>
<p>The original article is available here, and it's more complete than mine. Go read it!<br>
<a href="http://freedif.org/goaccess-bandwidth-statistics-by-folderurl-with-virtualhost/">http://freedif.org/goaccess-bandwidth-statistics-by-folderurl-with-virtualhost/</a></p>
<h2>Install</h2>
<p>We will install GoAccess from the GoAccess Debian repository,
so that we enjoy the latest cool features.</p>
<p>The several ways to install GoAccess are documented on the official webpage:<br>
<a href="https://goaccess.io/download">https://goaccess.io/download</a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"deb http://deb.goaccess.io/ </span><span class="k">$(</span>lsb_release<span class="w"> </span>-cs<span class="k">)</span><span class="s2"> main"</span><span class="w"> </span>><span class="w"> </span>/etc/apt/sources.list.d/goaccess.list
$<span class="w"> </span>wget<span class="w"> </span>-O<span class="w"> </span>-<span class="w"> </span>https://deb.goaccess.io/gnugpg.key<span class="w"> </span><span class="p">|</span><span class="w"> </span>apt-key<span class="w"> </span>add<span class="w"> </span>-
$<span class="w"> </span>apt-get<span class="w"> </span>update
</code></pre></div>
<p>If you want to install goaccess <em>without</em> Tokyo Cabinet storage support, type:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>goaccess
</code></pre></div>
<p><em>With</em> Tokyo Cabinet storage support:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>goaccess-tcb
</code></pre></div>
<p>Both gives you this output:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>goaccess<span class="w"> </span>--version
GoAccess<span class="w"> </span>-<span class="w"> </span><span class="m">1</span>.0.2.
For<span class="w"> </span>more<span class="w"> </span>details<span class="w"> </span>visit:<span class="w"> </span>http://goaccess.io
Copyright<span class="w"> </span><span class="o">(</span>C<span class="o">)</span><span class="w"> </span><span class="m">2009</span>-2016<span class="w"> </span>by<span class="w"> </span>Gerardo<span class="w"> </span>Orellana
</code></pre></div>
<h2>Configure websites for monitoring</h2>
<p>GoAccess works by scanning the logs of your HTTP server.</p>
<p>All we have to do is to ensure the log configuration of the HTTP server is OK.
I assume you host a few websites with Apache, and each one of them is a virtual host.</p>
<p>In such situation, what we want is to have one logfile per virtual host.
This is easily achieved with the <code>CustomLog</code> command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vi<span class="w"> </span>/etc/apache2/sites-enabled/website1.conf
<span class="nt"><VirtualHost</span> <span class="err">*:80</span><span class="nt">></span>
<span class="w"> </span>...
<span class="w"> </span>ErrorLog<span class="w"> </span><span class="cp">${</span><span class="n">APACHE_LOG_DIR</span><span class="cp">}</span>/website1_vhosts_error.log
<span class="w"> </span>CustomLog<span class="w"> </span><span class="cp">${</span><span class="n">APACHE_LOG_DIR</span><span class="cp">}</span>/website1_vhosts_access.log<span class="w"> </span>combined
<span class="nt"></VirtualHost></span>
</code></pre></div>
<p>That's all there is to do!</p>
<h2>Configure GoAccess</h2>
<p>Let's have a look at the configuration file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vi<span class="w"> </span>/etc/goaccess.conf
</code></pre></div>
<p>To match the Apache configuration mentioned above, let's uncomment the following lines:</p>
<div class="highlight"><pre><span></span><code>time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
</code></pre></div>
<p>That's it! GoAccess is ready to parse Apache logs now.</p>
<h2>Run GoAccess</h2>
<p>Nothing easier!</p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">website1_vhosts_access</span><span class="o">.</span><span class="n">log</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">goaccess</span><span class="o">.</span><span class="n">html</span>
</code></pre></div>
<ul>
<li><code>-f</code> option is of course the log file you want to analyse.</li>
<li><code>-o</code> is the output file.</li>
</ul>
<p>With this command, GoAccess generate a static HTML page containing all the statistics,
nicely ordered and displayed. Just open this page in your web browser and enjoy.</p>
<h2>Run GoAccess with persistent logs</h2>
<p>You will find out quickly that just parsing a log file is limited, because log files
are usually rotated by your loggingn daemon. Quickly enough, a log file is moved
and replaced by a fresh, empty file. As a result, GoAccess will only be able to display
statistics on a short period, maybe one day.</p>
<p>To overcome this limitation, GoAccess can keep its own database if it's compiled with
Tokyo Cabinet storage support (I'm sure you wondered what the hell was that Tokyo thing...).
With that, GoAccess gain a memory of it own!</p>
<p>Then you just need to use it with a different set of options (see <strong>PROCESSING LOGS INCREMENTALLY</strong>
in the man page for additional information).</p>
<p>The right command is then:</p>
<div class="highlight"><pre><span></span><code><span class="n">goaccess</span><span class="w"> </span><span class="o">--</span><span class="nb">load</span><span class="o">-</span><span class="n">from</span><span class="o">-</span><span class="n">disk</span><span class="w"> </span><span class="o">--</span><span class="n">keep</span><span class="o">-</span><span class="n">db</span><span class="o">-</span><span class="n">files</span><span class="w"> </span>\
<span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">website1_vhosts_access</span><span class="o">.</span><span class="n">log</span><span class="w"> </span>\
<span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">goaccess</span><span class="o">.</span><span class="n">html</span>
</code></pre></div>
<h2>Choose where to store this output</h2>
<p>Ok, now there's two things we still need to do. Assuming you run GoAccess on a remote server,
you want the result to be accessible from outside. You have plenty of ways to do that.</p>
<p>For example, you could put the report at the root of your website named <code>website1</code>,
under the name <code>goaccess.html</code>, and then you could get your stats easily by connecting
to the address <code>http://www.website1.com/goaccess.html</code> or similar. You get the idea?</p>
<p>Another way is to have a dedicated directory where you put all your statistics.
That's the solution I use. My Apache server serves a default page located in
<code>/srv/www/default</code> when someone connects to my bare domain name. So I just added
a sub-directory named <code>goaccess</code>, and that's where I put the result.</p>
<p>You can then see the stats for this blog at the following address:<br>
<a href="https://goaccess.arnaudr.io/">https://goaccess.arnaudr.io/</a></p>
<h2>Automate the process to keep the stats updated</h2>
<p>Now the only thing left to do is to automate the process, and to have GoAccess generate a new report
on a regular basis. We can schedule that hourly with cron.</p>
<div class="highlight"><pre><span></span><code>crontab -e
</code></pre></div>
<p>Then add such a line (changes the pathes according to your setup, of course).</p>
<div class="highlight"><pre><span></span><code><span class="err">@</span><span class="n">hourly</span><span class="w"> </span><span class="n">goaccess</span><span class="w"> </span><span class="o">--</span><span class="nb">load</span><span class="o">-</span><span class="n">from</span><span class="o">-</span><span class="n">disk</span><span class="w"> </span><span class="o">--</span><span class="n">keep</span><span class="o">-</span><span class="n">db</span><span class="o">-</span><span class="n">files</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">apache2</span><span class="o">/</span><span class="n">website1_vhosts_access</span><span class="o">.</span><span class="n">log</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">default</span><span class="o">/</span><span class="n">goaccess</span><span class="o">/</span><span class="n">website1</span><span class="o">.</span><span class="n">html</span>
</code></pre></div>
<p>Done!</p>HDMI Autoswitch2016-09-05T00:00:00+00:002016-09-05T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-09-05:/2016/09/05/hdmi-autoswitch/<p>A nice script that does everything you want when you plug a monitor to your HDMI plug:
enable the monitor with xrandr, and redirect the sound to the HDMI output.</p>
<p>A nice script that does everything you want when you plug a monitor to your HDMI plug:
enable the monitor with xrandr, and redirect the sound to the HDMI output.</p>
<h2>Introduction</h2>
<p>Lately I started to plug my laptop on a TV using the HDMI plug.
I hoped that it would work out the box, since HDMI has been around for quite a while.
But I use a quite bare Debian, with Openbox as a window manager, and plain ALSA for the sound. That's it.
In such environment, NO-THING works out of the box.</p>
<p>And as often, I'm amazed at how long and complicated is the way to get things done.
Especially since, in this case, I just want to get the HDMI thing to work.
I'm not trying to do anything fancy, instead I just want my hardware to do its job.</p>
<p>OK, so if you're in the same situation, follow my steps.</p>
<h2>Screen test: playing around with lxrandr</h2>
<p>There's a simple GUI tool to get started: <a href="http://wiki.lxde.org/en/LXRandR">lxrandr</a>.</p>
<div class="highlight"><pre><span></span><code>apt-get install lxrandr
</code></pre></div>
<p>Plug your HDMI cable, launch lxrandr, click around, and everything is easy!</p>
<p>Lxrandr is the LXDE graphical frontend for the command <code>xrandr</code>.
It's pretty convenient to have it around, but it won't get anything automated.
If you don't want to launch lxrandr each time you connect your laptop to your TV, read on.</p>
<h2>Screen automation: udev solution</h2>
<p>There are various solutions to automatically enable/disable an external monitor when HDMI is plugged.
It's well covered on this StackExchange page: <a href="http://unix.stackexchange.com/q/4489">http://unix.stackexchange.com/q/4489</a></p>
<p>I opted for an <a href="https://en.wikipedia.org/wiki/Udev">udev</a> script. Makes your hands dirty a bit, but it's quite simple and reliable.</p>
<p>First thing first, do some testing with <code>udevadm</code>. Launch the monitor command, plug your HDMI cable, and ensure it's alive.</p>
<div class="highlight"><pre><span></span><code>udevadm monitor --environment --udev
</code></pre></div>
<p>Be sure to turn on the TV at the other side of the cable, otherwise nothing happens.
You might get this kind of output when you plug/unplug:</p>
<div class="highlight"><pre><span></span><code>UDEV [2247.166677] change /devices/pci0000:00/0000:00:02.0/drm/card0 (drm)
ACTION=change
DEVNAME=/dev/dri/card0
DEVPATH=/devices/pci0000:00/0000:00:02.0/drm/card0
DEVTYPE=drm_minor
HOTPLUG=1
ID_FOR_SEAT=drm-pci-0000_00_02_0
ID_PATH=pci-0000:00:02.0
ID_PATH_TAG=pci-0000_00_02_0
MAJOR=226
MINOR=0
SEQNUM=2120
SUBSYSTEM=drm
TAGS=:seat:master-of-seat:uaccess:
USEC_INITIALIZED=15405142
</code></pre></div>
<p>Based on this information, let's create an udev rule (edit as needed).</p>
<div class="highlight"><pre><span></span><code>echo '
KERNEL=="card0", SUBSYSTEM=="drm", ENV{DISPLAY}=":0", ENV{XAUTHORITY}="/home/user/.Xauthority", RUN+="/usr/local/bin/hotplug_monitor.sh"
' > /etc/udev/rules.d/95-monitor-hotplug.rules
</code></pre></div>
<p>Now, let's write the script <code>/usr/local/bin/hotplug_monitor.sh</code> that will do the job.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="k">function</span><span class="w"> </span>connect<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>xrandr<span class="w"> </span>--output<span class="w"> </span>HDMI-1<span class="w"> </span>--same-as<span class="w"> </span>LVDS-1<span class="w"> </span>--preferred<span class="w"> </span>--primary<span class="w"> </span>--output<span class="w"> </span>LVDS-1<span class="w"> </span>--preferred
<span class="o">}</span>
<span class="k">function</span><span class="w"> </span>disconnect<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>xrandr<span class="w"> </span>--output<span class="w"> </span>HDMI-1<span class="w"> </span>--off
<span class="o">}</span>
xrandr<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"HDMI-1 connected"</span><span class="w"> </span><span class="p">&</span>><span class="w"> </span>/dev/null<span class="w"> </span><span class="o">&&</span><span class="w"> </span>connect<span class="w"> </span><span class="o">||</span><span class="w"> </span>disconnect
</code></pre></div>
<p>I basically used the script provided in the StackExchange discussion here <a href="http://unix.stackexchange.com/a/171916">http://unix.stackexchange.com/a/171916</a>,
with minors modifications. You might need to adapt a bit according to your needs.</p>
<h2>Sound test: ALSA just needs to be tamed</h2>
<p>Up to now, we just mastered half of the HDMI: the image part. Now comes the other half: the sound.</p>
<p>If you're using plain ALSA as I do, you probably still have the sound coming out of your laptop,
and you probably wish to have it go through the HDMI instead.</p>
<p>Well, at first, let's check that everything is working. Let's perform a quick sound check. My favorite command for that is:</p>
<div class="highlight"><pre><span></span><code>speaker-test -c2 -twav
</code></pre></div>
<p>Here, we didn't specify any soundcard, so ALSA send the sound to the default sound card.
In my case, the sound is played by my laptop speakers.</p>
<p>Now, we can tell ALSA to use the HDMI device to play the sound.</p>
<div class="highlight"><pre><span></span><code>speaker-test -D hdmi -c2 -twav
</code></pre></div>
<p>In my case, I hear the sound from the TV now. It works!</p>
<p>So, if you're in the same situation, everything works fine. That is:
any sound is played through the default soundcard, unless you ask otherwise.
If you want an application to send the sound to the HDMI, you need to tell it explicitly.</p>
<p>For example, if you play a movie with VLC, you can go to <em>Audio -> Audio Device</em> and set it to HDMI.</p>
<p>That's not too bad, but you can already see that it will be a pain. Each time you plug the HDMI,
you will need to reconfigure VLC, probably also your web browser, your music player... Want a better solution?</p>
<p>There is one. Don't reconfigure any application, but reconfigure ALSA instead, so that the HDMI becomes the default sound card.</p>
<p>Doing that is ridiculously easy in my case. First, let me show you what my sound devices are.</p>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">aplay</span><span class="w"> </span><span class="o">-</span><span class="n">l</span>
<span class="o">****</span><span class="w"> </span><span class="n">List</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">PLAYBACK</span><span class="w"> </span><span class="n">Hardware</span><span class="w"> </span><span class="n">Devices</span><span class="w"> </span><span class="o">****</span>
<span class="n">card</span><span class="w"> </span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">PCH</span><span class="w"> </span><span class="p">[</span><span class="n">HDA</span><span class="w"> </span><span class="n">Intel</span><span class="w"> </span><span class="n">PCH</span><span class="p">],</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">VT1802</span><span class="w"> </span><span class="n">Analog</span><span class="w"> </span><span class="p">[</span><span class="n">VT1802</span><span class="w"> </span><span class="n">Analog</span><span class="p">]</span>
<span class="w"> </span><span class="nl">Subdevices</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span>
<span class="w"> </span><span class="n">Subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span>
<span class="n">card</span><span class="w"> </span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">PCH</span><span class="w"> </span><span class="p">[</span><span class="n">HDA</span><span class="w"> </span><span class="n">Intel</span><span class="w"> </span><span class="n">PCH</span><span class="p">],</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">2</span><span class="o">:</span><span class="w"> </span><span class="n">VT1802</span><span class="w"> </span><span class="n">Alt</span><span class="w"> </span><span class="n">Analog</span><span class="w"> </span><span class="p">[</span><span class="n">VT1802</span><span class="w"> </span><span class="n">Alt</span><span class="w"> </span><span class="n">Analog</span><span class="p">]</span>
<span class="w"> </span><span class="nl">Subdevices</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span>
<span class="w"> </span><span class="n">Subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span>
<span class="n">card</span><span class="w"> </span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">PCH</span><span class="w"> </span><span class="p">[</span><span class="n">HDA</span><span class="w"> </span><span class="n">Intel</span><span class="w"> </span><span class="n">PCH</span><span class="p">],</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="mi">3</span><span class="o">:</span><span class="w"> </span><span class="n">HDMI</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">[</span><span class="n">HDMI</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="nl">Subdevices</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span>
<span class="w"> </span><span class="n">Subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span><span class="o">:</span><span class="w"> </span><span class="n">subdevice</span><span class="w"> </span><span class="err">#</span><span class="mi">0</span>
</code></pre></div>
<p>I've got one soundcard (integrated to the laptop), divided in three devices.
All I want is that ALSA uses the device 3 instead of the device 0 for the default.
And for that, I just need to create an <a href="http://www.alsa-project.org/main/index.php/Asoundrc">asoundrc</a> file:</p>
<div class="highlight"><pre><span></span><code>echo '
# Use HDMI as the default playback device
defaults.pcm.device 3
' > ~/.asoundrc
</code></pre></div>
<p>It looks simple, but believe me it's not that simple, since the syntax of this asoundrc is somehow more
about magic than anything else. I found many ways to achieve that on the Net, most of them looked roughly the same,
but somehow broke the sound in the web browser, or just didn't work.</p>
<p>Anyway. With that simple line, the whole sound is send to HDMI. Notice, however, that if you had applications launched,
you need to restart them for the change to take effect.</p>
<h2>Sound automation</h2>
<p>To automate this change, just add a few lines in the script <code>hotplug_monitor.sh</code> we wrote above.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="k">function</span><span class="w"> </span>connect<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Screen</span>
<span class="w"> </span>xrandr<span class="w"> </span>--output<span class="w"> </span>HDMI-1<span class="w"> </span>--same-as<span class="w"> </span>LVDS-1<span class="w"> </span>--preferred<span class="w"> </span>--primary<span class="w"> </span>--output<span class="w"> </span>LVDS-1<span class="w"> </span>--preferred
<span class="w"> </span><span class="c1"># Sound</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>-f<span class="w"> </span>/etc/asound.conf<span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>mv<span class="w"> </span>-f<span class="w"> </span>/etc/asound.conf<span class="w"> </span>/etc/asound.before-hdmi.conf
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s1">'defaults.pcm.device 3'</span><span class="w"> </span>><span class="w"> </span>/etc/asound.conf
<span class="o">}</span>
<span class="k">function</span><span class="w"> </span>disconnect<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Screen</span>
<span class="w"> </span>xrandr<span class="w"> </span>--output<span class="w"> </span>HDMI-1<span class="w"> </span>--off
<span class="w"> </span><span class="c1"># Sound</span>
<span class="w"> </span>rm<span class="w"> </span>-f<span class="w"> </span>/etc/asound.conf
<span class="w"> </span><span class="o">[</span><span class="w"> </span>-f<span class="w"> </span>/etc/asound.before-hdmi.conf<span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>mv<span class="w"> </span>-f<span class="w"> </span>/etc/asound.before-hdmi.conf<span class="w"> </span>/etc/asound.conf
<span class="o">}</span>
xrandr<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"HDMI-1 connected"</span><span class="w"> </span><span class="p">&</span>><span class="w"> </span>/dev/null<span class="w"> </span><span class="o">&&</span><span class="w"> </span>connect<span class="w"> </span><span class="o">||</span><span class="w"> </span>disconnect
</code></pre></div>
<p>As you can notice, here I chose to write changes in the global file <code>/etc/asound.conf</code>,
but you can choose to edit the user file <code>~/.asoundrc</code>. It works all the same.</p>
<h1>Conclusion</h1>
<p>Now it's alsmot all automated. The only things that we have to do, apart from plugging and unpluggin the HDMI,
is to restart the applications that produce sound. It's not too bad!</p>
<p>One last word: with this setup, you will notice that you can't change the volume anymore from your laptop.
That's normal, that's because the HDMI device has no hardware way to change the volume. It's just a numeric channel,
basically. If you need to change the volume from your laptop, you can keep on fiddling with the asoundrc file
and create a softvol channel. More explanations here: <a href="https://bbs.archlinux.org/viewtopic.php?id=136790">https://bbs.archlinux.org/viewtopic.php?id=136790</a></p>
<p>In my situation, it's more a trouble than a solution, so I won't cover that. I just change volume with the TV remote,
and I'm happy with that :)</p>Rendering Markdown To PDF2016-07-21T00:00:00+00:002016-07-21T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-07-21:/2016/07/21/rendering-markdown-to-pdf/<p>When I have stuff to write, I like to keep it simple. It means plain text, editable with simple, dumb editors. As long as I can stay away from word processors, I do.</p>
<p>When I have stuff to write, I like to keep it simple. It means plain text, editable with simple, dumb editors. As long as I can stay away from word processors, I do.</p>
<p>With time, I found myself increasingly respecting the markdown syntax when I write. First, because having a consistent syntax doesn't harm. And second, because it allows to easily render the text to something a bit more appealing, with italic, bold, titles, and so on.</p>
<p>So, lately, I was writting some stuff for someone else. Could I send him the markdown file directly? I thought I could do better, and send him a formatted version. And I immediately thought: PDF.</p>
<p>So, how do we do that? Looking around, it appears that there are different solutions. The good starting point in your research is on Superuser at the moment:<br>
<a href="http://superuser.com/q/689056">How Can I Convert Github-Flavored Markdown To A PDF</a></p>
<p>It seems that most solutions (maybe all of them) are a two-steps process: first you render your Markdown to HTML, then you convert the HTML to PDF. There are a variety of tools for each task, which gives you some room to experiment. You could use the simple <code>markdown</code> tool for the first step, then a tool like <code>wkhtmltopdf</code> for the second.</p>
<p>Or you can use an all-in-one tool. After a few trials, I settled for this latest option, and opted for <a href="https://github.com/alanshaw/markdown-pdf">markdown-pdf</a>.</p>
<p>No Debian package, the best is to install it through the NodeJS package manager. Here we go. Make sure you have NodeJS already installed on your system.</p>
<div class="highlight"><pre><span></span><code>sudo npm install -g markdown-pdf
</code></pre></div>
<p>Converting boils down to a single command. It's a zero hassle solution, but also an almost zero customisation. I was happy with the result, mainly because the page break was smart enough not to happen in the middle of a paragraph. That's all, but still, <code>wkhtmltopdf</code> didn't had such a smart break policy.</p>
<p>To automate the conversion of my bunch of Markdown files, I wrote down a simple Makefile.</p>
<div class="highlight"><pre><span></span><code><span class="nv">MD</span><span class="o">=</span><span class="k">$(</span>wildcard<span class="w"> </span>*.md<span class="k">)</span>
<span class="nv">PDF</span><span class="o">=</span><span class="k">$(</span>patsubst<span class="w"> </span>%.md,%.pdf,<span class="k">$(</span>MD<span class="k">))</span>
<span class="nf">all</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PDF</span><span class="k">)</span>
<span class="nf">%.pdf</span><span class="o">:</span><span class="w"> </span>%.<span class="n">md</span>
<span class="w"> </span>markdown-pdf<span class="w"> </span>$^
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>rm<span class="w"> </span>-f<span class="w"> </span>*.pdf
</code></pre></div>
<p>Now it all boils down to <code>make</code> and <code>make clean</code>. Pretty cool, isn't it?</p>Unbloating The Sony Xperia M4 Aqua 8GB2016-06-10T00:00:00+00:002016-06-10T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-06-10:/2016/06/10/unbloating-the-sony-xperia-m4-aqua-8gb/<p>Sony Xperia M4 Aqua (E2353). A promising mobile phone, mid range,
not too expensive, dustproof and waterproof.
The only problem is the storage. Some comes with 16GB (lucky you),
but mine has only 8GB. And this is not enough!</p>
<p>Sony Xperia M4 Aqua (E2353). A promising mobile phone, mid range,
not too expensive, dustproof and waterproof.
The only problem is the storage. Some comes with 16GB (lucky you),
but mine has only 8GB. And this is not enough!</p>
<p>Out of the box, after updating everything, there's only 1.26GB free.
After using it a few weeks, I'm constantly hit by the message about
the memory being full. In fact, the phone proves to be a pain...</p>
<p>For a better and longer writing about this problem, have a look at:<br>
<a href="http://www.xda-developers.com/8gb-sony-xperia-m4-aqua-comes-with-only-1-26gb-free/">http://www.xda-developers.com/8gb-sony-xperia-m4-aqua-comes-with-only-1-26gb-free/</a></p>
<p>At first, I thought I could reflash it with a clean CyanogenMod image.
This would free my phone of all the Sony bloatware, and save some precious memory I'm sure.
But at the moment of this writing, the best I could find is an unofficial ROM,
and it's still under development. Some things don't work yet.
For those interested it's available here:<br>
<a href="http://projectmarshmallow.altervista.org/unofficial-cyanogenmod-12-1-for-sony-m4-aqua/">http://projectmarshmallow.altervista.org/unofficial-cyanogenmod-12-1-for-sony-m4-aqua/</a></p>
<p>So I thought the best was to stick with the official Sony firmware,
and to uninstall all the bloatware. But you can't do that out of the box.
The stock apps can't be uninstalled, only disabled. They still use memory.
The reason, as far as I understand, is quite simple. Stock apps are installed
on the system partition, and this partition can't be modified without having root access.
For a quick overview of Android partitions scheme, refer to:<br>
<a href="http://www.addictivetips.com/mobile/android-partitions-explained-boot-system-recovery-data-cache-misc/">http://www.addictivetips.com/mobile/android-partitions-explained-boot-system-recovery-data-cache-misc/</a></p>
<p>Since I'm a very newb with smartphone in general, and Android in particular,
it took me quite a while to understand the whole procedure needed to remove
this stock apps. I loosely followed this tutorial:<br>
<a href="http://forum.xda-developers.com/m4-aqua/general/tut-blu-twrp-root-xperia-m4-aqua-t3238962">http://forum.xda-developers.com/m4-aqua/general/tut-blu-twrp-root-xperia-m4-aqua-t3238962</a></p>
<p>But it doesn't give any explanations about <em>why</em> we do the things, so I dug
further to get to understand it properly. And now I'm able to sum that up in a few words!
And here they are...</p>
<h2>Edit October 2016</h2>
<p>Sony has now released a firmware based on Android 6 (aka Marshmallow):
- <a href="http://www.ibtimes.co.uk/xperia-m4-aqua-m5-getting-android-6-0-1-marshmallow-update-1571792">http://www.ibtimes.co.uk/xperia-m4-aqua-m5-getting-android-6-0-1-marshmallow-update-1571792</a>
- <a href="http://support.sonymobile.com/global-en/xperiam4aqua/kb/8019307466b15f3d01556b8631c4004c80/">http://support.sonymobile.com/global-en/xperiam4aqua/kb/8019307466b15f3d01556b8631c4004c80/</a></p>
<p>With Marshmallow comes a feature named <a href="http://www.greenbot.com/article/3039136/android/adoptable-storage-in-android-6-0-what-it-is-how-it-works.html">Adoptive Storage</a>,
which allows you to use part of your SD-Card as system storage. It's all transparent to the system,
so it's really an extension of the internal memory.</p>
<p>Although this feature doesn't come natively with the Sony ROM, it's possible to use it anyway.
There's more info about that here:<br>
<a href="http://forum.xda-developers.com/m4-aqua/general/adoptive-storage-marshmallow-t3421828">http://forum.xda-developers.com/m4-aqua/general/adoptive-storage-marshmallow-t3421828</a></p>
<p>It looks like this is so far the best solution for the M4 storage problem.
If I have time to go through these steps I'll write a post about that.</p>
<p>Thx to Bryden for giving me the info.</p>
<h2>Introduction</h2>
<p>In this tuto, we're going to rid our Xperia M4 to its pre-installed software.</p>
<p>In order to do that, we need several things:</p>
<ul>
<li>an uninstaller app: <a href="https://play.google.com/store/apps/details?id=com.jumobile.manager.systemapp">System app remover (ROOT)</a>.
This app needs to have a root access on the device, and for that we need another app...</li>
<li>a superuser access management app: <a href="https://play.google.com/store/apps/details?id=eu.chainfire.supersu">SuperSU</a>.
This app can't be installed through the Google Play Store, since it needs to modify the system partition of Android.
Therefore, it needs to be installed from the recovery software. And the stock recovery software doesn't allow that.
So we need to install a custom recovery software.</li>
<li>a custom recovery software: <a href="https://twrp.me/">TWRP</a>. However, in order to be able to modify the recovery partition,
we need...</li>
<li>unlocking the bootloader.</li>
</ul>
<p>Here you can see clearly the VERY long and complicated path that we're about to follow, simply to uninstall a few apps...</p>
<p>But cheer up, it's not that hard to do, and every Android hacker has to start from here I guess.
There are hundreds of tutorials on the net, but most of them are very quickly written,
and only describe the steps without explaining much. But I like to speak and explain stuff,
so hopefully here you'll get to understand what you're doing.</p>
<p>So we're gonna do all these things to your Android device. I have a Sony Xperia M4 Aqua,
but I guess the same steps work more or less for most of the Android devices.
You just have to ensure, each time, that you're device is compatible with the tools you're using.</p>
<p>As usual, I assume you're having a GNU/Linux computer in front of you, and it's Debian powered,
but any other distro can do, there's nothing Debian specific here, except for package manager.</p>
<h2>Warnings</h2>
<p>Following this tutorial will wipe your phone. That means a factory reset.
You will loose all your data, and your phone will be brand new.</p>
<p>Also, this procedure will rip you off your device's warranty.</p>
<p>You can't held me responsible for that.
I warned you, I assume you know what you're doing.</p>
<p>You can keep on reading now!</p>
<h2>Unlock the bootloader</h2>
<p>At first, you should know, right now, that unlocking the bootloader will wipe your phone.
Erase all the data. That's it. It may not be obvious at first, since there seem to be
no relation between the bootloader and the data on your phone. However, from a security
point of view, it makes sense. Here's a good discussion about that:<br>
<a href="http://android.stackexchange.com/q/33691">http://android.stackexchange.com/q/33691</a></p>
<p>Ok, now, let's get started with this first step.
As usual, we start by installing everything we need.</p>
<div class="highlight"><pre><span></span><code> apt-get install android-sdk
</code></pre></div>
<p>Amidst the Android SDK are some tools that we will need:
<code>fastboot</code> to send commands to the Android bootloader,
and <code>adb</code> to send commands to the booted Android device.</p>
<p>Ok, so most of this procedure happens in your web browser.
You need a code to unlock your device.
Visit this weppage from Sony and follow the steps, it's straight-forward.<br>
<a href="http://developer.sonymobile.com/unlockbootloader/unlock-yourboot-loader/">http://developer.sonymobile.com/unlockbootloader/unlock-yourboot-loader/</a></p>
<p>To sum up what happens:</p>
<ul>
<li>choose your Xperia model (M4 Aqua)</li>
<li>check your mails, click the confirmation link</li>
<li>get the IMEI of your phone by dialing <code>*#06#</code></li>
<li>feed that to the webpage</li>
<li>now you get an unlock code!</li>
</ul>
<p>The following steps are clearly detailed on Sony website, except for one little details, so read on!</p>
<p>On your device, at first, you can check whether the bootloader <em>can</em> be unlocked.
This is done by dialing <code>*#*#7378423#*#*</code>. This opens the <code>Service Menu</code> (it may take a few seconds).
In here, go to <code>Service Info -> Configuration</code> and check the value of <code>Rooting status</code>.
You should have that: <code>Bootloader unlock allowed: Yes</code></p>
<p>Ok, so now there's two things to do on the phone to prepare it for unlocking.</p>
<p>First, turn on USB debugging by going to <code>Settings > Developer options</code> and click to enable USB debugging.
As of Android Jelly Bean 4.2 the Developer options are hidden by default.
That's why you don't see it anywhere...
To enable them tap on <code>Settings > About Phone > Build Version</code> multiple times.
Then you will be able to access the <code>Settings > Developer options</code>.</p>
<p>Second, allow OEM unlock (it's the missing bit in the Sony tutorial) by going to
<code>Settings > Developer options</code> and clicking <code>OEM unlocking</code>.</p>
<p>Now, turn off your Xperia M4 Aqua.</p>
<p>Press (and keep pressed) the <strong>volume up</strong> button, and connect the device to your computer with the USB cable.
This prevent the device from starting normally, instead it will just sit in the bootloader mode and wait.
The LED next to the USB connector turns red, then blue, and remains blue like that.</p>
<p>Check that the device is recognized and ready to communicate by running this command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>fastboot<span class="w"> </span>devices
AB123CDEF4<span class="w"> </span>fastboot
</code></pre></div>
<p>Now enter the unlock key to unlock the bootloader.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>fastboot<span class="w"> </span>-i<span class="w"> </span>0x0fce<span class="w"> </span>oem<span class="w"> </span>unlock<span class="w"> </span>0x<your-unlock-key>
...
OKAY<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">4</span>.487s<span class="o">]</span>
finished.<span class="w"> </span>total<span class="w"> </span>time:<span class="w"> </span><span class="m">4</span>.487s
</code></pre></div>
<p>Done! Start your phone, it's fresh as if it was out of the factory!</p>
<h2>Install TWRP</h2>
<p>Now that the bootloader is unlocked, we will be able to install a custom recovery software.</p>
<p>If you're wondering what the hell is that, you may have a look here:<br>
<a href="http://geekwarning.com/mobiles/twrpcwm-recovery-on-sony-xperia-m4-aqua/">http://geekwarning.com/mobiles/twrpcwm-recovery-on-sony-xperia-m4-aqua/</a></p>
<p>We need a custom recovery software to install a superuser app.
There seem to be many custom recovery for Android. We have to pickup one of them.
We're gonna install TWRP, as it seems to be fairly popular.</p>
<p>At the moment of this writing, there's no official support for the Xperia M4 in TWRP.
However, there's a beta that is reported to work well. It's available here:<br>
<a href="http://projectmarshmallow.altervista.org/twrp-sony-m4-aqua/">http://projectmarshmallow.altervista.org/twrp-sony-m4-aqua/</a></p>
<p>A discussion and feedback on this beta is available here:<br>
<a href="http://forum.xda-developers.com/m4-aqua/development/recovery-twrp-touch-recovery-sony-m4-t3229873">http://forum.xda-developers.com/m4-aqua/development/recovery-twrp-touch-recovery-sony-m4-t3229873</a></p>
<p>Download the TWRP image.</p>
<p>Ensure the USB debugging is enabled on your M4.</p>
<p>Reboot you Xperia M4 in bootloader mode. You know how to do that by now, right?
Turn off the device, then press the volume up button and plug it to your computer.</p>
<p>Here's another way to do that. Assuming your device is already up and running,
and is connected to your computer, just type in a terminal:</p>
<div class="highlight"><pre><span></span><code><span class="n">adb</span><span class="w"> </span><span class="n">reboot</span><span class="w"> </span><span class="n">bootloader</span>
</code></pre></div>
<p>Now that the device is sitting in bootloader, install TWRP:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">fastboot</span><span class="w"> </span><span class="n">flash</span><span class="w"> </span><span class="n">recovery</span><span class="w"> </span><span class="n">recovery</span><span class="o">.</span><span class="n">img</span>
<span class="n">target</span><span class="w"> </span><span class="n">reported</span><span class="w"> </span><span class="nb">max</span><span class="w"> </span><span class="n">download</span><span class="w"> </span><span class="n">size</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="mi">268435456</span><span class="w"> </span><span class="n">bytes</span>
<span class="n">sending</span><span class="w"> </span><span class="s1">'recovery'</span><span class="w"> </span><span class="p">(</span><span class="mi">24640</span><span class="w"> </span><span class="n">KB</span><span class="p">)</span><span class="o">...</span>
<span class="n">OKAY</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">0.775</span><span class="n">s</span><span class="p">]</span>
<span class="n">writing</span><span class="w"> </span><span class="s1">'recovery'</span><span class="o">...</span>
<span class="n">OKAY</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">1.087</span><span class="n">s</span><span class="p">]</span>
<span class="n">finished</span><span class="o">.</span><span class="w"> </span><span class="n">total</span><span class="w"> </span><span class="n">time</span><span class="p">:</span><span class="w"> </span><span class="mf">1.862</span><span class="n">s</span>
</code></pre></div>
<p>(I assume here that the TWRP image you downloaded is named <code>recovery.img</code>,
and that it's in the current directory)</p>
<p>That's it! Done already! You replaced the stock recovery software by a custom one: TWRP.</p>
<p>You can reboot the phone with the command:</p>
<div class="highlight"><pre><span></span><code>fastboot reboot
</code></pre></div>
<h2>Install SuperSU</h2>
<p>And now we need to install a superuser management app, and it seems that the most popular
out there is SuperSU. So let's go for it.</p>
<p>A good tutorial for that:<br>
<a href="http://www.howtogeek.com/115297/how-to-root-your-android-why-you-might-want-to/">http://www.howtogeek.com/115297/how-to-root-your-android-why-you-might-want-to/</a></p>
<p>A few words on what is Super SU:<br>
<a href="http://www.android.gs/install-supersu-on-android-devices-to-improve-superuser-access/">http://www.android.gs/install-supersu-on-android-devices-to-improve-superuser-access/</a></p>
<p>Basically, it's what we need to have root access on the device, and to provide root access
to other third-party apps.</p>
<p>SuperSU is available from the Google Playstore, so you could install it like any other app,
however if you do so you will only have the frontend. The backend (the <code>su</code> command)
can't be installed that way, it needs to be installed using a custom recovery software.</p>
<p>Discussion and explanation about that:<br>
<a href="http://android.stackexchange.com/q/96736">http://android.stackexchange.com/q/96736</a></p>
<p>So, at first, download the SuperSU zip archive from this page:<br>
<a href="https://download.chainfire.eu/supersu">https://download.chainfire.eu/supersu</a></p>
<p>Now, let's boot your Xperia M4 in recovery mode. To do that,
I had to turn off the Xperia, then turn it on by holding <strong>power button</strong>
and <strong>volume down</strong> button simultaneously. Doing that, the phone boots in
recovery mode, and we get a nice screen from TWRP.</p>
<p>On the first run, TWRP asks you if you want to run in read-only mode. You can say yes.</p>
<p>Now, connect the device to your computer with USB cable.</p>
<p>From your computer, mount the Xperia M4 device if needed.
You should be able to see the Xperia in your file browser,
assuming that the MTP is properly installed on your computer...</p>
<p>Now, just copy the SuperSU archive (<code>UPDATE-SuperSU-v2.65-20151226141550.zip</code> at the moment of this writing)
somewhere on your Xperia M4.</p>
<p>Then, on your Xperia M4, currently running TWRP, choose <code>Install</code>, navigate in the directories to where you copied the zip,
and choose to install it.</p>
<p>Done! SuperSU is installed!</p>
<p>Now you can reboot the device, and you'll see that SuperSU has been added to the apps.</p>
<h2>Cleaning the Xperia M4</h2>
<p>We're almost there, be patient. Now, we need to install an application that allow to uninstall
the bloatware. I choosed <code>System app remover (ROOT)</code> since it was well ranked in the Play Store.</p>
<p>After that, launch it, and you will see that there are many many things that are installed.</p>
<p>What can be safely removed? Hard to say, it depends on what apps you don't need,
and how reckless you are with the system app that you have no clue about...</p>
<p>There's a fairly good list available here:<br>
<a href="http://forum.xda-developers.com/z3/general/apps-safe-to-uninstall-z3-t3019684">http://forum.xda-developers.com/z3/general/apps-safe-to-uninstall-z3-t3019684</a></p>
<p>And a more advanced discussion, with a longer and more reckless list, here:<br>
<a href="http://forum.xda-developers.com/crossdevice-dev/sony/wip-debloat-lp-690-correct-functioning-t3076161">http://forum.xda-developers.com/crossdevice-dev/sony/wip-debloat-lp-690-correct-functioning-t3076161</a></p>
<p>Before doing anything foolish, I backed up my fresh Android on the SD-Card with TWRP.
You should do the same, since now we start doing things that may break the system.
Well, it's no big deal, I guess you can always re-install a clean Sony Xperia M4 firmware
if you mess up the phone. But backing up with TWRP saves time.</p>
<p>Ok, anyway, here's my list of apps removed. Yours may be different.</p>
<ul>
<li>Active Clip</li>
<li>Ant HAL Service</li>
<li>AR effect</li>
<li>AR fun</li>
<li>Backup & restore</li>
<li>Black Hole</li>
<li>Bubbles</li>
<li>Calculator (Small Apps)</li>
<li>Chrome (I use Firefox)</li>
<li>Clocks</li>
<li>Creative effect</li>
<li>DrmDialogs</li>
<li>Enchanted forest</li>
<li>Enterprise Single Sign On</li>
<li>Exchange Services</li>
<li>File Commander</li>
<li>Google lyrics extension</li>
<li>Google Play Movies</li>
<li>Google Text-to-speech Engine</li>
<li>Hangouts</li>
<li>Live Wallpaper Picker</li>
<li>Lounge</li>
<li>Movie Creator</li>
<li>Multi Camera</li>
<li>Music Visualization Wallpapers</li>
<li>my Xperia</li>
<li>OfficeSuite</li>
<li>OMA Client Provisioning</li>
<li>Oma Download Client</li>
<li>OmaV1AgentDownloadService</li>
<li>Phase Beam</li>
<li>Portrait retouch</li>
<li>Retail Demo</li>
<li>Setup guide</li>
<li>Setup Wizard</li>
<li>Simple Home</li>
<li>Sketch</li>
<li>Small App *</li>
<li>Software update</li>
<li>Sound Photo</li>
<li>Street View</li>
<li>Support</li>
<li>TalkBack</li>
<li>Timer</li>
<li>Top Contacts provider</li>
<li>Touch Block</li>
<li>Trusted Face</li>
<li>Video</li>
<li>Video Editor</li>
<li>Visualiser</li>
<li>What's New</li>
<li>Wikipedia extension</li>
<li>Xperia Calendar sync</li>
<li>Xperia Music Likes</li>
<li>Xperia Social Engine Photos</li>
<li>Xperia camera add-ons</li>
<li>Xperia Chinese keyboard</li>
<li>Xperia extension for Throw</li>
<li>Xperia Lounge Pass</li>
<li>Xperia wallpapers</li>
<li>Xperia Web Runtime</li>
<li>Xperia with Facebook</li>
<li>Xperia with Twitter</li>
<li>Youtube extension</li>
<li>Youtube karaoke extension</li>
</ul>
<p>Also, in the app remover, you can sort apps by path. Then you can remove anything with a path
starting with <code>/system/vendor/overlay/</code> and <code>/system/vendor/app</code>.</p>
<h2>Conclusion</h2>
<p>After removing all that things, how much space did I gain? Around 250MB.</p>
<p>Wait... What? 250MB??? All that efforts for so little result?</p>
<p>Yeah, that's a bit disappointing... Well we can rejoyce by remembering that we also saved some RAM,
therefore some CPU, therefore some battery life. But how much exactly, no idea. Probably very little.</p>
<p>I guess now, the best hope for a really lighter Android install would be to have
a custom rom to install. An official CyanogenMod would be great. I'll keep an eye on that.</p>
<p>In the meantime, I'll keep the number of apps installed as small as possible,
and throw as much as I can on the SD-Card. If it's not enough, I'll probably
give a try to <a href="https://play.google.com/store/apps/details?id=com.buak.Link2SD">Links2SD</a>.</p>
<p>Cheers!</p>Building And Installing Rockbox2016-04-27T00:00:00+00:002016-04-27T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-04-27:/2016/04/27/building-and-installing-rockbox/<p>Here is a very quick howto that walks you through the few steps
needed to compile the <a href="http://www.rockbox.org">Rockbox</a> firmware from source.</p>
<p>Here is a very quick howto that walks you through the few steps
needed to compile the <a href="http://www.rockbox.org">Rockbox</a> firmware from source.</p>
<h2>Get yourself ready</h2>
<p>Just make sure that you have all the basic tools installed.</p>
<div class="highlight"><pre><span></span><code><span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">build</span><span class="o">-</span><span class="n">essential</span><span class="w"> </span><span class="n">libtool</span><span class="o">-</span><span class="n">bin</span><span class="w"> </span><span class="n">flex</span><span class="w"> </span><span class="n">bison</span><span class="w"> </span><span class="n">texinfo</span>
</code></pre></div>
<h2>Get the source code</h2>
<p>Rockbox is versioned on github:</p>
<div class="highlight"><pre><span></span><code>git clone git://git.rockbox.org/rockbox
</code></pre></div>
<h2>Build the toolchain</h2>
<p>This is documented on the official wiki: <a href="http://www.rockbox.org/wiki/CrossCompiler">http://www.rockbox.org/wiki/CrossCompiler</a>.</p>
<div class="highlight"><pre><span></span><code><span class="n">cd</span><span class="w"> </span><span class="n">rockbox</span>
<span class="k">export</span><span class="w"> </span><span class="n">RBDEV_PREFIX</span><span class="o">=</span><span class="s2">"/opt/rockbox"</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="s2">"$RBDEV_PREFIX"</span>
<span class="c1"># Set yourself as the owner of the build directory to avoid compiling as root</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">chown</span><span class="w"> </span><span class="o">$</span><span class="n">USER</span><span class="p">:</span><span class="o">$</span><span class="n">USER</span><span class="w"> </span><span class="s2">"$RBDEV_PREFIX"</span>
<span class="n">cd</span><span class="w"> </span><span class="n">tools</span>
<span class="o">./</span><span class="n">rockboxdev</span><span class="o">.</span><span class="n">sh</span>
<span class="c1"># Set ownership back to root</span>
<span class="n">sudo</span><span class="w"> </span><span class="n">chown</span><span class="w"> </span><span class="o">-</span><span class="n">R</span><span class="w"> </span><span class="n">root</span><span class="p">:</span><span class="n">root</span><span class="w"> </span><span class="s2">"$RBDEV_PREFIX"</span>
</code></pre></div>
<p>It may happen that the build fails with such error:</p>
<div class="highlight"><pre><span></span><code><span class="cp">../../gcc-4.4.4/gcc/toplev.c: At top level:</span>
<span class="cp">../../gcc-4.4.4/gcc/toplev.c:536:1: error: redefinition of ‘floor_log2’</span>
floor_log2 (unsigned HOST_WIDE_INT x)
</code></pre></div>
<p>In such case, you need to modify the CFLAGS used to build the toolchain, as suggested on
this page: <a href="http://www.rockbox.org/irc/logs/rockbox-20150803.txt">http://www.rockbox.org/irc/logs/rockbox-20150803.txt</a>. At the moment it's
at the line 214 of the file <code>rockboxdev.sh</code>. Replace <code>CFLAGS=-U_FORTIFY_SOURCE</code> by
<code>CFLAGS="-U_FORTIFY_SOURCE -fgnu89-inline"</code>.</p>
<h2>Build the firmware</h2>
<p>This is documented on the official wiki: <a href="http://www.rockbox.org/wiki/HowToCompile">http://www.rockbox.org/wiki/HowToCompile</a></p>
<div class="highlight"><pre><span></span><code><span class="n">mkdir</span><span class="w"> </span><span class="n">build</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">cd</span><span class="w"> </span><span class="n">build</span>
<span class="n">PATH</span><span class="o">=/</span><span class="n">opt</span><span class="o">/</span><span class="n">rockbox</span><span class="o">/</span><span class="n">bin</span><span class="p">:</span><span class="o">$</span><span class="n">PATH</span>
<span class="o">../</span><span class="n">tools</span><span class="o">/</span><span class="n">configure</span>
<span class="n">make</span><span class="w"> </span><span class="o">-</span><span class="n">j</span>
</code></pre></div>
<p>Then, you need to pack all the firmware files in an archive. For the small archive,
use the target <code>zip</code>. For a bigger archive with additional stuff (like plenty of fonts),
use the target <code>fullzip</code>.</p>
<div class="highlight"><pre><span></span><code>make fullzip
</code></pre></div>
<h2>Install the firmware on your device</h2>
<p>I assume you already installed Rockbox before on your favorite audio player,
and you're just updating the firmware right now. It means that I won't explain
how to install the bootloader.</p>
<p>So just plug your device, mount it. Here we assume it's mounted at the location
<code>/media/$USER/iPodClassic</code> (yep I've got an iPod, now you know).</p>
<p>Installing the firmware is a simple as unpacking the archive at the root your
device filesystem. The archive only contains a directory named <code>.rockbox</code> that
holds the firmware files. If you really want to restart from scratch, you can
delete this directory beforehand.</p>
<div class="highlight"><pre><span></span><code>rm -fr /media/$USER/iPodClassic/.rockbox
</code></pre></div>
<p>Otherwise, just unpack the archive and overwrite everything.</p>
<div class="highlight"><pre><span></span><code><span class="n">unzip</span><span class="w"> </span><span class="n">rockbox</span><span class="o">-</span><span class="k">full</span><span class="p">.</span><span class="n">zip</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">/</span><span class="n">media</span><span class="o">/</span><span class="err">$</span><span class="k">USER</span><span class="o">/</span><span class="n">iPodClassic</span><span class="o">/</span>
<span class="w"> </span><span class="nl">Archive</span><span class="p">:</span><span class="w"> </span><span class="n">rockbox</span><span class="o">-</span><span class="k">full</span><span class="p">.</span><span class="n">zip</span>
<span class="w"> </span><span class="nf">replace</span><span class="w"> </span><span class="o">/</span><span class="n">media</span><span class="o">/</span><span class="k">user</span><span class="o">/</span><span class="n">iPodClassic</span><span class="o">/</span><span class="p">.</span><span class="n">rockbox</span><span class="o">/</span><span class="n">rockbox</span><span class="o">-</span><span class="n">info</span><span class="p">.</span><span class="n">txt</span><span class="vm">?</span><span class="w"> </span><span class="o">[</span><span class="n">y</span><span class="o">]</span><span class="n">es</span><span class="p">,</span><span class="w"> </span><span class="o">[</span><span class="n">n</span><span class="o">]</span><span class="n">o</span><span class="p">,</span><span class="w"> </span><span class="o">[</span><span class="n">A</span><span class="o">]</span><span class="n">ll</span><span class="p">,</span><span class="w"> </span><span class="o">[</span><span class="n">N</span><span class="o">]</span><span class="n">one</span><span class="p">,</span><span class="w"> </span><span class="o">[</span><span class="n">r</span><span class="o">]</span><span class="nl">ename</span><span class="p">:</span><span class="w"> </span><span class="n">A</span>
<span class="w"> </span><span class="n">inflating</span><span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">...</span>
<span class="n">sync</span>
</code></pre></div>
<p>If unsure, backup before doing anything foolish!</p>
<h2>Reboot your audio player</h2>
<p>That's it. Just reboot and unplug your audio player.</p>
<p>If you're the owner of an iPod Classic/6g like me, and tend to forget how to restart it,
here is the procedure: hold the "Menu" and "Select" buttons for about 5 seconds.</p>First Debian Package2016-01-14T00:00:00+00:002016-01-14T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2016-01-14:/2016/01/14/first-debian-package/<p>In this post we learn how to create our first Debian package.
We get lost in the maze of Debian documentation available out there.
We try not to fall asleep reading these endless pages.
And finally we succeed creating a simple yet beautiful Debian package.</p>
<p>In this post we learn how to create our first Debian package.
We get lost in the maze of Debian documentation available out there.
We try not to fall asleep reading these endless pages.
And finally we succeed creating a simple yet beautiful Debian package.</p>
<p><a href="https://github.com/nicklan/pnmixer">PNMixer</a> is being packaged here,
as it's a project I'm working on, and it's indeed my first Debian package.</p>
<h2>References</h2>
<p>The most complete piece of information out there is the
<a href="https://www.debian.org/doc/manuals/maint-guide/">Debian New Maintainers' Guide</a>.
If you're a noob like me, you'd better take some time to read it.
I refer to it often in this post.</p>
<p>We also use a little bit of the <a href="https://www.debian.org/doc/debian-policy/">Debian Policy Manual</a>.</p>
<p>The <a href="https://www.debian.org/doc/manuals/developers-reference/tools">Overview of Debian Maintainer Tools</a>
can be a good reading.</p>
<p>I also found a good tutorial at this address, it's quick but there are every steps you need:<br>
<a href="https://coderwall.com/p/urkybq/how-to-create-debian-package-from-source">How to create debian package from source</a></p>
<h2>Installing stuff</h2>
<p>We're gonna use <code>dh_make</code> (dh stands for Debian helper) to make the package.
Beware of the tricky naming, the package name has an hyphen while the binary has an underscore.</p>
<div class="highlight"><pre><span></span><code>apt-get install dh-make
</code></pre></div>
<p>To package a software that uses autotools, we will need <code>dh-autoreconf</code>.</p>
<div class="highlight"><pre><span></span><code>apt-get install dh-autoreconf
</code></pre></div>
<p>We will also need plenty of other utilities, and they all come from the <code>devscripts</code> package.</p>
<div class="highlight"><pre><span></span><code>apt-get install devscripts
</code></pre></div>
<p>To check the package for errors, we will use <code>lintian</code>.</p>
<div class="highlight"><pre><span></span><code>apt-get install lintian
</code></pre></div>
<h2>Creating the package skeleton</h2>
<p>Before using <code>dh_make</code> we may define some environment variables. <code>dh_make</code> will pick them up,
and use them automatically, making our task easier. You can even add that to your <code>.bashrc</code>
if you feel like packaging often.</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">DEBEMAIL</span><span class="o">=</span><span class="s1">'your@mail.com'</span>
<span class="k">export</span><span class="w"> </span><span class="n">DEBFULLNAME</span><span class="o">=</span><span class="s1">'Your Name'</span>
</code></pre></div>
<p>OK, so now, let's grab the archive of the software we're packaging.</p>
<div class="highlight"><pre><span></span><code><span class="n">wget</span><span class="w"> </span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">nicklan</span><span class="o">/</span><span class="n">pnmixer</span><span class="o">/</span><span class="n">releases</span><span class="o">/</span><span class="n">download</span><span class="o">/</span><span class="n">v0</span><span class="o">.</span><span class="mi">6</span><span class="o">/</span><span class="n">pnmixer</span><span class="o">-</span><span class="mf">0.6</span><span class="o">.</span><span class="mf">1.</span><span class="n">tar</span><span class="o">.</span><span class="n">gz</span>
</code></pre></div>
<p>Unpack it, enter the directory:</p>
<div class="highlight"><pre><span></span><code>tar -xzf pnmixer-0.6.1.tar.gz
cd pnmixer-0.6.1
</code></pre></div>
<p>Then create the package skeleton. </p>
<div class="highlight"><pre><span></span><code>dh_make --copyright gpl3 -f ../pnmixer-0.6.1.tar.gz
</code></pre></div>
<p>Here, you just have one question to answer: the type of package.
A quick <code>man dh_make</code> will give you hints. For PNMixer, the package type is "single binary".</p>
<p><code>dh_make</code> created a <code>debian</code> sub-directory: this is the skeleton, holding all the files needed to
build the package. You can have a look at it, don't be shy.</p>
<div class="highlight"><pre><span></span><code>ls debian
</code></pre></div>
<p>But also (and that's a little bit unusual), <code>dh_make</code> created some files in the <em>parent</em> directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>..
pnmixer-0.6.1
pnmixer_0.6.1.orig.tar.gz
pnmixer-0.6.1.tar.gz
</code></pre></div>
<p>You have to get used to that, because that's the way the Debian helpers work.
So you'd better have a clean parent directory, otherwise you'll be confused quickly.
Just for you to know, the package will end up here after it's built.</p>
<p>OK, now let's start the real job: edit the skeleton files created by <code>dh_make</code>.</p>
<h2>The <em>control</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#control">https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#control</a>.</p>
<p>At first, the most important: read the doc I just mentioned :) Then, you can
safely edit the <code>control</code> file:</p>
<div class="highlight"><pre><span></span><code>vi debian/control
</code></pre></div>
<p>Start by filling the obvious fields, like <code>Maintainer</code> or <code>Homepage</code>.
Uncomment the <code>VCS-*</code> fields if you intend to submit your package to Debian.</p>
<p>To fill the <code>Section</code>, have a look at the <a href="https://packages.debian.org/sid/">Debian list of sections</a>.</p>
<p>It's all easy to fill, except for the field <code>Build-Depends</code>.
As you can see, it is already populated with a few things:</p>
<ul>
<li><code>debhelper</code> because we're building the package using Debian helpers.</li>
<li><code>autotools-dev</code> because PNMixer uses the autotools.</li>
</ul>
<p><code>build-essential</code> is not mentioned, but it is implicit for binary packages.
It is mentioned <a href="https://www.debian.org/doc/debian-policy/ch-relationships.html#s-sourcebinarydeps">here</a>
and <a href="https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#control">here</a>.</p>
<p>At first, if the package uses <code>autoreconf</code>, it is recommanded to use <code>dh-autoreconf</code> instead
of <code>autotools-dev</code>. It is not clear in the documentation, but I was told to do so by a
reviewer of my package. Then I found the <a href="https://wiki.debian.org/Autoreconf">Debian Autoreconf Wiki page</a>,
where everything is explained.</p>
<p>OK, so good-bye autotools-dev.</p>
<div class="highlight"><pre><span></span><code><span class="n">sed</span><span class="w"> </span><span class="o">-</span><span class="n">i</span><span class="w"> </span><span class="s1">'s/autotools-dev/dh-autoreconf/'</span><span class="w"> </span><span class="n">debian</span><span class="o">/</span><span class="n">control</span>
</code></pre></div>
<p>Now we must add the other dependencies of our package.
Libraries are probably easy to figure if you know well the program you're packaging.
But the <em>exact</em> list of build tools needed is more complicated than it seems.</p>
<p>Indeed, what exactly implies <code>build-essential</code>? And <code>dh-autoreconf</code>?</p>
<h3>build-essential</h3>
<p><code>apt-cache show build-essential</code> outputs an interesting line:</p>
<blockquote>
<p>the real definition is in the Debian Policy Manual.</p>
</blockquote>
<p>OK, where's the Debian Policy? I googled it, and after digging a little bit,
here is the info we're after:
<a href="https://www.debian.org/doc/debian-policy/ch-source.html#s-pkg-relations">https://www.debian.org/doc/debian-policy/ch-source.html#s-pkg-relations</a></p>
<blockquote>
<p>The required packages are called build-essential, and an informational list can be found in
/usr/share/doc/build-essential/list (which is contained in the build-essential package).</p>
</blockquote>
<p>Alright, let's have a look at the file!</p>
<div class="highlight"><pre><span></span><code>vi /usr/share/doc/build-essential/list
</code></pre></div>
<p>From what I understand, it contains <code>make</code>, <code>gcc</code> and <code>libc6-dev</code> (the C standard library).</p>
<h3>dh-autoreconf</h3>
<p>From the wiki page mentioned above:</p>
<blockquote>
<p>In general dh-autoreconf is a superset of autotools-dev...</p>
</blockquote>
<p>Ok, so let's try to know more about <code>autotools-dev</code>.
<code>apt-cache show autotools-dev</code> is our friend in this matter, let me quote it:</p>
<blockquote>
<p>It also documents in /usr/share/doc/autotools-dev/README.Debian.gz
best practices and guidelines for using autoconf, automake and
friends on Debian packages. This is a must-read for any developers
packaging software that uses the GNU autotools, or GNU gettext.</p>
</blockquote>
<p>OK, let's go there:</p>
<div class="highlight"><pre><span></span><code><span class="n">vi</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">share</span><span class="o">/</span><span class="n">doc</span><span class="o">/</span><span class="n">autotools</span><span class="o">-</span><span class="n">dev</span><span class="o">/</span><span class="n">README</span><span class="o">.</span><span class="n">Debian</span><span class="o">.</span><span class="n">gz</span>
</code></pre></div>
<p>It's a big piece of writing, that I barely read to be honest.
But from my average knowledge of what are the autotools, I assume it
provides <code>autoconf</code>, <code>automake</code> and <code>gettext</code>.</p>
<h3>what's missing then?</h3>
<p>Alright, after this little hide & seek game through the documentation,
let's get back to our matter. What other build tool do I need for my package?
Answer is: <code>pkg-config</code> and <code>intltool</code>. I believe there not implied by any of
the dependencies we just went through, so I wrote them down.</p>
<h3>libraries</h3>
<p>Figuring out the libraries you package need may be easy if it's a little piece
of software and you know it well. A quick look at the <code>autoconf.ac</code> should tell
you the truth. For PNMixer, I could write the libraries needed without the need
of any tool.</p>
<p>However, being curious, I gave a try to <code>dpkg-depcheck</code>, as suggested by the Debian manual.</p>
<div class="highlight"><pre><span></span><code>pkg-depcheck -d ./configure
</code></pre></div>
<p>The output is rather... crowded. It's difficult to pick up the right things
in such a long list. So, I don't know if there a better tool for the job,
but I was not really convinced by this one.</p>
<h2>The final <em>control</em> file</h2>
<div class="highlight"><pre><span></span><code><span class="n">Source</span><span class="o">:</span><span class="w"> </span><span class="n">pnmixer</span>
<span class="n">Section</span><span class="o">:</span><span class="w"> </span><span class="n">sound</span>
<span class="n">Priority</span><span class="o">:</span><span class="w"> </span><span class="n">optional</span>
<span class="n">Maintainer</span><span class="o">:</span><span class="w"> </span><span class="n">My</span><span class="w"> </span><span class="n">Name</span><span class="w"> </span><span class="o"><</span><span class="n">my</span><span class="err">@</span><span class="n">name</span><span class="o">.</span><span class="na">com</span><span class="o">></span>
<span class="n">Build</span><span class="o">-</span><span class="n">Depends</span><span class="o">:</span><span class="w"> </span><span class="n">debhelper</span><span class="w"> </span><span class="o">(>=</span><span class="w"> </span><span class="mi">9</span><span class="o">),</span><span class="w"> </span><span class="n">dh</span><span class="o">-</span><span class="n">autoreconf</span><span class="o">,</span><span class="w"> </span><span class="n">pkg</span><span class="o">-</span><span class="n">config</span><span class="o">,</span><span class="w"> </span><span class="n">intltool</span><span class="o">,</span>
<span class="w"> </span><span class="n">libasound2</span><span class="o">-</span><span class="n">dev</span><span class="o">,</span><span class="w"> </span><span class="n">libglib2</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">dev</span><span class="o">,</span><span class="w"> </span><span class="n">libgtk</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">dev</span><span class="w"> </span><span class="o">(>=</span><span class="w"> </span><span class="mf">3.6</span><span class="o">),</span><span class="w"> </span><span class="n">libnotify</span><span class="o">-</span><span class="n">dev</span><span class="o">,</span><span class="w"> </span><span class="n">libx11</span><span class="o">-</span><span class="n">dev</span>
<span class="n">Standards</span><span class="o">-</span><span class="n">Version</span><span class="o">:</span><span class="w"> </span><span class="mf">3.9</span><span class="o">.</span><span class="mi">6</span>
<span class="n">Homepage</span><span class="o">:</span><span class="w"> </span><span class="n">https</span><span class="o">://</span><span class="n">github</span><span class="o">.</span><span class="na">com</span><span class="sr">/nicklan/</span><span class="n">pnmixer</span>
<span class="n">Vcs</span><span class="o">-</span><span class="n">Git</span><span class="o">:</span><span class="w"> </span><span class="n">git</span><span class="o">://</span><span class="n">anonscm</span><span class="o">.</span><span class="na">debian</span><span class="o">.</span><span class="na">org</span><span class="sr">/collab-maint/</span><span class="n">pnmixer</span><span class="o">.</span><span class="na">git</span>
<span class="n">Vcs</span><span class="o">-</span><span class="n">Browser</span><span class="o">:</span><span class="w"> </span><span class="n">https</span><span class="o">://</span><span class="n">anonscm</span><span class="o">.</span><span class="na">debian</span><span class="o">.</span><span class="na">org</span><span class="sr">/gitweb/?p=collab-maint/</span><span class="n">pnmixer</span><span class="o">.</span><span class="na">git</span><span class="o">;</span><span class="n">a</span><span class="o">=</span><span class="n">summary</span>
<span class="n">Package</span><span class="o">:</span><span class="w"> </span><span class="n">pnmixer</span>
<span class="n">Architecture</span><span class="o">:</span><span class="w"> </span><span class="n">any</span>
<span class="n">Depends</span><span class="o">:</span><span class="w"> </span><span class="n">$</span><span class="o">{</span><span class="n">shlibs</span><span class="o">:</span><span class="n">Depends</span><span class="o">},</span><span class="w"> </span><span class="n">$</span><span class="o">{</span><span class="n">misc</span><span class="o">:</span><span class="n">Depends</span><span class="o">}</span>
<span class="n">Description</span><span class="o">:</span><span class="w"> </span><span class="n">Simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span>
<span class="w"> </span><span class="n">PNMixer</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">simple</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">application</span><span class="w"> </span><span class="n">designed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span><span class="o">.</span>
<span class="w"> </span><span class="n">It</span><span class="w"> </span><span class="n">integrates</span><span class="w"> </span><span class="n">nicely</span><span class="w"> </span><span class="n">into</span><span class="w"> </span><span class="n">desktop</span><span class="w"> </span><span class="n">environments</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">don</span><span class="s1">'t have a panel that</span>
<span class="s1"> supports applets and therefore can'</span><span class="n">t</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">mixer</span><span class="w"> </span><span class="n">applet</span><span class="o">.</span><span class="w"> </span><span class="n">In</span><span class="w"> </span><span class="n">particular</span><span class="w"> </span><span class="n">it</span><span class="err">'</span><span class="n">s</span>
<span class="w"> </span><span class="n">been</span><span class="w"> </span><span class="n">used</span><span class="w"> </span><span class="n">quite</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">lot</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">fbpanel</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="n">tint2</span><span class="o">,</span><span class="w"> </span><span class="n">but</span><span class="w"> </span><span class="n">should</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">fine</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">any</span>
<span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">tray</span><span class="o">.</span>
<span class="w"> </span><span class="o">.</span>
<span class="w"> </span><span class="n">PNMixer</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">designed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">work</span><span class="w"> </span><span class="n">on</span><span class="w"> </span><span class="n">systems</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="n">ALSA</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">sound</span><span class="w"> </span><span class="n">management</span><span class="o">.</span>
<span class="w"> </span><span class="n">Any</span><span class="w"> </span><span class="n">other</span><span class="w"> </span><span class="n">sound</span><span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="n">like</span><span class="w"> </span><span class="n">OSS</span><span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">FFADO</span><span class="o">,</span><span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">sound</span><span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="n">like</span><span class="w"> </span><span class="n">PulseAudio</span>
<span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">Jackd</span><span class="o">,</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">currently</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">supported</span><span class="w"> </span><span class="o">(</span><span class="n">patches</span><span class="w"> </span><span class="n">welcome</span><span class="o">).</span>
</code></pre></div>
<h2>The <em>copyright</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#copyright">https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#copyright</a></p>
<p>OK! Now it's time to take care of the <code>copyright</code> file. Read the doc mentioned above,
then edit the file and add what's missing.</p>
<div class="highlight"><pre><span></span><code>vi debian/copyright
</code></pre></div>
<p>Don't forget to read the comments at the end of the file, and remove it afterward.</p>
<h2>The <em>changelog</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog">https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog</a></p>
<p>Easy to setup, just stick to the syntax and you'll be fine. Here's my changelog as an example:</p>
<div class="highlight"><pre><span></span><code><span class="n">pnmixer</span><span class="w"> </span><span class="p">(</span><span class="mf">0.6.1</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">unstable</span><span class="p">;</span><span class="w"> </span><span class="n">urgency</span><span class="o">=</span><span class="n">low</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Initial</span><span class="w"> </span><span class="k">release</span><span class="w"> </span><span class="p">(</span><span class="nl">Closes</span><span class="p">:</span><span class="w"> </span><span class="n">#745669</span><span class="p">)</span>
<span class="w"> </span><span class="c1">-- My Name <my@name.com> Sun, 06 Dec 2015 19:18:02 +0700</span>
</code></pre></div>
<p>The bug number here is the ITP (Intent To Package) bug number. It makes sense if you want your
package to make it to Debian upstream, and therefore follow the procedure for package submission.</p>
<h2>The <em>rules</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#rules">https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#rules</a></p>
<p>As you can see, this is a Makefile (hopefully you're familiar with the syntax).</p>
<p>You can remove the <code>rules.dh7</code> file, which is the same as <code>rules</code>.</p>
<p>Then, edit the <code>rules</code> file. You can remove everything that's not needed,
ie every comment line, to make the file more readable. I was told by a reviewer
to remove also the two uncommented lines, so in the end there's nothing
left in the file, except the build rule.</p>
<p>Now, you may remember that earlier on this page, we decided to use <code>dh-autoreconf</code>
instead of <code>autotools-dev</code>. It's now time to make this change effective.
Edit the main rule so that <code>dh</code> is invoked with autoreconf instead of autotools-dev.</p>
<div class="highlight"><pre><span></span><code><span class="c">#!/usr/bin/make -f</span>
<span class="nf">%</span><span class="o">:</span>
<span class="w"> </span>dh<span class="w"> </span><span class="nv">$@</span><span class="w"> </span>--with<span class="w"> </span>autoreconf
</code></pre></div>
<h2>The <em>docs</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dother.en.html#docs">https://www.debian.org/doc/manuals/maint-guide/dother.en.html#docs</a></p>
<p>Specify documentation files that should be installed. By default, it already includes
some standard files. Just add other files if you want. As an example:</p>
<div class="highlight"><pre><span></span><code>echo 'README.md' >> debian/docs
</code></pre></div>
<h2>The man page</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dother.en.html#manpage">https://www.debian.org/doc/manuals/maint-guide/dother.en.html#manpage</a></p>
<p>If your package doesn't provide a man page, you should write one.
Example files are provided, written in different languages, it's up to you to pickup
your favorite language and write your manual page. As a quick example:</p>
<div class="highlight"><pre><span></span><code>cd debian
mv manpage.1.ex pnmixer.1
vi pnmixer.1
man -l pnmixer.1
</code></pre></div>
<p>Then, list your manual pages in the file <code>package.manpages</code>. So in our case, we just
have one file to list:</p>
<div class="highlight"><pre><span></span><code>echo 'debian/pnmixer.1' > debian/pnmixer.manpages
</code></pre></div>
<h2>The <em>watch</em> file</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dother.en.html#watch">https://www.debian.org/doc/manuals/maint-guide/dother.en.html#watch</a></p>
<p>The <code>watch</code> file is not so easy to setup. Some common upstream source sites are mentioned
in the <code>watch.ex</code> file, but Github is not amongst them. However you can find more here:
<a href="https://wiki.debian.org/debian/watch">https://wiki.debian.org/debian/watch</a></p>
<p>To test your setup, type something like that:</p>
<div class="highlight"><pre><span></span><code>uscan pnmixer-0.6.1 --report --verbose
</code></pre></div>
<h2>The <em>menu</em> file</h2>
<p>At the moment of this writing, it seems that the menu file shouldn't be used anymore if
your package already provides a desktop file. It was not very clear at the beginning.
If you add a menu file, and check your package with <code>lintian</code> (see below),
you will have this warning:</p>
<div class="highlight"><pre><span></span><code>The command is listed both in a menu file and a desktop file
Per the tech-ctte decision on #741573, this is now prohibited.
Please remove the reference from the menu file.
</code></pre></div>
<p>But after removing the <code>command</code> line from the menu file, I get an error:</p>
<div class="highlight"><pre><span></span><code><span class="n">pnmixer</span><span class="o">:</span><span class="w"> </span><span class="n">menu</span><span class="o">-</span><span class="n">item</span><span class="o">-</span><span class="n">missing</span><span class="o">-</span><span class="n">required</span><span class="o">-</span><span class="n">tag</span><span class="w"> </span><span class="n">command</span><span class="w"> </span><span class="n">usr</span><span class="sr">/share/menu/</span><span class="n">pnmixer</span><span class="o">:</span><span class="mi">5</span>
</code></pre></div>
<p>Puzzled I was, and indeed, the messages are a little bit misleading.
I had to dig the net, and I found a few interesting discussions on that matter.
I'll provide the links here for the sake of history, and for people who like to
read IT stories :)</p>
<ul>
<li><a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806387">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806387</a>.</li>
<li><a href="https://lwn.net/Articles/597697/">https://lwn.net/Articles/597697/</a></li>
<li><a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741573">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741573</a></li>
</ul>
<p>The last link tells the end of the story. </p>
<h2>The other files</h2>
<p>All the other example files that are unused should be removed. It's best to use <code>lintian</code>
(explained below) for that, it will help you in that process.</p>
<p>In the end, here is what my archive looks like:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>debian
changelog
compat
control
copyright
docs
files
pnmixer
pnmixer.1
pnmixer.debhelper.log
pnmixer.manpages
pnmixer.substvars
rules
<span class="nb">source</span>
watch
</code></pre></div>
<h2>Building the package</h2>
<p>Now it's time to build the package. If you're lucky like me, the command
will succeed at the first try.</p>
<div class="highlight"><pre><span></span><code>dpkg-buildpackage -us -uc
</code></pre></div>
<p>Now have a look at the parent directory, this is where the package files are created:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-1<span class="w"> </span>..
pnmixer-0.6.1
pnmixer_0.6.1-1_amd64.changes
pnmixer_0.6.1-1_amd64.deb
pnmixer_0.6.1-1.debian.tar.xz
pnmixer_0.6.1-1.dsc
pnmixer_0.6.1.orig.tar.gz
pnmixer-0.6.1.tar.gz
pnmixer-dbgsym_0.6.1-1_amd64.deb
</code></pre></div>
<p>You can have a look at the content of your packages with the <code>debc</code> command.</p>
<div class="highlight"><pre><span></span><code>debc pnmixer_0.6.1-1_amd64.changes
</code></pre></div>
<h2>Installing</h2>
<p>You may want to try to install the package. The manual recommends to use the <code>debi</code> command:</p>
<div class="highlight"><pre><span></span><code>sudo debi pnmixer_0.6.1-1_amd64.changes
</code></pre></div>
<h2>Checking for errors</h2>
<p>References: <a href="https://www.debian.org/doc/manuals/maint-guide/dother.en.html">https://www.debian.org/doc/manuals/maint-guide/dother.en.html</a>.</p>
<p>To check a package for errors, we use <code>lintian</code>. The basic command:</p>
<div class="highlight"><pre><span></span><code><span class="nv">lintian</span><span class="w"> </span><span class="o">-</span><span class="nv">i</span><span class="w"> </span><span class="o">-</span><span class="nv">I</span><span class="w"> </span><span class="o">--</span><span class="k">show</span><span class="o">-</span><span class="nv">overrides</span><span class="w"> </span><span class="nv">pnmixer_0</span>.<span class="mi">6</span>.<span class="mi">1</span><span class="o">-</span><span class="mi">1</span><span class="nv">_amd64</span>.<span class="nv">changes</span>
</code></pre></div>
<p>You will probably have warnings and errors, nobody gets it right the first time.
Be patient and fix them one by one :)</p>
<h2>More tips and tricks</h2>
<h3>Adding a missing file to the skeleton</h3>
<p>Let's suppose that, when you created your package skeleton for the first time,
your forgot to specify the licence. How to add it afterward? It's really easy!</p>
<div class="highlight"><pre><span></span><code>rm debian/copyright
dh_make --addmissing --copyright gpl3 -f ../pnmixer-0.6.1.tar.gz
</code></pre></div>
<h3>Getting rid of the "useless dependency" warning</h3>
<p>Let's suppose you have this kind of warning messages when building the package:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dpkg</span><span class="o">-</span><span class="nv">shlibdeps</span>:<span class="w"> </span><span class="nv">warning</span>:<span class="w"> </span><span class="nv">package</span><span class="w"> </span><span class="nv">could</span><span class="w"> </span><span class="nv">avoid</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">useless</span><span class="w"> </span><span class="nv">dependency</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">debian</span><span class="o">/</span><span class="nv">pnmixer</span><span class="o">/</span><span class="nv">usr</span><span class="o">/</span><span class="nv">bin</span><span class="o">/</span><span class="nv">pnmixer</span><span class="w"> </span><span class="nv">was</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">linked</span><span class="w"> </span><span class="nv">against</span><span class="w"> </span>...
</code></pre></div>
<p>If you feel that you're not responsible for this (ie, you're not explicitely linking against an unused library),
you can get rid of that by adding this line to the <code>rules</code> file:</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">DEB_LDFLAGS_MAINT_APPEND</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">-</span><span class="n">Wl</span><span class="p">,</span><span class="o">--</span><span class="k">as</span><span class="o">-</span><span class="n">needed</span>
</code></pre></div>Wake-On-WAN, The Long Road Of The Magic Packet2015-12-11T00:00:00+00:002015-12-11T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-11:/2015/12/11/wake-on-wan-the-long-road-of-the-magic-packet/<p>How to setup wake-on-WAN, using a machine on the LAN to
broadcast the magic packet.</p>
<p>How to setup wake-on-WAN, using a machine on the LAN to
broadcast the magic packet.</p>
<h2>Introduction</h2>
<p>Maybe you already heard about <a href="https://en.wikipedia.org/wiki/Wake-on-LAN">Wake-on-LAN</a>
(or wol), which is the ability to wake-up a computer on the LAN
(<a href="https://en.wikipedia.org/wiki/Local_area_network">Local Area Network</a>)
by sending it a "magic" network message.
Wake-on-WAN (or wow) is roughly the same thing,
except that you don't send the message from inside the LAN,
but from the outside world: the WAN
(<a href="https://en.wikipedia.org/wiki/Wide_area_network">Wide Area Network</a>).</p>
<p>It sounds easy at first, a simple NAT rule should do, right? Actually, it's much more
complicated, due to the fact that we need to <em>broadcast</em> the wake-up message on the LAN.</p>
<p>If your router allows that, you just need to configure it, and you're done.
But if it doesn't, it becomes a complicated story, that I'm gonna relate
in this post.</p>
<h2>My personal use-case</h2>
<p>Just to provide some background: I have a QNAP NAS that I use mainly to
backup stuff. This NAS is at the other end of the world. I don't want to
have it powered on all the time, because it costs electricity. And I don't
want to have my backup online all the time, it's an useless risk.</p>
<p>Fortunately, the NAS comes with the wake-on-LAN feature, and I make a good
use of it.</p>
<p>So my NAS is turned off 99% of the time. When I need it, I wake it up
by sending it the "magic packet". And I can do that from whatever place of
the world I am, which is great. Then, I do my stuff, and send the NAS back
to sleep.</p>
<p>It sound very simple, but is quite a pain in the ass to achieve...</p>
<h2>Enabling wake-on-LAN on your device</h2>
<p>You need a device that supports the wake-on-LAN standard, and you need
to be sure that the feature is enabled. The procedure depends on your
device, of course. So here I'll just describe the steps involved for my
use-case. My NAS is a QNAP NAS TS-219P II, and it runs Debian.</p>
<p>To enable the wake-on-LAN feature, just a few commands:</p>
<div class="highlight"><pre><span></span><code><span class="n">ethtool</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="n">eth0</span><span class="w"> </span><span class="n">wol</span><span class="w"> </span><span class="n">g</span>
<span class="n">qcontrol</span><span class="w"> </span><span class="n">wol</span><span class="w"> </span><span class="n">on</span>
</code></pre></div>
<p>If you want to keep it enabled, you should run this commands at each startup.
A good place to put that is in the <code>/etc/network/interfaces</code> file, in the <code>post-up</code> rules.
It looks like that on my config:</p>
<div class="highlight"><pre><span></span><code><span class="n">allow</span><span class="o">-</span><span class="n">hotplug</span><span class="w"> </span><span class="n">eth0</span>
<span class="n">face</span><span class="w"> </span><span class="n">eth0</span><span class="w"> </span><span class="n">inet</span><span class="w"> </span><span class="n">dhcp</span>
<span class="w"> </span><span class="n">post</span><span class="o">-</span><span class="n">up</span><span class="w"> </span><span class="o">/</span><span class="n">sbin</span><span class="o">/</span><span class="n">ethtool</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="o">$</span><span class="n">IFACE</span><span class="w"> </span><span class="n">wol</span><span class="w"> </span><span class="n">g</span>
<span class="w"> </span><span class="n">post</span><span class="o">-</span><span class="n">up</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">sbin</span><span class="o">/</span><span class="n">qcontrol</span><span class="w"> </span><span class="n">wol</span><span class="w"> </span><span class="n">on</span>
</code></pre></div>
<p>If you want to read more, this is the definitive piece of information about Debian on QNAP devices:<br>
<a href="http://www.cyrius.com/debian/kirkwood/qnap/ts-119/">http://www.cyrius.com/debian/kirkwood/qnap/ts-119/</a></p>
<p>And more specifically, stuff about wake-on-LAN with Debian on QNAP device:<br>
<a href="http://michael.stapelberg.de/Artikel/qnap_ts119_wol/">http://michael.stapelberg.de/Artikel/qnap_ts119_wol/</a></p>
<h2>Testing wake-on-LAN</h2>
<p>On your workstation, you must install <code>wakeonlan</code>.</p>
<div class="highlight"><pre><span></span><code>apt-get install wakeonlan
</code></pre></div>
<p>To wake-up your device, just issue this simple command:</p>
<div class="highlight"><pre><span></span><code>wakeonlan de:ad:be:ef:ca:fe
</code></pre></div>
<p>We assume that:</p>
<ul>
<li>the device you intend to wake-up has been powered off previously</li>
<li>the device and your workstation are both on the same LAN</li>
<li>the device MAC address is <code>de:ad:be:ef:ca:fe</code></li>
</ul>
<p><code>wakeonlan</code> is a simple command that sends the "magic packet" as defined by
the standard. You can also tell it on which address you want to send the
packet, for example, assuming your sub-network is <code>192.168.1.0</code>:</p>
<div class="highlight"><pre><span></span><code>wakeonlan -i 192.168.1.255 de:ad:be:ef:ca:fe
</code></pre></div>
<p>Now, let me warn you about one thing. You MUST use a broadcast address.
You may think that you can use the IP address of the device (assuming it's
a static address). And indeed, if you just powered off your device and try
to wake it up using its IP address, it may work. But this is only due
to some cache effects, probably in the ARP cache somewhere but I'm not so
sure so I won't tell. If you wait more time after your device is powered off,
you will see that using the IP address of the device doesn't work at all.
So just broadcast.</p>
<h2>The power failure issue</h2>
<p>One of the first thing to understand is that the wake-on-LAN feature works only
if the device has been shut down <em>properly</em>.</p>
<p>Actually, what happens when you shutdown your device (and assuming you enabled the
wake-on-LAN feature) is that the Ethernet card will just go into a shallow sleep.
It's not completely dead, it's still powered on, and it still receives packet.
And when the "magic packet" arrives, the Ethernet card will turn on the whole device.
The device in itself was not <em>sleeping</em>, it was really <em>off</em>. Only the Ethernet
card was <em>sleeping</em>.</p>
<p>Now, what happens if you disconnect the power plug of your device while it's in this
state? Well, the Ethernet card is powered off for real this time, not sleeping anymore.
And now, if you connect the power plug again, what happens? Nothing. The whole device
is plugged to AC, but it's completely off, including the Ethernet card. You can send
the "magic packet" now, it's useless, the Ethernet card is off.</p>
<p>So what does that mean? It means that if there's a power cut, you're screwed. If
someone unplug the power cable, you're screwed. It means that your device, when powered
off in this "wake-on-LAN fashion", is in a weak position, and you won't be able to wake
it up 100% of the time. Except if you take great care that it's never powered off.</p>
<p>The easier solution for that: you can plug your device to an
<a href="https://en.wikipedia.org/wiki/Uninterruptible_power_supply">UPS</a>.</p>
<p>Second solution, your device may have an automatic power on feature. It means that,
if there's a power cut, the device turns on itself after power recovery. My NAS has
this feature, and that's what I use. It works great, but it involves more configuration
on your device...</p>
<h2>Wake-on-WAN, can your router do that?</h2>
<p>OK, now let's start the interesting part of this post. Waking a device from the WAN.</p>
<p>At first, you must know the public IP address of your router, the one that allows your
NAS to be online. If it's a static IP address, just write it down and you're OK.
If it's dynamic, you will have to setup a dynamic DNS account and enable that on your router.
Then you will use the dynamic DNS to reach your router.</p>
<p>With the <code>wakeonlan</code> command, you can send the magic packet out on the WAN, this is not an issue.
The issue lies on the receiving side. How to handle the incoming packet, and how
to broadcast it on the LAN?</p>
<p>Your router may have this feature built-in, and in this case it's easy, you can stop reading :).
However, my router has no such thing, so I had to keep on digging...</p>
<p>The first idea I had was simply to add a NAT rule to my router, that handles the incoming
packet and broadcasts it on the LAN. Easy, right? If you're used to network configuration,
you already had the same idea. Well, try it, why not? If it works, congratulations, you're done!</p>
<p>Unfortunately for me, my router doesn't allow that, for security reason they say,
but most likely because they didn't bother implementing it. And I don't have root access
on the router, so I'm a little bit stuck here.</p>
<p>However, I have a Raspberry Pi 2 (RPi2) on the LAN, running Debian.
This little RPi2 is supposed to be alive all the time, so I thought I could use it
as a middleman to broadcast the "magic packet".</p>
<h2>Wake-on-WAN using a middleman to broadcast the packet</h2>
<p>OK, so the first thing to do is to add a NAT rule to the router, to redirect
the incoming packet to the RPi2.</p>
<p>Then, I thought I was done. I thought I just had to setup a NAT rule
with <a href="https://en.wikipedia.org/wiki/Iptables">iptables</a> on my RPi2,
to broadcast the magic packet. Easy... Alas! I was terribly wrong!</p>
<p>It turns out that broadcasting a packet with <code>iptables</code> is not possible.
If you don't believe me, just try it and tell me if you succeed. I'm not
an expert here, so I may be wrong. But I tried hard, and stumbled on
this StackExchange post, which is worth reading:<br>
<a href="http://unix.stackexchange.com/q/77874/105794">http://unix.stackexchange.com/q/77874/105794</a></p>
<p>One of the guys answering there gives a pretty good explanation, and concludes
that "Broadcast packets are not routed/forwarded by design".</p>
<p>Thinking of it, I believe that's why most routers don't allow to broadcast an
incoming packet using NAT rules. Under the hood, most routers run Linux,
and install NAT rules with iptables. That's as simple as that.</p>
<p>Anyway! The same post provides also the solution, using the
<a href="http://www.dest-unreach.org/socat/">socat</a> command.
It's a little bit cryptic at first sight, but if you spend a little bit of time
reading the manual it becomes clearer.</p>
<p>My socat command looks like that:</p>
<div class="highlight"><pre><span></span><code>socat -u -T1 UDP-LISTEN:9,fork UDP-DATAGRAM:192.168.1.255:9,broadcast
</code></pre></div>
<p>I run that on my RPi2 and it works! At last!!</p>
<p>Of course, such command must be running all the time, so I just add it to my
<code>/etc/rc.local</code> file.</p>
<h2>The final picture</h2>
<p>Here is a summary of what happens when I wake up the NAS:</p>
<ul>
<li>
<p>On my laptop, I send the magic packet to my router on a special port.</p>
<p>wakeonlan -i my.router.dynamic.ip -p 56789 de:ad:be:ef:ca:fe</p>
</li>
<li>
<p>On my router, the packet is redirected to the RPi2 on the port 9.
Of course, this is not automatic, I took care to setup the proper NAT rule.</p>
</li>
<li>
<p>On my RPi2, when the packet arrives, it's caught by the socat command running
in the background, and broadcasted to the LAN.</p>
</li>
<li>
<p>On my NAS, the packet is caught by the half-sleepy ethernet card. The NAS wakes up.</p>
</li>
</ul>
<p>In the end, you can see that it was not straight-forward at all...</p>
<p>Furthermore, to handle the power cut problem described above, I had to enable the
autopoweron feature of the NAS, then write an autopoweroff script, and run it in the background.
The autopoweroff script turns off the NAS when there's nothing happening.
It's not so complicated to write, but it takes a little bit of care and time.</p>
<p>But this is a must-have, if you're in the same situation as me, you have to go through that.
Autopoweron without autopoweroff makes no sense.</p>
<h2>More readings</h2>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/Wake-on-LAN">https://wiki.archlinux.org/index.php/Wake-on-LAN</a></li>
<li><a href="http://www.finalclap.com/tuto/wake-on-lan-internet-wan-79/">http://www.finalclap.com/tuto/wake-on-lan-internet-wan-79/</a></li>
</ul>Backing Up Ghost Content2015-12-10T00:00:00+00:002015-12-10T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-10:/2015/12/10/backing-up-ghost-content/<p>My homemade script to backup my ghost blog, smartly named <code>ghost-backup</code>.</p>
<p>My homemade script to backup my ghost blog, smartly named <code>ghost-backup</code>.</p>
<h2>Introduction</h2>
<p>Once you start investing time in things, backing it up is a must-have.
Blogging is one of these things that take time, and if you host your
blog by yourself, it's up to you to take care of the backup.</p>
<h2>References</h2>
<p>Let's start with some good articles I read on the net here and there:</p>
<ul>
<li><a href="https://www.ghostforbeginners.com/backing-up-ghost-on-the-command-line/">https://www.ghostforbeginners.com/backing-up-ghost-on-the-command-line/</a></li>
<li><a href="http://ahmed.amayem.com/different-methods-of-backing-up-and-automating-backups-of-ghost-on-linux-centos-6/">http://ahmed.amayem.com/different-methods-of-backing-up-and-automating-backups-of-ghost-on-linux-centos-6/</a></li>
</ul>
<h2>Important points</h2>
<p>Your Ghost content lives in the <code>content</code> sub-directory. You can choose to backup:</p>
<ul>
<li>only the database at <code>content/data/ghost.db</code> (the text of your posts is here)</li>
<li>the whole <code>content</code> directory (also includes pictures, theme, etc)</li>
</ul>
<p>Ghost must be stopped before you copy the database. This is to ensure that there's
no transaction in progress in the database at the moment you read it.</p>
<h2>How I do it</h2>
<p>I choose the most complicated path, as usual. I want to backup the whole content
because pictures are important too. But my blog lives on a VPS where I don't have
much space on the disk, so I can't afford to have too many backups, takes too much
space.</p>
<p>So I started to think. First, I can backup everyday, but for most of the content,
it's useless since it's not changed everyday. If you're curious and compare two
successive backups, let's says day 1 and day 2, the only thing that changed is
the database. Even if you don't edit your blog, the database is modified by Ghost.</p>
<p>The database doesn't take a lot of space, but is often modified. The pictures take
a lot of space, but are not modified often. So we have two different problems here,
and I thought I should solve them separately.</p>
<p>So my solution is to backup the whole <code>content</code> directory everyday. But then I
take the database out of it. I keep database backups for a long period, like 30 days.
The rest of the backup, I compare it against the previous backup. If the content is the
same, I just discard it. And I keep only few backups like this, because they're big.</p>
<p>In the end, I feel safe :) I've got a long period of database backup. And for the
pictures, I'm not even sure I really need to back it up, but we never know, so I've
got a few backups just in case.</p>
<p>If you're interested, you can check out my script on Gitlab:<br>
<a href="https://gitlab.com/arnaudr/ghost-backup">https://gitlab.com/arnaudr/ghost-backup</a></p>Ghost Theming2015-12-09T00:00:00+00:002015-12-09T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-09:/2015/12/09/ghost-theming/<p>How to get and install themes for your <a href="https://ghost.org/">Ghost</a> blog.
Then comes an example of Github-based workflow to customize your theme.</p>
<p>How to get and install themes for your <a href="https://ghost.org/">Ghost</a> blog.
Then comes an example of Github-based workflow to customize your theme.</p>
<h2>Getting and installing a new theme</h2>
<p>This procedure is well described on a <a href="https://www.digitalocean.com/">Digital Ocean</a> post:<br>
<a href="https://www.digitalocean.com/community/tutorials/how-to-change-themes-and-adjust-settings-in-ghost">How To Change Themes and Adjust Settings in Ghost</a></p>
<p>In two words:</p>
<ul>
<li>Find your theme. A good place to go is the <a href="http://marketplace.ghost.org/themes/free/">Ghost Marketplace</a>.</li>
<li>
<p>Most of the time, themes are versioned on Github. So you can just clone
them in your Ghost themes directory.</p>
<p>cd /srv/www/ghost/content/themes
git clone https://github.com/Skepton/Pixeltraveller.git</p>
</li>
<li>
<p>Change ownership of the theme if needed.</p>
<p>chown -R ghost:ghost Pixeltraveller</p>
</li>
<li>
<p>Then restart Ghost (this command depends on the service manager you use).</p>
<p>systemctl restart ghost</p>
</li>
</ul>
<p>Now, just open your web browser, navigate yo your blog admin page.
In the <code>Settings > General</code> page, change the theme, and you're done.</p>
<h2>Customizing a theme - A Github workflow</h2>
<p>After giving it some thoughts, I thought I could just go with the default theme,
and customize it a little bit.</p>
<p>You can edit the theme directly in your Ghost directory. It's OK if you just change a
thing or two, but if you want to make more changes, or play a little bit with it,
I recommend to version control the theme with Git.</p>
<p>The default theme (Casper) provided with Ghost is available on Github at this address:<br>
<a href="https://github.com/TryGhost/Casper">https://github.com/TryGhost/Casper</a></p>
<p>If you have a Github account, the best thing to do is to fork the Casper theme, then clone
your fork inside your Ghost directory. Be sure to give the repository another name (here
<code>casper-git</code>) so that you can have both theme available, the default and the one you're
working on.</p>
<div class="highlight"><pre><span></span><code><span class="n">cd</span><span class="w"> </span><span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">ghost</span><span class="o">/</span><span class="n">content</span><span class="o">/</span><span class="n">themes</span>
<span class="n">git</span><span class="w"> </span><span class="n">clone</span><span class="w"> </span><span class="n">git</span><span class="nv">@github</span><span class="p">.</span><span class="nl">com</span><span class="p">:</span><span class="n">elboulangero</span><span class="o">/</span><span class="n">Casper</span><span class="p">.</span><span class="n">git</span><span class="w"> </span><span class="n">casper</span><span class="o">-</span><span class="n">git</span>
</code></pre></div>
<p>Of course, replace the URL above by the one of your fork.</p>
<h2>Working locally on your theme</h2>
<p>You can work directly on the VPS that hosts the Ghost blog, but soon
you will find that it's not practical.</p>
<ul>
<li>you probably don't want to test things on a blog that is published, while
people are looking at it, do you?</li>
<li>it's way faster to work locally than on a remote server (depends on your
connection of course, but mine sucks)</li>
<li>sometimes, it's good to have a GUI (I'm a big fan of <code>gitk</code>).</li>
</ul>
<p>So you need to setup a Ghost blog locally on your machine. It's not so hard,
just follow the same steps than you did to install Ghost on your server,
except that there's no need to install Ghost in <code>/var/www</code>, you can just
put it in your own directory. No need to change ownership too.</p>
<p>Then, just <code>git clone</code> your theme inside Ghost, as you did before.</p>
<p>Create a few posts, and start playing with the theme. Serve the blog
locally with <code>npm start</code>, look at your changes, make yourself at home.</p>
<p>Once you're happy with your changes, <code>git push</code>.</p>
<h2>Syncing the theme with your server</h2>
<p>There are several ways to do that.</p>
<p>I personnally do that manually. Both on my server and on my laptop, I have my Ghost
theme <em>git-cloned</em> inside the Ghost directory.</p>
<p>I work on the theme on my laptop, <code>git push</code> the changes when I'm done. Then I ssh to my
server, I <code>git pull</code> the theme, restart Ghost and have a look at it on my real blog.
Often, I may find little details to fix that I didn't see before, and at this moment I find
it more convenient to fix it directly on the server. So I do my last changes here,
then I can <code>git push</code> from my server.</p>
<p>This is my own way of doing it, it has the advantage that I can work from both my laptop
and my server, but has the disadvantage that I have two concurrent git users working
on it, so I can mess a little bit with git if I'm not careful.</p>
<p>But if you google around a little bit, you'll see that there's another common workflow
that involves a Git hook server-side. The idea is to hook Git so that it deploys your
changes automatically each time you push. The advantage is that it's all automated with Git.
But it can also be a disadvantage, maybe you don't want that. Furthermore, if you version
your theme with Github, you can't setup a hook server-side.</p>
<p>You could also synchronize the theme with a rsync command if you like it, that's just
another way of doing it...</p>Getting Started With Ghost2015-12-08T00:00:00+00:002015-12-08T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-08:/2015/12/08/getting-started-with-ghost/<p>How to install Ghost from upstream, setup Apache as a frontend for Ghost,
and create a systemd service to start Ghost automatically.</p>
<p>How to install Ghost from upstream, setup Apache as a frontend for Ghost,
and create a systemd service to start Ghost automatically.</p>
<h2>Introduction</h2>
<p><a href="https://ghost.org/">Ghost</a> is a blogging platform, a kind of
<a href="https://wordpress.org/">Wordpress</a> that would have been re-written
from scratch, and that strives to remain simple.</p>
<p>In this post we will install and configure Ghost.</p>
<p>As often when we want to use the latest version, we're going to bypass
the package manager, and install the stuff from upstream. It's quite
easy with Ghost, no reason to be afraid ;)</p>
<h2>Node.js installation</h2>
<p>Ghost is coded in <a href="https://nodejs.org/">Node.js</a>, it's the first
thing that we must install.</p>
<p>First, let's be sure that you have all the build tools needed.</p>
<div class="highlight"><pre><span></span><code>apt-get update
apt-get install build-essential python
</code></pre></div>
<p>Now, get the right version of Node.js. Have a look at the
<a href="http://support.ghost.org/installation/">Ghost installation page</a>
and check what's the recommanded version. At the moment it's the
<code>0.10.x</code> serie.</p>
<div class="highlight"><pre><span></span><code>VERSION=v0.10.41
wget http://nodejs.org/dist/$VERSION/node-$VERSION.tar.gz
tar -xvzf node-$VERSION.tar.gz
cd node-$VERSION
./configure
make
make install
</code></pre></div>
<p>Compilation takes some time...</p>
<h2>Ghost installation</h2>
<p>You will need to unzip, be sure to be ready...</p>
<div class="highlight"><pre><span></span><code>apt-get install unzip
</code></pre></div>
<p>Now just go to the <a href="https://ghost.org/download/">Ghost download page</a>
to see what is the latest version. Or you can also use the symlink
to the latest version: <a href="https://ghost.org/zip/ghost-latest.zip">https://ghost.org/zip/ghost-latest.zip</a>.
Unzip that where your websites usually go, it may be <code>/var/www</code>
or <code>/srv/www</code>.</p>
<div class="highlight"><pre><span></span><code>VERSION=0.7.1
wget https://ghost.org/zip/ghost-$VERSION.zip
unzip ghost-$VERSION.zip -d /srv/www/ghost-$VERSION
ln -s ghost-$VERSION /srv/www/ghost
</code></pre></div>
<p>I like to symlink my current Ghost blog, so that I can have
different Ghost versions side to side, and switch from one to
another easily. But it's just my own way of doing things.</p>
<h2>Basic Ghost configuration</h2>
<p>There is an example of configuration provided, so the best thing
to do is to copy it, and start from that point.</p>
<div class="highlight"><pre><span></span><code>cd /srv/www/ghost
cp config.example.js config.js
vi config.js
</code></pre></div>
<p>Actually, there's not much to change, you're already setup to get started.</p>
<p>You can see that there are two different configurations, one for
production and one for development. I only use the production, but
once again it's just my own use case, feel free to do better.</p>
<p>So, there's one thing you should change, it's the <code>url</code> value, which should
point to the URL of your blog.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1">// Configure your URL and mail settings here</span>
<span class="w"> </span><span class="n">production</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">url</span><span class="p">:</span><span class="w"> </span><span class="s">'http://your.blog.com'</span><span class="p">,</span>
</code></pre></div>
<p>Also, you should configure the mail settings right now,
that's what they advise in the official documentation.
I use <a href="https://www.mandrill.com/">Mandrill</a> to send my emails,
here is the configuration:</p>
<div class="highlight"><pre><span></span><code> mail: {
transport: 'SMTP',
host: 'smtp.mandrillapp.com',
options: {
service: 'Mandrill',
auth: {
user: '<your-mandrill-username>',
pass: '<your-mandrill-api-key>'
},
},
},
</code></pre></div>
<p>More details on this page: <a href="http://rdbrdd.com/configuring-ghost-to-use-mandrill/">Adding Mandrill to your ghost blog</a></p>
<h2>Testing</h2>
<p>You can already test your setup. There's just one little thing to do.
Go to the config file. See the <code>server</code> part?</p>
<div class="highlight"><pre><span></span><code> server: {
host: '127.0.0.1',
port: '2368'
}
</code></pre></div>
<p>The <code>host</code> setting tells Ghost to serve only the localhost. This is because Ghost
is not supposed to serve the outside world directly. There should be another
web server between Ghost and the outside wild world.</p>
<p>So, if you run Ghost on your machine, this is OK, you can test without having
anything to change. But if you run it on a remote server, just for this quick
test, you must change the <code>host</code> parameter, and set it to <code>0.0.0.0</code>. Just be
sure to set if back to <code>127.0.0.1</code> afterward, for security reasons.</p>
<p>Now, just run Ghost for the first time! I do that in production mode directly.
Be sure to run this commands in the Ghost directory, since <code>npm</code>
(the <a href="https://www.npmjs.com">Node Package Manager</a>) installs plenty of stuff locally
inside the Ghost directory.</p>
<div class="highlight"><pre><span></span><code>cd /srv/www/ghost
npm install --production
npm start --production
</code></pre></div>
<p>That's it! Open your web browser, connect on the port <code>2368</code>, you should see
your blog. That was easy, wasn't it?</p>
<h2>Apache setup for Ghost</h2>
<p>As we said, Ghost should run behind a web server, which acts as a frontend.</p>
<p>I personnally use <a href="https://httpd.apache.org/">Apache</a>, so I describe quickly
the Apache setup. If you want to use <a href="nginx.org">Nginx</a>, you'll be fine, there's
plenty of doc available on the web.</p>
<p>At first, be sure to have Apache installed.</p>
<div class="highlight"><pre><span></span><code>apt-get install apache2
</code></pre></div>
<p>Then, enable the HTTP proxy mode.</p>
<div class="highlight"><pre><span></span><code>a2enmod proxy_http
</code></pre></div>
<p>Create the Ghost site configuration. As you can see, all the magic comes
from the HTTP Proxy config.</p>
<div class="highlight"><pre><span></span><code>echo<span class="w"> </span>'
<span class="nt"><VirtualHost</span> <span class="err">*:80</span><span class="nt">></span>
<span class="w"> </span>ServerAdmin<span class="w"> </span>your@mail.com
<span class="w"> </span>ServerName<span class="w"> </span>your.blog.com
<span class="w"> </span>#ServerAlias<span class="w"> </span>your.blog.alias.com
<span class="w"> </span>ErrorLog<span class="w"> </span><span class="cp">${</span><span class="n">APACHE_LOG_DIR</span><span class="cp">}</span>/ghost_vhosts_error.log
<span class="w"> </span><span class="nt"><Location</span> <span class="nt">/></span>
<span class="w"> </span>ProxyPass<span class="w"> </span>http://localhost:2368/
<span class="w"> </span>ProxyPassReverse<span class="w"> </span>http://localhost:2368/
<span class="w"> </span><span class="nt"></Location></span>
<span class="nt"></VirtualHost></span>
'<span class="w"> </span>><span class="w"> </span>/etc/apache2/sites-available/ghost.conf
</code></pre></div>
<p>At last, enable the site:</p>
<div class="highlight"><pre><span></span><code>a2ensite ghost
</code></pre></div>
<p>Now, you can just restart Apache. I personnaly use Debian, and therefore it's
up to systemd to do the job.</p>
<div class="highlight"><pre><span></span><code>systemctl restart apache2
</code></pre></div>
<p>OK, so now, be sure that you have a node instance running
(remember the <code>npm start --production</code> command?).
You should be able to see your blog in a web browser.</p>
<p>You should understand that, when you request your blog, it's Apache who
gets the request first. Thanks to the <code>ServerName</code> setting
(and possibly <code>ServerAlias</code> if you have one), it redirects the request
to <code>localhost:2368</code>, where there's a node instance listening. This node
instance is the one that serves your blog for real.</p>
<h2>Getting Node to run by itself</h2>
<p>Now, we're almost there. For the moment, you run the node instance manually
through the command <code>npm start</code>. That's OK for debug, but now we need it to
be running all the time, from the moment your server starts until it stops.</p>
<p>There are several ways to do that. I personnaly use
<a href="http://www.freedesktop.org/wiki/Software/systemd/">systemd</a>.</p>
<p>The first recommandation is to add a user that will run the node instance.
This is a good habit, to avoid running it as root. So, just create a new
user (we name it <code>ghost</code>, just to be simple), and let him own the Ghost directory.</p>
<div class="highlight"><pre><span></span><code>adduser --shell /bin/bash --gecos 'Ghost application' ghost
chown -H -R ghost:ghost /srv/www/ghost
</code></pre></div>
<p>Now, here is the systemd unit file that is used to start/stop Ghost.
Notice how we ask to run as the user/group ghost. Notice also that
we set an environment variable. I think it's needed, but I'm not
100% sure though...</p>
<div class="highlight"><pre><span></span><code><span class="n">echo</span><span class="w"> </span><span class="s1">'</span>
<span class="s1">[Service]</span>
<span class="s1">ExecStart=/usr/local/bin/node /srv/www/ghost/index.js</span>
<span class="s1">WorkingDirectory=/srv/www/ghost</span>
<span class="s1">Restart=always</span>
<span class="s1">StandardOutput=syslog</span>
<span class="s1">StandardError=syslog</span>
<span class="s1">SyslogIdentifier=ghost</span>
<span class="s1">User=ghost</span>
<span class="s1">Group=ghost</span>
<span class="s1">Environment=NODE_ENV=production</span>
<span class="s1">[Install]</span>
<span class="s1">WantedBy=multi-user.target</span>
<span class="s1">'</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">systemd</span><span class="o">/</span><span class="k">system</span><span class="o">/</span><span class="n">ghost</span><span class="p">.</span><span class="n">service</span>
</code></pre></div>
<p>Here you go, just start the service!</p>
<div class="highlight"><pre><span></span><code>systemctl enable ghost
systemctl start ghost
</code></pre></div>
<h2>Start blogging</h2>
<p>To start blogging, just open your web browser, visit the URL of your blog.
Append <code>/ghost</code> to the URL, this will lead you to the admin page.
Just create your account as you're asked. If you did setup your mail stuff
properly, you will receive an email once you finish creating your account.</p>
<p>And that's all about it, now your can start writing posts.</p>
<p>Ghost uses the <a href="https://daringfireball.net/projects/markdown/syntax">Markdown</a> syntax,
so if you don't know about that yet, maybe it's your next step :)</p>
<h2>References</h2>
<p>Here are some links that made me write this post:</p>
<ul>
<li><a href="http://bloggle.me/installation-de-ghost-et-apache-sur-un-vps-debian">Installer Ghost + Apache sur un VPS Debian 7 (Wheezy)</a></li>
<li><a href="http://seanvbaker.com/a-ghost-workflow/">A Ghost Workflow</a></li>
<li><a href="http://rdbrdd.com/configuring-ghost-to-use-mandrill/">Adding Mandrill to your ghost blog</a></li>
</ul>Secure Web Access Through SSH Tunnel (SOCKS Proxy Server)2015-12-07T00:00:00+00:002015-12-07T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-07:/2015/12/07/secure-web-access-through-ssh-tunnel-socks-proxy-server/<p>How to setup a dynamic SSH tunnel (also called SOCKS proxy server)
and use it daily with Firefox and Thunderbird.</p>
<p>How to setup a dynamic SSH tunnel (also called SOCKS proxy server)
and use it daily with Firefox and Thunderbird.</p>
<h2>Introduction</h2>
<p>I'm a kind of "digital nomad" (yeah I just like the expression :)).
I use to go around with my laptop on my back, and I connect to the net from
various places, like coffees, hotels, swimming-pools (yes :)),
or whatever places that provide wifi.</p>
<p>I don't trust these access-points so much, and I've been wondering
what kind of trick could be done to improve the security. It turns
out that it's quite easy to go with
<a href="https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling">SSH tunnels</a>,
and that there's no need of <a href="https://en.wikipedia.org/wiki/OpenVPN">OpenVPN</a>.</p>
<p>Notice that I have nothing against OpenVPN, but it just happens that
I have a <a href="https://en.wikipedia.org/wiki/Virtual_private_server">VPS</a>
out there, and of course I already have the
<a href="https://en.wikipedia.org/wiki/OpenSSH">OpenSSH</a> daemon up and running
on this VPS. I'm not a skilled sysadmin, so I don't want to install
too much stuff and add plenty of doors to my server.
The less doors the better.</p>
<p>SSH is already there, and it's fit for the job: SSH server can act
as a <a href="https://en.wikipedia.org/wiki/SOCKS">SOCKS</a> proxy server,
and it's so simple you won't believe it...</p>
<p>One last word, this post provides specific solutions for
<a href="https://www.mozilla.org/firefox">Firefox</a> and
<a href="https://www.mozilla.org/thunderbird">Thunderbird</a>,
as they're my favorite apps to surf the web.</p>
<h2>SSH tunneling</h2>
<p>Creating a tunnel to your remote server is achieved with only one line:</p>
<div class="highlight"><pre><span></span><code><span class="n">ssh</span><span class="w"> </span><span class="o">-</span><span class="n">N</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="mi">5222</span><span class="w"> </span><span class="k">user</span><span class="nv">@vps</span>
</code></pre></div>
<p>Let's quote the SSH manual page:</p>
<ul>
<li><code>-N</code> <em>Do not execute a remote command. This is useful for just forwarding ports.</em></li>
<li><code>-D port</code> <em>Specifies a local "dynamic" application-level port forwarding. [...]
ssh will act as a SOCKS server.</em></li>
<li>We use the port 5222 in this example, but you can use whatever port you want.</li>
</ul>
<p>This simple command opens a tunnel to <code>vps</code>, aka the host where a friendly SSH server
is waiting for you to connect.</p>
<p>If you want to know more, I recommend this Linux Journal article:<br>
<a href="http://www.linuxjournal.com/content/ssh-tunneling-poor-techies-vpn">SSH Tunneling - Poor Techie's VPN</a></p>
<h2>Testing with Firefox</h2>
<p>In order to test that, you must configure your web browser to use a proxy.
Here is the procedure with Firefox:</p>
<ul>
<li>Go to Preferences > Advanced > Network > Settings</li>
<li>Choose <code>Manual proxy configuration</code></li>
<li>Fill the field <code>SOCKS Host</code> with <code>localhost</code></li>
<li>Set the right port (<code>5222</code> in this example)</li>
<li>Check <code>SOCKS v5</code></li>
<li>Click OK</li>
</ul>
<p>And that's all. Now Firefox sends its traffic to <code>localhost:5222</code>, which is the
entrance of the SSH encrypted tunnel. Data goes through the tunnel and ends
up on your VPS, where the SSH daemon forwards it to the web...</p>
<h2>Daily usage with Firefox and Thunderbird</h2>
<p>Now that things work, what I want is a convenient way to use it. That is,
I don't need to go through a proxy when I'm home, but as soon as I'm out,
I'd like to enable the proxy easily. A one-line shell command would be sweet.</p>
<p>Basically, there are only two applications that should use the proxy:
Firefox for web browsing, and Thunderbird for emails.</p>
<p>Of course, I can change the settings of both apps each time I'm out,
and change it back when I'm home. But, how tedious is that!</p>
<p>Furthermore, if I do that manually, I must be careful to do things in the right order:</p>
<ol>
<li>Launch the apps, enable the proxy</li>
<li>Connect to the WLAN (Wireless LAN)</li>
<li>Open the SSH tunnel</li>
<li>Start doing stuff</li>
</ol>
<p>The key is to enable the proxy <em>before</em> connecting to the WLAN!
If I do it the other order, when I launch Thunderbird, the first thing
it does is to send my password out on the LAN before I have time to
change the proxy settings. Enabling the proxy after that is quite useless,
passwords have been leaked already...</p>
<p>Doing things manually is always error prone, but you already knows that if you're
an IT guy like me.</p>
<p>So it's better to automate things. Somehow, I would like to declare my proxy
settings system wide, so that applications can automatically know it,
and use the proxy to go on the WAN.</p>
<h2>Attempt 1: pure environment variables [FAILURE]</h2>
<p>Ok, so after browsing the web, I found what seems to be the perfect solution:
declaring the proxy settings in environment variables, and letting applications
handle that by themselves. For example, Firefox can be configured to
<code>Use system proxy settings</code>, as they say.</p>
<p>So the idea is to declare the proxy in the environment, but what variables should we use?
It seems that there is no consensus about that. If you have a look at the
<a href="https://wiki.archlinux.org/index.php/Proxy_settings#Environment_variables">ArchLinux wiki</a>,
you can see that the guys define plenty of variables, just to be sure.
Then you pray that your favorite application picks up one of these and honor it...</p>
<p>So here we go:</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">all_proxy</span><span class="o">=</span><span class="n">socks</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">5222</span>
<span class="k">export</span><span class="w"> </span><span class="n">http_proxy</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">https_proxy</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">ftp_proxy</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">rsync_proxy</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">ALL_PROXY</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">HTTP_PROXY</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">HTTPS_PROXY</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">FTP_PROXY</span><span class="o">=$</span><span class="n">all_proxy</span>
<span class="k">export</span><span class="w"> </span><span class="n">RSYNC_PROXY</span><span class="o">=$</span><span class="n">all_proxy</span>
</code></pre></div>
<p>Then, you launch Firefox from the same shell (since environment variables
only live in the shell where they're defined, and its sub-processes),
and Firefox should pick up these values and use it.</p>
<p>But somehow, it just doesn't work, Firefox ignores all the stuff we defined in the environment.
If you like to read, there's a good discussion about that on the Ubuntu forums:<br>
<a href="http://askubuntu.com/questions/23117/configuring-firefox-to-use-a-proxy-from-the-command-line">Configuring Firefox to use a proxy from the command line</a></p>
<p>So long for the environment variables...</p>
<h2>Attempt 2: environment variables & add-ons [SUCCESS]</h2>
<p>So I kept on googling around, and it just happens that someone wrote an
add-on for Firefox to fix that. It's named "Environment Proxy", and there's
a version for
<a href="https://addons.mozilla.org/en-US/firefox/addon/environment-proxy/">Firefox</a>
and for
<a href="https://addons.mozilla.org/en-US/thunderbird/addon/environment-proxy/">Thunderbird</a>.</p>
<p>I'm not a big fan of the add-on solution. Simply because the add-on can
break at any Firefox/Thunderbird update. If you update your system steadily
like me, you probably noticed that there's a new Firefox version every two months.
So the add-on may let you down on a regular basis, and it's a little bit scary.
Furthermore, if the add-on is not maintained thoroughly by its author,
you're screwed, and you find yourself with the same problem to solve all
over again...</p>
<p>Well, that's just my point of view on it. Anyway, I made an exception for this time,
I tried this add-on, it works!</p>
<p>Have a look at the documentation page, it says which environment variable you should define.
For SSH tunnel and SOCKS proxy, just one variable is needed:</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">SOCKS_SERVER</span><span class="o">=</span><span class="n">localhost</span><span class="p">:</span><span class="mi">5222</span>
</code></pre></div>
<p>But beware! Exporting a variable is nice, but don't forget it lives only in the shell and its
sub-processes. So now you must start Firefox from this shell, otherwise you won't use the proxy.</p>
<p>OK, so because I'd like to avoid mistakes, and don't want to type too many commands,
I wrote a script! Here it is:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#</span>
<span class="c1"># Ssh command, thanks to:</span>
<span class="c1"># http://unix.stackexchange.com/q/29912/105794 </span>
<span class="nv">USER</span><span class="o">=</span>root
<span class="nv">HOST</span><span class="o">=</span>vps
<span class="nv">PORT</span><span class="o">=</span><span class="m">5222</span>
<span class="nv">SOCKET</span><span class="o">=</span>/tmp/ssh_tunnel_<span class="si">${</span><span class="nv">HOST</span><span class="si">}</span>.sock
pgrep<span class="w"> </span>iceweasel<span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"Iceweasel already running, please stop it..."</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
pgrep<span class="w"> </span>icedove<span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"Icedove already running, please stop it..."</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="o">[[</span><span class="w"> </span>-e<span class="w"> </span><span class="nv">$SOCKET</span><span class="w"> </span><span class="o">]]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"SSH tunnel already opened, please close it..."</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Opening SSH tunnel, please wait..."</span>
ssh<span class="w"> </span>-f<span class="w"> </span>-N<span class="w"> </span>-D<span class="w"> </span><span class="nv">$PORT</span><span class="w"> </span>-M<span class="w"> </span>-S<span class="w"> </span><span class="nv">$SOCKET</span><span class="w"> </span>-o<span class="w"> </span><span class="nv">ExitOnForwardFailure</span><span class="o">=</span>yes<span class="w"> </span><span class="nv">$USER</span>@<span class="nv">$HOST</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="s2">"Failed to open SSH tunnel"</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Exporting environment variables..."</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">SOCKS_SERVER</span><span class="o">=</span>localhost:<span class="nv">$PORT</span>
<span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="s2">"Starting iceweasel... "</span>
<span class="o">(</span>iceweasel<span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="o">)</span><span class="w"> </span><span class="p">&</span>
<span class="nv">weasel_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span><span class="w"> </span><span class="nv">$weasel_pid</span>
<span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="s2">"Starting icedove... "</span>
<span class="o">(</span>icedove<span class="w"> </span>>/dev/null<span class="w"> </span><span class="m">2</span>><span class="p">&</span><span class="m">1</span><span class="o">)</span><span class="w"> </span><span class="p">&</span>
<span class="nv">dove_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span><span class="w"> </span><span class="nv">$dove_pid</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"To end the session, just close iceweasel & icedove"</span>
<span class="nb">wait</span><span class="w"> </span><span class="nv">$weasel_pid</span><span class="w"> </span><span class="nv">$dove_pid</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Unexporting environment variables..."</span>
<span class="nb">unset</span><span class="w"> </span>SOCKS_SERVER
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Closing tunnel..."</span>
ssh<span class="w"> </span>-S<span class="w"> </span><span class="nv">$SOCKET</span><span class="w"> </span>-O<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="nv">$USER</span>@<span class="nv">$HOST</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"Done"</span>
</code></pre></div>
<p>Notice that I use <code>iceweasel</code> and <code>icedove</code> instead of <code>firefox</code> and <code>thunderbird</code>.
This is the same thing, just the name is different on Debian.</p>
<h2>Another solution: proxychains [SUCCESS]</h2>
<p>If you don't like the add-on solution, or if you don't use Firefox and Thunderbird,
and prefer a more generic solution, here it is.</p>
<p>I found that there are some pieces of software called <em>proxifiers</em>.
They work by intercepting and modifying the network requests sent by a given
program. I can mention two of them: <a href="http://tsocks.sourceforge.net/">tsocks</a>
and <a href="http://proxychains.sourceforge.net/">proxychains</a>.
I think both are fit for the job.</p>
<p>I tried proxychains.</p>
<p>Pretty simple to install:</p>
<div class="highlight"><pre><span></span><code>apt-get install proxychains
</code></pre></div>
<p>Pretty simple to configure:</p>
<div class="highlight"><pre><span></span><code>vi /etc/proxychains.conf
</code></pre></div>
<p>Just comment out the last line, and add your own proxy config:</p>
<div class="highlight"><pre><span></span><code>socks5 127.0.0.1 5222
</code></pre></div>
<p>After that, just use proxychains to start your apps:</p>
<div class="highlight"><pre><span></span><code>proxychains firefox
</code></pre></div>
<h2>Conclusion</h2>
<p>So as you can see, it was a long way for a simple thing, as often it is
when you go through the command-line and want to understand a little bit
how things work.</p>
<p>Currently I go with the Firefox/Thunderbird add-ons, it works great. However
I need to go through the command-line to use my proxy, I can't launch
my applications from the menu icon. So it's not as transparent as I hoped in
the beginning,</p>
<p>Using OpenVPN would have make it totally transparent I think...</p>Bash Logging Helpers2015-12-06T00:00:00+00:002015-12-06T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-06:/2015/12/06/bash-logging-helpers/<p>Some smart bash lines that provide powerful logging for your scripts,
especially useful when run by Cron.</p>
<p>Some smart bash lines that provide powerful logging for your scripts,
especially useful when run by Cron.</p>
<h2>Introduction</h2>
<p>I wrote plenty of bash scripts lately.
And when it comes to logging info and errors, it's always the same.
I start simple, with just <code>echo</code> for info and <code>echo >&2</code> for errors.
I always hope I can get away with it, but it's not that simple.</p>
<p>It starts to be tricky when the script is launched automatically by
<a href="https://en.wikipedia.org/wiki/Cron">Cron</a>.
How to get the output of the script in the system logs?</p>
<h2>Piping the Script Output</h2>
<p>The first solution, and the easiest, is to pipe the script output to <code>logger</code>,
directly in the cronjob definition. So that the job definition looks like that:</p>
<div class="highlight"><pre><span></span><code><span class="nv">@daily</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="k">local</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">myscript</span><span class="w"> </span><span class="mi">2</span><span class="o">>&</span><span class="mi">1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">logger</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="n">myscript</span>
</code></pre></div>
<p>This solution is damn simple, but it has a slight disadvantage: the error output
is mixed with the standard output, therefore you loose the notion of error in
the logs. You error messages will appear as standard messages, with nothing warning
you that it's an error.</p>
<h2>Defining Log Functions in the Script</h2>
<p>This is probably a bit overkill for small and simple scripts. But if you think
you can afford a few extra lines to have a proper logging system in your script,
here's how I do it.</p>
<div class="highlight"><pre><span></span><code>tty<span class="w"> </span>-s<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="k">function</span><span class="w"> </span>log<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
tty<span class="w"> </span>-s<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="k">function</span><span class="w"> </span>log_err<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
tty<span class="w"> </span>-s<span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="k">function</span><span class="w"> </span>log<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span>logger<span class="w"> </span>-t<span class="w"> </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
tty<span class="w"> </span>-s<span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="k">function</span><span class="w"> </span>log_err<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span>logger<span class="w"> </span>-t<span class="w"> </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="w"> </span>-p<span class="w"> </span>user.err<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
</code></pre></div>
<p>I put these lines at the beginning of my script, or in a kind of bash library that is
included by my other scripts. Then, I use these <code>log*</code> functions to log whatever I want.</p>
<p>Now let's explain that a little bit.</p>
<ul>
<li><code>tty -s</code> let you know if the script is run in a terminal or not. Or, to be more accurate,
if the standard input (<em>stdin</em>) is connected to a terminal or not. If the script
is run by hand, the answer is yes. If it's run by Cron, the answer is no.</li>
<li>When run in a terminal, the logs are issued with <code>echo</code>, they appear in the console
only. It means that when I work on the script, I won't pollute the logs.</li>
<li>When run by Cron, logs are issued with <code>logger</code>, they go to the system log.</li>
</ul>
<p>So you might notice that we do something slightly unusual here: we define functions
conditionally. It's a little bit like macros in C.</p>
<p>If you feel uncomfortable with that, here is another way to achieve the same thing:</p>
<div class="highlight"><pre><span></span><code>log<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="o">[[</span><span class="w"> </span>-t<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">]]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>logger<span class="w"> </span>-t<span class="w"> </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
log_err<span class="o">()</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="o">[[</span><span class="w"> </span>-t<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">]]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>logger<span class="w"> </span>-t<span class="w"> </span><span class="k">$(</span>basename<span class="w"> </span><span class="nv">$0</span><span class="k">)</span><span class="w"> </span>-p<span class="w"> </span>user.err<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="o">}</span>
</code></pre></div>
<p>Explanations again:</p>
<ul>
<li><code>-t</code> is a bash builtin test that checks if a file descriptor is opened and refers to
a terminal (<code>man bash</code> for more info).</li>
<li>for normal logs that are supposed to go to <em>stdout</em> (file descriptor number 1), we just
check if stdout is connected to a terminal (<code>[[ -t 1 ]]</code>). If yes, we can use <code>echo</code>.
Otherwise, we log to the system logs with <code>loggers</code>.</li>
<li>for error logs, we check if <em>stderr</em> is connected to a terminal (<code>[[ -t 2 ]]</code>).</li>
</ul>
<p>OK, so in the end it just boils down to 2 smart lines. It's not much, is it?</p>
<p>Compared to simply piping the output of your script in the cron task, there are several advantages:</p>
<ul>
<li>Errors are logged as such, so they appear properly in the system logs (highlighted,
or in bold, or in red or whatever).</li>
<li>You may define more log functions, one for each log level (debug, warning and so on),
the same way you use to do with C or Python, or whatever language you're used to. And
as your script grows bigger, it's really handy.</li>
</ul>
<h2>Credits</h2>
<p>Piping the output of the script is not my idea, I got that from a ServerFault discussion:<br>
<a href="http://serverfault.com/q/137468/323199">Better logging for cronjobs? Send cron output to syslog?</a></p>
<p>There's a good thread that taught me how to define functions conditionnally,
but it was burried deep within the net:<br>
<a href="http://lists.squeakfoundation.org/pipermail/vm-dev/2014-June/015945.html">[Vm-dev] conditionally define a function in sh/bash?</a></p>
<p>There's a stackoverflow discussion that taught me how to check for the terminal:<br>
<a href="http://stackoverflow.com/q/3214935/776208">Can a bash script tell if it's being run via cron?</a></p>
<p>I also read a lot of bullshit about testing the <code>$-</code> bash variable.
<code>$-</code> can tell if you're in an interactive shell or not.
So if you're wondering exactly what is an interactive shell, please read:<br>
<a href="http://zsh.sourceforge.net/Guide/zshguide02.html#l7">http://zsh.sourceforge.net/Guide/zshguide02.html#l7</a></p>
<p>Just to quote the most interesting part:</p>
<blockquote>
<p>when you are typing at a prompt and waiting for each command to run, the shell is interactive;
in the other case, when the shell is reading commands from a file, it is, consequently, non-interactive.</p>
</blockquote>
<p>Checking <code>$-</code> doesn't distinguish between a script run by hand and a script run by Cron: both are non-interactive.
So it doesn't help at all, but I read plenty of discussions where people give it as a solution. Shame on them!</p>Setting Up A Wi-Fi Software Access-Point2015-12-05T00:00:00+00:002015-12-05T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-05:/2015/12/05/setting-up-a-wi-fi-software-access-point/<p>How to setup a wifi software access-point, using wpasupplicant and hostapd.</p>
<p>How to setup a wifi software access-point, using wpasupplicant and hostapd.</p>
<p>We create two virtual interfaces on top of one physical device.
One will act as a client, the other as an access-point.</p>
<p>Most of the credit for this tutorial goes to the
<a href="https://wiki.archlinux.org/index.php/Software_access_point">Archlinux Software Access Point Wiki</a>.</p>
<h2>Introduction</h2>
<p>We use an atheros-powered wifi dongle, but you can use any wifi chip,
as long as it can handle virtual interfaces. There's a list of hardware
with their capabilities on <a href="https://wireless.wiki.kernel.org">linux-wireless</a>.</p>
<h2>Installation</h2>
<p>At first, we have to check if the firmware for our wifi chip is installed.
You may not know that, but most (maybe every) wifi chip needs a firmware to operate,
and this firmware is loaded dynamically when the chip is powered.
Hence the firmware files must be installed. Plenty of firmwares are not free,
and they're not included by default in Debian.</p>
<p>Type <code>dmesg -w</code> in your console, then plug the wifi dongle.</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">dmesg</span><span class="w"> </span><span class="o">-</span><span class="n">w</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">new</span><span class="w"> </span><span class="n">high</span><span class="o">-</span><span class="n">speed</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">number</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">dwc_otg</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">New</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">found</span><span class="p">,</span><span class="w"> </span><span class="n">idVendor</span><span class="o">=</span><span class="mi">0846</span><span class="p">,</span><span class="w"> </span><span class="n">idProduct</span><span class="o">=</span><span class="mi">9030</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">New</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">strings</span><span class="p">:</span><span class="w"> </span><span class="n">Mfr</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span><span class="w"> </span><span class="n">Product</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span><span class="w"> </span><span class="n">SerialNumber</span><span class="o">=</span><span class="mi">48</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">Product</span><span class="p">:</span><span class="w"> </span><span class="n">WNA1100</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">Manufacturer</span><span class="p">:</span><span class="w"> </span><span class="n">NETGEAR</span><span class="w"> </span><span class="n">WNA</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">SerialNumber</span><span class="p">:</span><span class="w"> </span><span class="mi">12345</span>
<span class="n">cfg80211</span><span class="p">:</span><span class="w"> </span><span class="n">Calling</span><span class="w"> </span><span class="n">CRDA</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">update</span><span class="w"> </span><span class="n">world</span><span class="w"> </span><span class="n">regulatory</span><span class="w"> </span><span class="n">domain</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">Firmware</span><span class="w"> </span><span class="n">htc_9271</span><span class="o">.</span><span class="n">fw</span><span class="w"> </span><span class="n">requested</span>
<span class="n">usbcore</span><span class="p">:</span><span class="w"> </span><span class="n">registered</span><span class="w"> </span><span class="n">new</span><span class="w"> </span><span class="n">interface</span><span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="n">ath9k_htc</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">firmware</span><span class="p">:</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="n">htc_9271</span><span class="o">.</span><span class="n">fw</span><span class="w"> </span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">layer</span><span class="w"> </span><span class="n">deinitialized</span>
</code></pre></div>
<p>See the line <code>failed to load htc_9271.fw</code>?
The firmware requested is not installed.
Let's check what firmwares are installed.
On a fresh Debian install, there should be only the free firmwares.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>dpkg<span class="w"> </span>-l<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>firmware
ii<span class="w"> </span>firmware-linux-free<span class="w"> </span><span class="m">3</span>.3<span class="w"> </span>...
</code></pre></div>
<p>Ok, so we can probably find our firmware in the non-free Debian repository.
We must enable it, then find the package containing our firmware files.
The name of the package is common sense, in our case it's <code>firmware-atheros</code>. Let's go.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vi<span class="w"> </span>/etc/apt/sources.list
<span class="c1"># Add non-free to the main debian repository, at the end of the line</span>
$<span class="w"> </span>apt-get<span class="w"> </span>update
$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>firmware-atheros
</code></pre></div>
<p>Now, plug the wifi dongle again. This time, the output of <code>dmesg</code> is different:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">dmesg</span><span class="w"> </span><span class="o">-</span><span class="n">w</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">Firmware</span><span class="w"> </span><span class="n">htc_9271</span><span class="o">.</span><span class="n">fw</span><span class="w"> </span><span class="n">requested</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">firmware</span><span class="p">:</span><span class="w"> </span><span class="n">direct</span><span class="o">-</span><span class="n">loading</span><span class="w"> </span><span class="n">firmware</span><span class="w"> </span><span class="n">htc_9271</span><span class="o">.</span><span class="n">fw</span>
<span class="n">usb</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">Transferred</span><span class="w"> </span><span class="n">FW</span><span class="p">:</span><span class="w"> </span><span class="n">htc_9271</span><span class="o">.</span><span class="n">fw</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="p">:</span><span class="w"> </span><span class="mi">51272</span>
<span class="n">ath9k_htc</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="mf">1.0</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">HTC</span><span class="w"> </span><span class="n">initialized</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="mi">33</span><span class="w"> </span><span class="n">credits</span>
<span class="n">ath9k_htc</span><span class="w"> </span><span class="mi">1</span><span class="o">-</span><span class="mf">1.4</span><span class="p">:</span><span class="mf">1.0</span><span class="p">:</span><span class="w"> </span><span class="n">ath9k_htc</span><span class="p">:</span><span class="w"> </span><span class="n">FW</span><span class="w"> </span><span class="n">Version</span><span class="p">:</span><span class="w"> </span><span class="mf">1.3</span>
<span class="n">ath</span><span class="p">:</span><span class="w"> </span><span class="n">EEPROM</span><span class="w"> </span><span class="n">regdomain</span><span class="p">:</span><span class="w"> </span><span class="mh">0x60</span>
<span class="n">ath</span><span class="p">:</span><span class="w"> </span><span class="n">EEPROM</span><span class="w"> </span><span class="n">indicates</span><span class="w"> </span><span class="n">we</span><span class="w"> </span><span class="n">should</span><span class="w"> </span><span class="n">expect</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">direct</span><span class="w"> </span><span class="n">regpair</span><span class="w"> </span><span class="n">map</span>
<span class="n">ath</span><span class="p">:</span><span class="w"> </span><span class="n">Country</span><span class="w"> </span><span class="n">alpha2</span><span class="w"> </span><span class="n">being</span><span class="w"> </span><span class="n">used</span><span class="p">:</span><span class="w"> </span><span class="mi">00</span>
<span class="n">ath</span><span class="p">:</span><span class="w"> </span><span class="n">Regpair</span><span class="w"> </span><span class="n">used</span><span class="p">:</span><span class="w"> </span><span class="mh">0x60</span>
<span class="n">ieee80211</span><span class="w"> </span><span class="n">phy0</span><span class="p">:</span><span class="w"> </span><span class="n">Atheros</span><span class="w"> </span><span class="n">AR9271</span><span class="w"> </span><span class="n">Rev</span><span class="p">:</span><span class="mi">1</span>
</code></pre></div>
<h2>Virtual interfaces</h2>
<p>Install the tools needed to configure wireless devices.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>iw
</code></pre></div>
<p>Have a look at the wireless devices present on the machine.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iw<span class="w"> </span>dev
phy#0
<span class="w"> </span>Interface<span class="w"> </span>wlan0
<span class="w"> </span>ifindex<span class="w"> </span><span class="m">3</span>
<span class="w"> </span>wdev<span class="w"> </span>0x1
<span class="w"> </span>addr<span class="w"> </span><span class="m">12</span>:34:56:78:ab:cd
<span class="w"> </span><span class="nb">type</span><span class="w"> </span>managed
</code></pre></div>
<p>Then show capabilities for the device.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iw<span class="w"> </span>phy<span class="w"> </span>phy0<span class="w"> </span>info
...
<span class="w"> </span>Supported<span class="w"> </span>interface<span class="w"> </span>modes:
<span class="w"> </span>*<span class="w"> </span>IBSS
<span class="w"> </span>*<span class="w"> </span>managed
<span class="w"> </span>*<span class="w"> </span>AP
<span class="w"> </span>*<span class="w"> </span>AP/VLAN
<span class="w"> </span>*<span class="w"> </span>monitor
<span class="w"> </span>*<span class="w"> </span>mesh<span class="w"> </span>point
<span class="w"> </span>*<span class="w"> </span>P2P-client
<span class="w"> </span>*<span class="w"> </span>P2P-GO
...
<span class="w"> </span>valid<span class="w"> </span>interface<span class="w"> </span>combinations:
<span class="w"> </span>*<span class="w"> </span><span class="c1">#{ managed, P2P-client } <= 2, #{ AP, mesh point, P2P-GO } <= 2,</span>
<span class="w"> </span>total<span class="w"> </span><<span class="o">=</span><span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="c1">#channels <= 1</span>
</code></pre></div>
<p>These lines shows that we can have only two virtual interfaces (<code>total <= 2</code>),
and that the combination <code>managed</code> (aka station) and <code>AP</code> (aka access-point) is valid.
That's what we want. <code>#channels <= 1</code> means that both interfaces will have to operate
on the same channel. That's a common limitation for most (maybe every) hardware.</p>
<p>Alright, let's create our virtual interfaces:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iw<span class="w"> </span>dev<span class="w"> </span>wlan0<span class="w"> </span>interface<span class="w"> </span>add<span class="w"> </span>wlan0_sta<span class="w"> </span><span class="nb">type</span><span class="w"> </span>station
$<span class="w"> </span>iw<span class="w"> </span>dev<span class="w"> </span>wlan0<span class="w"> </span>interface<span class="w"> </span>add<span class="w"> </span>wlan0_ap<span class="w"> </span><span class="nb">type</span><span class="w"> </span>__ap
</code></pre></div>
<p>If you look at them with <code>ifconfig -a</code>, you notice that the interfaces have the same MAC address.
This needs to be changed, every interface must have a unique MAC. You can put <em>almost</em> whathever
you want (check <a href="http://en.wikipedia.org/wiki/MAC_address">MAC address</a> on Wikipedia,
the most significant octet is NOT everything you want).
I personnally just tune the original address a little bit.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>dev<span class="w"> </span>wlan0_sta<span class="w"> </span>address<span class="w"> </span><span class="m">12</span>:34:56:78:ab:c0
$<span class="w"> </span>ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>dev<span class="w"> </span>wlan0_ap<span class="w"> </span>address<span class="w"> </span><span class="m">12</span>:34:56:78:ab:c1
</code></pre></div>
<h2>Configure the station (wpasupplicant)</h2>
<p>Credit: <a href="http://linuxcommando.blogspot.fr/2013/10/how-to-connect-to-wpawpa2-wifi-network.html">http://linuxcommando.blogspot.fr/2013/10/how-to-connect-to-wpawpa2-wifi-network.html</a></p>
<p>If you want your station to connect to a WPA-secured access-point,
you will need <a href="http://w1.fi/wpa_supplicant">WPA Supplicant</a>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>wpasupplicant
</code></pre></div>
<p>Alright, let's have a look at the access-points around.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ifconfig<span class="w"> </span>wlan0_sta<span class="w"> </span>up
$<span class="w"> </span>iw<span class="w"> </span>dev<span class="w"> </span>wlan0_sta<span class="w"> </span>scan
...
BSS<span class="w"> </span>de:ad:be:ef:ca:fe<span class="o">(</span>on<span class="w"> </span>wlan0_sta<span class="o">)</span>
<span class="w"> </span>...
<span class="w"> </span>SSID:<span class="w"> </span>MYHAPPYAP
<span class="w"> </span>RSN:<span class="w"> </span>*<span class="w"> </span>Version:<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>*<span class="w"> </span>Group<span class="w"> </span>cipher:<span class="w"> </span>TKIP
<span class="w"> </span>*<span class="w"> </span>Pairwise<span class="w"> </span>ciphers:<span class="w"> </span>CCMP<span class="w"> </span>TKIP
<span class="w"> </span>*<span class="w"> </span>Authentication<span class="w"> </span>suites:<span class="w"> </span>PSK
<span class="w"> </span>*<span class="w"> </span>Capabilities:<span class="w"> </span><span class="m">16</span>-PTKSA-RC<span class="w"> </span><span class="m">1</span>-GTKSA-RC<span class="w"> </span><span class="o">(</span>0x000c<span class="o">)</span>
...
</code></pre></div>
<p>Here, I assume you want to connect to the access-point named <code>MYHAPPYAP</code>.
The security protocol in use is <code>RSN</code>, which is another name for WPA2.
Alright, to create the configuration file needed to connect to this access-point,
there's nothing easier:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wpa_passphrase<span class="w"> </span>MYHAPPYAP<span class="w"> </span>>><span class="w"> </span>/etc/wpa_supplicant/wlan0_sta.conf
...<span class="w"> </span><span class="nb">type</span><span class="w"> </span><span class="k">in</span><span class="w"> </span>the<span class="w"> </span>passphrase<span class="w"> </span>and<span class="w"> </span>hit<span class="w"> </span>enter<span class="w"> </span>...
</code></pre></div>
<p>Now you can look at the configuration file generated if you're curious,
with something like <code>cat /etc/wpa_supplicant/wlan0_sta.conf</code>.</p>
<p>OK, we're almost there. Ensure that <code>wlan0_sta</code> is not connected
(just to show you one more command I learnt a few seconds ago :)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iw<span class="w"> </span>wlan0_sta<span class="w"> </span>link
Not<span class="w"> </span>connected.
</code></pre></div>
<p>Alright, let's launch wpasupplicant.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wpa_supplicant<span class="w"> </span>-i<span class="w"> </span>wlan0_sta<span class="w"> </span>-c<span class="w"> </span>/etc/wpa_supplicant/wlan0_sta.conf
Successfully<span class="w"> </span>initialized<span class="w"> </span>wpa_supplicant
wlan0_sta:<span class="w"> </span>SME:<span class="w"> </span>Trying<span class="w"> </span>to<span class="w"> </span>authenticate<span class="w"> </span>with<span class="w"> </span>de:ad:be:ef:ca:fe<span class="w"> </span><span class="o">(</span><span class="nv">SSID</span><span class="o">=</span><span class="s1">'MYHAPPYAP'</span><span class="w"> </span><span class="nv">freq</span><span class="o">=</span><span class="m">2462</span><span class="w"> </span>MHz<span class="o">)</span>
wlan0_sta:<span class="w"> </span>Trying<span class="w"> </span>to<span class="w"> </span>associate<span class="w"> </span>with<span class="w"> </span>de:ad:be:ef:ca:fe<span class="w"> </span><span class="o">(</span><span class="nv">SSID</span><span class="o">=</span><span class="s1">'MYHAPPYAP'</span><span class="w"> </span><span class="nv">freq</span><span class="o">=</span><span class="m">2462</span><span class="w"> </span>MHz<span class="o">)</span>
wlan0_sta:<span class="w"> </span>Associated<span class="w"> </span>with<span class="w"> </span>de:ad:be:ef:ca:fe
wlan0_sta:<span class="w"> </span>WPA:<span class="w"> </span>Key<span class="w"> </span>negotiation<span class="w"> </span>completed<span class="w"> </span>with<span class="w"> </span>de:ad:be:ef:ca:fe<span class="w"> </span><span class="o">[</span><span class="nv">PTK</span><span class="o">=</span>CCMP<span class="w"> </span><span class="nv">GTK</span><span class="o">=</span>TKIP<span class="o">]</span>
wlan0_sta:<span class="w"> </span>CTRL-EVENT-CONNECTED<span class="w"> </span>-<span class="w"> </span>Connection<span class="w"> </span>to<span class="w"> </span>de:ad:be:ef:ca:fe<span class="w"> </span>completed<span class="w"> </span><span class="o">[</span><span class="nv">id</span><span class="o">=</span><span class="m">0</span><span class="w"> </span><span class="nv">id_str</span><span class="o">=]</span>
</code></pre></div>
<p>In another shell, get to know more:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>iw<span class="w"> </span>wlan0_sta<span class="w"> </span>link
Connected<span class="w"> </span>to<span class="w"> </span>de:ad:be:ef:ca:fe<span class="w"> </span><span class="o">(</span>on<span class="w"> </span>wlan0_sta<span class="o">)</span>
...
</code></pre></div>
<p>The only thing left to do is to obtain an IP adress:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>dhclient<span class="w"> </span>wlan0_sta
</code></pre></div>
<h2>Configure the access-point (hostapd, dhcpd)</h2>
<p>Credit: <a href="https://wireless.wiki.kernel.org/en/users/Documentation/hostapd">https://wireless.wiki.kernel.org/en/users/Documentation/hostapd</a></p>
<p>To make an access-point, you definitely need to install <a href="https://w1.fi/hostapd/">Hostapd</a>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>hostapd
</code></pre></div>
<p>A basic configuration file:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s1">'#change wlan0_ap to your wireless device</span>
<span class="s1">interface=wlan0_ap</span>
<span class="s1">driver=nl80211</span>
<span class="s1">ssid=test</span>
<span class="s1">channel=1</span>
<span class="s1">'</span><span class="w"> </span>><span class="w"> </span>~/hostapd.conf
</code></pre></div>
<p>Let's launch it:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>hostapd<span class="w"> </span>~/hostapd.conf
Configuration<span class="w"> </span>file:<span class="w"> </span>.../hostapd.conf
Using<span class="w"> </span>interface<span class="w"> </span>wlan0<span class="w"> </span>with<span class="w"> </span>hwaddr<span class="w"> </span>xx:xx:xx:xx:xx:xx<span class="w"> </span>and<span class="w"> </span>ssid<span class="w"> </span><span class="s2">"test"</span>
wlan0:<span class="w"> </span>interface<span class="w"> </span>state<span class="w"> </span>UNINITIALIZED->ENABLED
wlan0:<span class="w"> </span>AP-ENABLED
</code></pre></div>
<p>If you see something like this, everything is alright.
If you scan for wireless access-points on another machine, you can see that the <code>test</code> access-point exists.
Now, the next step is to install and configure a DHCP server.
Since we're gonna play with it a little bit, it's better to disable the service,
and start/stop it manually.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>isc-dhcp-server
$<span class="w"> </span>systemctl<span class="w"> </span>disable<span class="w"> </span>isc-dhcp-server
</code></pre></div>
<p>Create the configuration file for your DHCP server
(this is just a quick example, change the IP adress according to your needs):</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s1">'</span>
<span class="s1">subnet 192.168.2.0 netmask 255.255.255.0 {</span>
<span class="s1"> option routers 192.168.2.1;</span>
<span class="s1"> range 192.168.2.10 192.168.2.20;</span>
<span class="s1">}</span>
<span class="s1">'</span><span class="w"> </span>><span class="w"> </span>/etc/dhcp/wlan0_ap.conf
</code></pre></div>
<p>Set the interface up and launch the DHCP server:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ifconfig<span class="w"> </span>wlan0_ap<span class="w"> </span><span class="m">192</span>.168.2.1<span class="w"> </span>up
$<span class="w"> </span>dhcpd<span class="w"> </span>-cf<span class="w"> </span>/etc/dhcp/wlan0_ap.conf<span class="w"> </span>wlan0_ap
</code></pre></div>
<p>Alright, from this point you should be able to connect to your AP from another device.</p>
<h2>The channel issue</h2>
<p>There is one thing that you should not underastimate: both virtual devices must operate on the same channel.
The implication of that is simple. If your station is connected to a remote AP with auto-channel enabled, you
will be in trouble pretty quickly. Because as soon as the remote AP will change channel, your station
won't be able to re-connect on a different channel, since you already have your own local AP up and runnning.</p>
<p>To handle that, you need to bring both interfaces down, then bring them up. And of course,
you must bring up the station first, because it's the one who decides which channel to use.</p>
<p>If you write a script that's supposed to maintain the wifi up all the time, you will have a little bit
of work to handle that properly.</p>
<p>The other solution, if you can, is to disable auto-channel on the remote AP, so that you KNOW for sure
that your wifi operates on a given channel and never changes.</p>
<h2>References</h2>
<p>This was a long post, wasn't it? I wouldn't have made it without help:</p>
<ul>
<li><a href="https://wireless.wiki.kernel.org">https://wireless.wiki.kernel.org</a></li>
<li><a href="https://wiki.archlinux.org/index.php/Software_access_point">https://wiki.archlinux.org/index.php/Software_access_point</a></li>
<li><a href="http://superuser.com/q/155795">http://superuser.com/q/155795</a></li>
<li><a href="http://wiki.openwrt.org/doc/howto/wireless.utilities">http://wiki.openwrt.org/doc/howto/wireless.utilities</a></li>
</ul>Debian Jessie on Raspberry Pi 22015-12-04T00:00:00+00:002015-12-04T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-04:/2015/12/04/debian-jessie-on-raspberry-pi-2/<p>How to install <a href="https://www.debian.org/releases/jessie/">Debian Jessie</a>
on a <a href="https://www.raspberrypi.org/products/raspberry-pi-2-model-b/">Raspberry Pi 2</a>
(aka RPi2), using the Debian image provided by <a href="https://www.collabora.com/">Collabora</a>.</p>
<p>How to install <a href="https://www.debian.org/releases/jessie/">Debian Jessie</a>
on a <a href="https://www.raspberrypi.org/products/raspberry-pi-2-model-b/">Raspberry Pi 2</a>
(aka RPi2), using the Debian image provided by <a href="https://www.collabora.com/">Collabora</a>.</p>
<p>More info here: <a href="http://sjoerd.luon.net/posts/2015/02/debian-jessie-on-rpi2/">http://sjoerd.luon.net/posts/2015/02/debian-jessie-on-rpi2/</a>.</p>
<h2>Installing Debian on the Micro SD-Card</h2>
<p>Grab the Debian image files provided by Collabora,
thanks to the work of Sjoerd Simons.
You can find it here: <a href="https://images.collabora.co.uk/rpi2/">https://images.collabora.co.uk/rpi2/</a>.
Download all the files in the same directory.</p>
<p>In order to install it on the SD-Card, we need <code>bmap-tools</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">bmap</span><span class="o">-</span><span class="n">tools</span>
</code></pre></div>
<p>We need a Micro SD-Card. I personnaly cleaned it completely with <code>gparted</code>,
I mean, removed every partition.
Then assuming the SD-Card is at <code>/dev/sdb</code> and is not mounted, run:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>bmaptool<span class="w"> </span>copy<span class="w"> </span>jessie-rpi2-20150202.img.gz<span class="w"> </span>/dev/sdb<span class="w"> </span>
bmaptool:<span class="w"> </span>info:<span class="w"> </span>discovered<span class="w"> </span>bmap<span class="w"> </span>file<span class="w"> </span><span class="s1">'jessie-rpi2-20150202.img.bmap'</span>
bmaptool:<span class="w"> </span>info:<span class="w"> </span>block<span class="w"> </span>map<span class="w"> </span>format<span class="w"> </span>version<span class="w"> </span><span class="m">2</span>.0
bmaptool:<span class="w"> </span>info:<span class="w"> </span><span class="m">768256</span><span class="w"> </span>blocks<span class="w"> </span>of<span class="w"> </span>size<span class="w"> </span><span class="m">4096</span><span class="w"> </span><span class="o">(</span><span class="m">2</span>.9<span class="w"> </span>GiB<span class="o">)</span>,<span class="w"> </span>mapped<span class="w"> </span><span class="m">123252</span><span class="w"> </span>blocks<span class="w"> </span><span class="o">(</span><span class="m">481</span>.5<span class="w"> </span>MiB<span class="w"> </span>or<span class="w"> </span><span class="m">16</span>.0%<span class="o">)</span>
bmaptool:<span class="w"> </span>info:<span class="w"> </span>copying<span class="w"> </span>image<span class="w"> </span><span class="s1">'jessie-rpi2-20150202.img.gz'</span><span class="w"> </span>to<span class="w"> </span>block<span class="w"> </span>device<span class="w"> </span><span class="s1">'/dev/sdb'</span><span class="w"> </span>using<span class="w"> </span>bmap<span class="w"> </span>file<span class="w"> </span><span class="s1">'jessie-rpi2-20150202.img.bmap'</span>
bmaptool:<span class="w"> </span>info:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span>copied
bmaptool:<span class="w"> </span>info:<span class="w"> </span>synchronizing<span class="w"> </span><span class="s1">'/dev/sdb'</span>
bmaptool:<span class="w"> </span>info:<span class="w"> </span>copying<span class="w"> </span>time:<span class="w"> </span><span class="m">40</span>.7s,<span class="w"> </span>copying<span class="w"> </span>speed<span class="w"> </span><span class="m">11</span>.8<span class="w"> </span>MiB/sec
</code></pre></div>
<p>After that, we can have a look at it again with gparted, and mount partitions, just out of curiosity.
There's a small FAT16 partition for firmwares. It contains the kernel image, amongst other things.
The rootfs is written to the EXT4 partition.</p>
<p>It's the best moment to enlarge the rootfs partition if you feel you will need more space for the system.</p>
<p>I also created an EXT4 partition on the remaining space, that I will mount as <code>/home</code>. I took care to label it <code>home</code>.</p>
<h2>First steps with the RPI2</h2>
<p>We're going to communicate with the RPI2 through an ethernet cable.
When booted, the RPI2 will send a DHCP request: it expects that a DHCP server somewhere will answer.
So you can either:</p>
<ul>
<li>plug the RPI2 to your router, and there's nothing to do (except to find which IP address the RPI2 received).</li>
<li>plug the RPI2 directly to your computer. In this case, you must have a DHCP server up and running.</li>
</ul>
<p>When the RPI2 is booted, we can connect with SSH. As mentionned, the default root password is <code>debian</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">ssh</span><span class="w"> </span><span class="n">root</span><span class="mf">@192.168.1.11</span>
</code></pre></div>
<p>Easy, wasn't it? Let's have a look around...</p>
<div class="highlight"><pre><span></span><code># Have a look at the CPU
lscpu
# Get the temperature
awk '{printf "%3.1f\n", $1/1000}' /sys/class/thermal/thermal_zone0/temp
</code></pre></div>
<p>Now, it's time to be serious. Let's change the root password:</p>
<div class="highlight"><pre><span></span><code>passwd
</code></pre></div>
<p>Change the hostname:</p>
<div class="highlight"><pre><span></span><code>echo 'rpi2' > /etc/hostname
reboot
</code></pre></div>
<p>Generate new SSH keys (because SSH keys were embedded in the Debian image by mistake):</p>
<div class="highlight"><pre><span></span><code>rm /etc/ssh/ssh_host_*_key*
dpkg-reconfigure openssh-server
</code></pre></div>
<p>Reconfigure time and locales:</p>
<div class="highlight"><pre><span></span><code>dpkg-reconfigure locales
dpkg-reconfigure tzdata
</code></pre></div>
<p>Let's tweak a little bit the root account to make it more comfortable:</p>
<div class="highlight"><pre><span></span><code>vi ~/.bashrc
</code></pre></div>
<p>Automount the <code>/home</code> partition.</p>
<div class="highlight"><pre><span></span><code>cat /proc/partitions
echo '/dev/mmcblk0p3 /home ext4 relatime,errors=remount-ro,discard 0 2' >> /etc/fstab
</code></pre></div>
<h2>Upgrading Debian</h2>
<p>At first, we must update Collabora keyring, which expired beginning of June,
and causes a <code>KEYEXPIRED error</code> when running <code>apt-get update</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>apt-key<span class="w"> </span>list<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>collabora<span class="w"> </span>-A<span class="w"> </span><span class="m">4</span>
/etc/apt/trusted.gpg.d/collabora-obs-archive-keyring.gpg
--------------------------------------------------------
pub<span class="w"> </span>2048R/B5FBB289<span class="w"> </span><span class="m">2013</span>-05-09<span class="w"> </span><span class="o">[</span>expired:<span class="w"> </span><span class="m">2015</span>-06-01<span class="o">]</span>
uid<span class="w"> </span>Collabora<span class="w"> </span>OBS<span class="w"> </span>Automatic<span class="w"> </span>Signing<span class="w"> </span>Key<span class="w"> </span><example@example.com>
</code></pre></div>
<p>I'm not good when it comes to security. So, it took me a while to find a solution,
and it may not be the right one. Anyway, I downloaded the latest keyring package from Collabora,
installed it with <code>dpkg</code>, and it did the tricks.</p>
<div class="highlight"><pre><span></span><code>wget https://repositories.collabora.co.uk/debian/pool/rpi2/c/collabora-obs-archive-keyring/collabora-obs-archive-keyring_0.5+b1_all.deb
dpkg -i collabora-obs-archive-keyring_0.5+b1_all.deb
</code></pre></div>
<p>Then, we must give priority to Collabora repository. This is needed, otherwise official
Debian packages may replace the Collabora ones. This could be bad, since Collabora
brings what's currently missing in the official Debian to work on RPI2.
At the moment, Collabora provides 4 packages, and only one is also provided by Debian: <code>flash-kernel</code>.
You can <code>apt-pin</code> only this package, or the whole Collabora repository. I choosed the later option:</p>
<div class="highlight"><pre><span></span><code>echo '
Package: *
Pin: origin repositories.collabora.co.uk
Pin-Priority: 1000
' > /etc/apt/preferences.d/collabora
</code></pre></div>
<p>If you're curious, you can have a look at the Collabora packages that are currently installed, however you need <code>aptitude</code> for that.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># List origins for packages that are installed</span>
<span class="n">apt</span><span class="o">-</span><span class="n">cache</span><span class="w"> </span><span class="n">policy</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sed</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="s1">'s/.*o=\([^,]\+\).*/</span><span class="se">\1</span><span class="s1">/p'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">uniq</span>
<span class="c1"># List packages installed from a given origin</span>
<span class="n">apt</span><span class="o">-</span><span class="n">get</span><span class="w"> </span><span class="n">install</span><span class="w"> </span><span class="n">aptitude</span>
<span class="n">aptitude</span><span class="w"> </span><span class="n">search</span><span class="w"> </span><span class="s1">'~S ~i (?origin("Collabora"))'</span>
<span class="n">i</span><span class="w"> </span><span class="n">collabora</span><span class="o">-</span><span class="n">obs</span><span class="o">-</span><span class="n">archive</span><span class="o">-</span><span class="n">keyring</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">GnuPG</span><span class="w"> </span><span class="n">keys</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">Collabora</span><span class="w"> </span><span class="n">OBS</span><span class="w"> </span><span class="n">repositories</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">archives</span>
<span class="n">i</span><span class="w"> </span><span class="n">flash</span><span class="o">-</span><span class="n">kernel</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">utility</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">make</span><span class="w"> </span><span class="n">certain</span><span class="w"> </span><span class="n">embedded</span><span class="w"> </span><span class="n">devices</span><span class="w"> </span><span class="n">bootable</span>
<span class="n">i</span><span class="w"> </span><span class="n">linux</span><span class="o">-</span><span class="n">image</span><span class="o">-</span><span class="mf">3.18</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">trunk</span><span class="o">-</span><span class="n">rpi2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Linux</span><span class="w"> </span><span class="mf">3.18</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">Raspberry</span><span class="w"> </span><span class="n">Pi</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">model</span><span class="w"> </span><span class="n">B</span><span class="o">+</span>
<span class="n">i</span><span class="w"> </span><span class="n">raspberrypi</span><span class="o">-</span><span class="n">bootloader</span><span class="o">-</span><span class="n">nokernel</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Raspberry</span><span class="w"> </span><span class="n">Pi</span><span class="w"> </span><span class="n">bootloader</span>
</code></pre></div>
<p>Now, we should add the security update repository.</p>
<div class="highlight"><pre><span></span><code><span class="err">echo</span><span class="w"> </span><span class="err">'</span>
<span class="k">deb</span><span class="w"> </span><span class="s">http://security.debian.org/</span><span class="w"> </span><span class="kp">jessie/updates</span><span class="w"> </span><span class="kp">main</span><span class="w"> </span><span class="kp">non-free</span>
<span class="err">'</span><span class="w"> </span><span class="err">>></span><span class="w"> </span><span class="err">/etc/apt/sources.list</span>
</code></pre></div>
<p>Alright, cross your fingers and invoke the almighty apt commands:</p>
<div class="highlight"><pre><span></span><code>apt-get update && apt-get dist-upgrade
</code></pre></div>
<h2>More stuff</h2>
<p>Make yourself at home</p>
<div class="highlight"><pre><span></span><code>apt-get install bash-completion
reboot
apt-get install vim
</code></pre></div>
<p>Add the <code>sid</code> repository, pin it to a low priority, so that you can use it only for some packages.</p>
<div class="highlight"><pre><span></span><code><span class="err">echo</span><span class="w"> </span><span class="err">'</span>
<span class="k">deb</span><span class="w"> </span><span class="s">http://ftp.debian.org/debian</span><span class="w"> </span><span class="kp">sid</span><span class="w"> </span><span class="kp">main</span>
<span class="err">'</span><span class="w"> </span><span class="err">>></span><span class="w"> </span><span class="err">/etc/apt/sources.list</span>
<span class="err">echo</span><span class="w"> </span><span class="err">'</span>
<span class="err">Pin:</span><span class="w"> </span><span class="err">release</span><span class="w"> </span><span class="err">a=unstable</span>
<span class="err">Pin-Priority:</span><span class="w"> </span><span class="err">100</span>
<span class="err">'</span><span class="w"> </span><span class="err">></span><span class="w"> </span><span class="err">/etc/apt/preferences.d/sid</span>
</code></pre></div>A Post-Receive Git Hook For Octopress2015-12-03T00:00:00+00:002015-12-03T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-03:/2015/12/03/a-post-receive-git-hook-for-octopress/<p>Here comes the post-receive git hook I used along with <a href="http://octopress.org/">Octopress</a>,
to finish deploying the static blog server-side.</p>
<p>Here comes the post-receive git hook I used along with <a href="http://octopress.org/">Octopress</a>,
to finish deploying the static blog server-side.</p>
<h2>Introduction</h2>
<p>What do we want to achieve?
Well, each time a new version of the static site, built by Octopress,
is pushed to the git repository, we'd like git to copy it to the
document root served by Apache (<code>/srv/www/blog-test</code> in this example).</p>
<p>In order to do that, we need to write a git hook, and to fiddle a little bit
with permissions.</p>
<h2>Git hook</h2>
<p>We're talking about a post-receive git hook. Here it is:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#</span>
<span class="c1"># Git hook to finish deploying an Octopress website server-side.</span>
<span class="c1">#</span>
<span class="c1"># Each time the 'master' branch is updated, just clone it, and copy</span>
<span class="c1"># the content (aka the static website) to the location expected by</span>
<span class="c1"># the web server.</span>
<span class="c1"># Git must have the permission to write there.</span>
<span class="nb">set</span><span class="w"> </span>-e
<span class="nv">REPO</span><span class="o">=</span><span class="k">$(</span>basename<span class="w"> </span><span class="s2">"</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>s/<span class="se">\.</span>git$//<span class="k">)</span>
<span class="nv">TMPDIR</span><span class="o">=</span>/tmp/<span class="nv">$REPO</span>
<span class="nv">WWWDIR</span><span class="o">=</span>/srv/www/<span class="nv">$REPO</span>
<span class="nv">stdin</span><span class="o">=</span><span class="k">$(</span>cat<span class="k">)</span>
<span class="nv">refs</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$stdin</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="s1">' '</span><span class="w"> </span>-f3<span class="k">)</span>
<span class="c1"># Exit if nothing was pushed to the master branch</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$refs</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-q<span class="w"> </span><span class="s1">'refs/heads/master'</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
<span class="c1"># Ensure the target directory is writable</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$WWWDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"'</span><span class="nv">$WWWDIR</span><span class="s2">' doesn't exist!"</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="o">[</span><span class="w"> </span>-w<span class="w"> </span><span class="s2">"</span><span class="nv">$WWWDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"'</span><span class="nv">$WWWDIR</span><span class="s2">' is not writable!"</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="c1"># Clone the master branch, and copy the static website</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Removing '</span><span class="nv">$TMPDIR</span><span class="s2">'..."</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>rm<span class="w"> </span>-fr<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">"</span>
git<span class="w"> </span>clone<span class="w"> </span>.<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">"</span><span class="w"> </span>--branch<span class="w"> </span>master<span class="w"> </span><span class="o">&&</span><span class="w"> </span>rm<span class="w"> </span>-fr<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">/.git"</span>
<span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="nv">$WWWDIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Emptying '</span><span class="nv">$WWWDIR</span><span class="s2">'..."</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>rm<span class="w"> </span>-fr<span class="w"> </span><span class="s2">"</span><span class="nv">$WWWDIR</span><span class="s2">/*"</span>
mv<span class="w"> </span>-v<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">/*"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$WWWDIR</span><span class="s2">"</span>
rmdir<span class="w"> </span><span class="s2">"</span><span class="nv">$TMPDIR</span><span class="s2">"</span>
</code></pre></div>
<p>There's a few things to notice in this script:</p>
<ul>
<li>the <code>-e</code> in the first line, to exit if an error happens</li>
<li><code>WWWDIR</code> that you may change to suit your config</li>
<li><code>rm -fr</code> here and there, so you'd better read and understand before running that wildly</li>
<li><code>echo</code> commands are there for a good reason. Indeed, stdout is forwarded by git,
and ends up in the console of the user that did the <code>git push</code>.
Or more precisely, <code>octopress deploy</code> in our case.</li>
</ul>
<h2>Settling write permissions</h2>
<p>After that, we just have a few ownership details to solve.
As you can see in the script above, git will write inside the directory <code>/srv/www/WWWDIR</code>.
We must manually create this directory, and allow git to write there.
A solution is to add git to the group <code>www-data</code> (which is the user Apache is running),
so that they can both access it.</p>
<p>First, let's create the document root directory:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/srv/www
$<span class="w"> </span>mkdir<span class="w"> </span>blog-test
$<span class="w"> </span>chgrp<span class="w"> </span>www-data<span class="w"> </span>blog-test
$<span class="w"> </span>chmod<span class="w"> </span>g+w<span class="w"> </span>blog-test
$<span class="w"> </span>ls<span class="w"> </span>-l
total<span class="w"> </span><span class="m">4</span>
drwxrwxr-x<span class="w"> </span><span class="m">2</span><span class="w"> </span>root<span class="w"> </span>www-data<span class="w"> </span><span class="m">4096</span><span class="w"> </span>Sep<span class="w"> </span><span class="m">29</span><span class="w"> </span><span class="m">19</span>:08<span class="w"> </span>blog-test
</code></pre></div>
<p>And now, just add git to the right group:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>addgroup<span class="w"> </span>git<span class="w"> </span>www-data
</code></pre></div>Getting Started with Apache2015-12-02T00:00:00+00:002015-12-02T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-02:/2015/12/02/getting-started-with-apache/<p>A quick introduction to <a href="http://httpd.apache.org">Apache</a> for the noobs.</p>
<p>A quick introduction to <a href="http://httpd.apache.org">Apache</a> for the noobs.</p>
<h2>Introduction</h2>
<p>I host this blog on a <a href="https://www.debian.org/">Debian</a> VPS.
So at some point I had to setup a web server,
otherwise how could you be reading this blog right now?</p>
<p>There are many web servers around, some of the most populars being
<a href="http://httpd.apache.org">Apache</a>, <a href="http://nginx.org">Nginx</a> or
<a href="http://www.lighttpd.net">Lighttpd</a>, to cite only a few.
I never used Apache before, so I thought it was the opportunity to fill this gap.</p>
<h2>Installation</h2>
<p>Installing is nothing fancy:</p>
<div class="highlight"><pre><span></span><code>apt-get install apache2
</code></pre></div>
<p>Right now Apache is configured and ready to serve.
If you have a look at your server through your web browser,
you should see the default Apache page for Debian.</p>
<h2>Basic Configuration</h2>
<p>Now, let's create a new site. A basic site definition looks like that:</p>
<div class="highlight"><pre><span></span><code>echo<span class="w"> </span>'
<span class="nt"><VirtualHost</span><span class="w"> </span><span class="err">*:80</span><span class="nt">></span>
<span class="w"> </span>DocumentRoot<span class="w"> </span>/srv/www/blog-test
<span class="nt"></VirtualHost></span>
'><span class="w"> </span>/etc/apache2/sites-available/blog-test.conf
</code></pre></div>
<p>Notice that I like to put my websites in <code>/srv/www</code>.
On Debian, the default Apache configuration only allows serving websites
from <code>/var/www</code>. To allow <code>/srv/www</code> (or whatever suits you), edit the file
<code>/etc/apache2/apache2.conf</code>. Look for <code>/srv</code>.
The configuration is already there, you just have to uncomment it.</p>
<p>Then, we disable the default site served by Apache, and enable our site:</p>
<div class="highlight"><pre><span></span><code>a2dissite 000-default
a2ensite blog-test
systemctl restart apache2
</code></pre></div>
<p>And that's it! Now Apache serves the data from <code>/srv/www/blog-test</code>.
So you should copy your website in this location. Then, don't forget
to change the ownership for this directory. On Debian, Apache is configured
to run as <code>www-data</code>.</p>
<div class="highlight"><pre><span></span><code>cp -r path/of/your/website/* /srv/www/blog-test
chown -R www-data:www-data /srv/www/blog-test
</code></pre></div>Setting up Gitolite2015-12-01T00:00:00+00:002015-12-01T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-12-01:/2015/12/01/setting-up-gitolite/<p>How to install and setup <a href="http://gitolite.com">Gitolite</a> on a remote server,
using the upstream source.</p>
<p>How to install and setup <a href="http://gitolite.com">Gitolite</a> on a remote server,
using the upstream source.</p>
<h2>Introduction</h2>
<p>I'd like to point out that the official Gitolite documentation
is very well written, more than complete, and constantly updated.
You will find everything you need and more.
If you're new to Gitolite, you'd better learn from the official doc, this is my advice.</p>
<p>I may also mention that there is a <a href="https://www.debian.org/">Debian</a> for Gitolite.
I tried it, but in the end I prefered to install Gitolite from upstream.
Mainly because:</p>
<ul>
<li>Debian's default location for repositories is at <code>/var/lib/</code>.
I like it better at <code>/home</code>, since this partition is usually much bigger
(at least on my setup), and is the perfect place to hold data.
This default location is said to be changeable through <code>dpkg-reconfigure</code>,
but I'm not sure it works that well, I saw some tickets about that.</li>
<li>The Debian package is a little bit outdated at the moment.</li>
<li>There are some unresolved tickets lingering for a while, and that makes me wary.</li>
</ul>
<p>In the end, I felt like I was wasting my time with the package,
and I decided to go for upstream install.
It turns out that installing Gitolite from upstream is damn easy and well-documented,
so that's what we're going to do in this post.</p>
<h2>Prerequisites</h2>
<p>I assume we have to machines: a server and a workstation.
We're going to install and configure Gitolite on the server.
Of course, there's a SSH server up and running on the server.
The server hostname is <code>server</code>, our username is <code>bob</code>.
We use Debian (though it doesn't matter much).</p>
<h2>Installing (server-side)</h2>
<p>At first, let's ensure that every dependencies for Gitolite are satisfied.
For that, <code>apt-cache</code> can help, even if we're not going to install Gitolite with apt.</p>
<div class="highlight"><pre><span></span><code><span class="nv">apt</span><span class="o">-</span><span class="nv">cache</span><span class="w"> </span><span class="k">show</span><span class="w"> </span><span class="nv">gitolite3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">grep</span><span class="w"> </span><span class="o">^</span><span class="nv">Depends</span>
<span class="nv">apt</span><span class="o">-</span><span class="nv">get</span><span class="w"> </span><span class="nv">install</span><span class="w"> </span><span class="nv">git</span><span class="w"> </span><span class="nv">git</span><span class="o">-</span><span class="nv">core</span><span class="w"> </span><span class="nv">perl</span>
</code></pre></div>
<p>We need a new user that will host our git repositories.
I name it <code>git</code>, but if you don't like that just pick another name.</p>
<div class="highlight"><pre><span></span><code>adduser git
</code></pre></div>
<p>Let's switch to this new user, then get Gitolite sources from upstream:</p>
<div class="highlight"><pre><span></span><code>su - git
git clone git://github.com/sitaramc/gitolite
</code></pre></div>
<p>I prefer to run an official version rather than the current development version.
A simple <code>git checkout</code> of the latest tag will do.</p>
<div class="highlight"><pre><span></span><code>cd gitolite
git tag -l
git checkout v3.6.3
</code></pre></div>
<p>Now, let's install Gitolite. There are a few options for installation,
you can check it out with <code>./install -h</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">cd</span>
<span class="n">mkdir</span><span class="w"> </span><span class="n">bin</span>
<span class="n">gitolite</span><span class="o">/</span><span class="n">install</span><span class="w"> </span><span class="o">-</span><span class="nf">ln</span>
<span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="n">bin</span>
<span class="w"> </span><span class="n">total</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="n">lrwxrwxrwx</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">git</span><span class="w"> </span><span class="n">git</span><span class="w"> </span><span class="mi">31</span><span class="w"> </span><span class="n">Sep</span><span class="w"> </span><span class="mi">25</span><span class="w"> </span><span class="mi">15</span><span class="o">:</span><span class="mi">22</span><span class="w"> </span><span class="n">gitolite</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">git</span><span class="o">/</span><span class="n">gitolite</span><span class="o">/</span><span class="n">src</span><span class="o">/</span><span class="n">gitolite</span>
</code></pre></div>
<h2>Setting up</h2>
<p>At this point, Gitolite is installed and almost ready to go.
We just need to feed it the administrator's public key.
So, back on your workstation, create a SSH keypair if you don't have one:</p>
<div class="highlight"><pre><span></span><code>ssh-keygen
</code></pre></div>
<p>Then copy the public key to the server.
The file name is important here, it should be your name as a Gitolite user.</p>
<div class="highlight"><pre><span></span><code><span class="n">scp</span><span class="w"> </span><span class="o">~/</span><span class="p">.</span><span class="n">ssh</span><span class="o">/</span><span class="n">id_rsa</span><span class="p">.</span><span class="n">pub</span><span class="w"> </span><span class="n">git</span><span class="nv">@server</span><span class="err">:</span><span class="o">~/</span><span class="k">admin</span><span class="p">.</span><span class="n">pub</span>
</code></pre></div>
<p>Back on the server, you can now finish to configure Gitolite:</p>
<div class="highlight"><pre><span></span><code>gitolite setup -pk admin.pub
rm admin.pub
</code></pre></div>
<p>Done!</p>
<h2>Testing</h2>
<p>Now you can try the most basic test on your workstation:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ssh<span class="w"> </span>git@server
PTY<span class="w"> </span>allocation<span class="w"> </span>request<span class="w"> </span>failed<span class="w"> </span>on<span class="w"> </span>channel<span class="w"> </span><span class="m">0</span>
hello<span class="w"> </span>admin,<span class="w"> </span>this<span class="w"> </span>is<span class="w"> </span>git@server<span class="w"> </span>running<span class="w"> </span>gitolite3<span class="w"> </span>v3.6.3-0-g5d24ae6<span class="w"> </span>on<span class="w"> </span>git<span class="w"> </span><span class="m">2</span>.1.4
<span class="w"> </span>R<span class="w"> </span>W<span class="w"> </span>gitolite-admin
<span class="w"> </span>R<span class="w"> </span>W<span class="w"> </span>testing
Connection<span class="w"> </span>to<span class="w"> </span>server<span class="w"> </span>closed.
</code></pre></div>
<p>The fist line talking about 'request failed' is OK. This is due to the way Gitolite uses SSH.
To unveil this black magic, there's plenty of explanation on the official Gitolite website.
You may also have a look at the <code>~/.ssh/authorized_keys</code> file for the git user (server-side).</p>
<h2>Administrating</h2>
<p>Now that everything is setup, you can start administrating your repositories,
using the special repo <code>gitolite-admin</code>. You do that from your workstation.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>git@server:gitolite-admin
</code></pre></div>
<p>Let's suppose you want to rename the user <code>admin</code> into <code>bob</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>gitolite-admin
$<span class="w"> </span>git<span class="w"> </span>mv<span class="w"> </span>keydir/admin.pub<span class="w"> </span>keydir/bob.pub
$<span class="w"> </span>sed<span class="w"> </span>-i<span class="w"> </span><span class="s1">'s/admin/bob/'</span><span class="w"> </span>conf/gitolite.conf
$<span class="w"> </span>git<span class="w"> </span>commit<span class="w"> </span>-am<span class="w"> </span><span class="s2">"Rename user: admin -> bob"</span>
$<span class="w"> </span>git<span class="w"> </span>push
</code></pre></div>
<p>That was easy, wasn't it?</p>
<h2>Upgrading</h2>
<p>Upgrading to a newer version of Gitolite is done in 3 steps:</p>
<ul>
<li>Enter the Gitolite sources and run <code>git pull</code> to get the latest version</li>
<li>Then use the same install command as the one that was used to install (with option <code>-ln</code> here)</li>
<li>Then run <code>gitolite setup</code></li>
</ul>My Daily Jekyll Workflow2015-11-30T00:00:00+00:002015-11-30T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-11-30:/2015/11/30/my-daily-jekyll-workflow/<p>My personal <a href="http://jekyllrb.com/">Jekyll</a> workflow, involving
<a href="https://git-scm.com/">Git</a>, <a href="https://rsync.samba.org/">Rsync</a>,
and my own homebrewed script, the almighty Jekyllkenny...</p>
<p>My personal <a href="http://jekyllrb.com/">Jekyll</a> workflow, involving
<a href="https://git-scm.com/">Git</a>, <a href="https://rsync.samba.org/">Rsync</a>,
and my own homebrewed script, the almighty Jekyllkenny...</p>
<h2>Introduction</h2>
<p>For people who already know about the "<a href="https://pages.github.com/">GitHub Pages</a>"
workflow (you know, the weird two-branches workflow, one branch for the source,
the other for the compiled static site), I stop you right now. This is not what
I'm going to talk about.</p>
<p>My setup is slightly different, simpler and much more easier to picture.
I do not version the compiled static site. I use Git to version ONLY the source.
Then I use Rsync to update the compiled site on the server. That's all.
It's just the usual way of doing things, nothing fancy. And it's real easy to setup.</p>
<h2>Source versionning with git</h2>
<p>So, just to tell you the context: I have a VPS where I version the source of my blog with git.
On the VPS, I manage my git repositories with <a href="http://gitolite.com/">Gitolite</a>
(I have another post talking about that).</p>
<p>OK, so how do you configure that? It's damn easy if you're used to git.
First, as soon as you create your new blog, start versionning it.</p>
<div class="highlight"><pre><span></span><code>jekyll new blog
cd blog
git init
git add .gitignore *
git commit -m"Initial commit: jekyll skeleton"
</code></pre></div>
<p>Now, let's assume that your have a vps somewhere, which is known under the name <code>vps</code>.
I also assume that you access it through the user <code>git</code> (just the usual way), and that
you already created an empty git repository on the server, named <code>blog.git</code>. The following
commands come easy:</p>
<div class="highlight"><pre><span></span><code><span class="n">git</span><span class="w"> </span><span class="n">remote</span><span class="w"> </span><span class="k">add</span><span class="w"> </span><span class="n">origin</span><span class="w"> </span><span class="n">git</span><span class="nv">@vps</span><span class="err">:</span><span class="n">blog</span>
<span class="n">git</span><span class="w"> </span><span class="n">push</span><span class="w"> </span><span class="c1">--set-upstream origin master</span>
</code></pre></div>
<p>That's all, your blog is now versionned and safe, you can commit and push whenever you want,
and undo your worst mistakes (like the dreaded <code>rm -rf _posts</code>) without stress :)</p>
<p>Versionning with git is really the basic. Blogging is pretty much like coding, you can't afford
to loose it all once you started investing some time in it. So you must version, at least for
the sake of having a backup.</p>
<h2>Pushing to the server with rsync</h2>
<p>This is only one of the several ways of sending your static blog to the server.
I have several reasons to choose rsync for this task:</p>
<ul>
<li>it's made for syncing data, you can't find a better tool for the task</li>
<li>it uses SSH authentication with the server, something that's already setup</li>
<li>I never heard nobody blame rsync :)</li>
<li>it's easy to use</li>
</ul>
<p>It just boils down to a simple command:</p>
<div class="highlight"><pre><span></span><code><span class="n">rsync</span><span class="w"> </span><span class="o">--</span><span class="n">archive</span><span class="w"> </span><span class="o">--</span><span class="n">delete</span><span class="w"> </span><span class="o">--</span><span class="n">compress</span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="n">chown</span><span class="w"> </span><span class="n">www</span><span class="o">-</span><span class="n">data</span><span class="p">:</span><span class="n">www</span><span class="o">-</span><span class="n">data</span><span class="w"> </span>\
<span class="w"> </span><span class="o">./</span><span class="n">_site</span><span class="o">/</span><span class="w"> </span><span class="n">user</span><span class="err">@</span><span class="n">vps</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">blog</span>
</code></pre></div>
<p>Notice some smart options here and there:</p>
<ul>
<li><code>--chown</code> to change the user and group of the synced files</li>
<li>a trailing slash at the end of the source <code>./_site/</code> to avoid creating a sub-directory inside <code>/var/www/blog</code></li>
</ul>
<p>Of course, it's tedious to remember such command each time. To make it easier, you can
alias it, or if you're brave enough, you can start writing a little script.</p>
<h2>Writing posts</h2>
<p>I create a new post in the <code>_drafts</code> sub-directory. I work on it and preview locally,
using the <code>--draft</code> option of Jekyll.</p>
<div class="highlight"><pre><span></span><code>mkdir _drafts
vi _drafts/my-post.md
jekyll server --draft
</code></pre></div>
<p>When I want to publish, I twist the usual Jekyll workflow a little bit.
The common way would be just to move the draft to the post directory, and name it
according to the pattern expected by Jekyll, that's to say, prefixed by the date.
So the command would be:</p>
<div class="highlight"><pre><span></span><code>mv _drafts/my-post.md _posts/2015-11-18-my-post.md
</code></pre></div>
<p>This is no problem, except that I don't like too much this naming.
This is because my blog is mainly a memento for myself, where I write commands
and stuff that I would forget otherwise. When I need to recall something, I
don't look for it in my web browser. I'm a command-line guy, so I prefer to
go in my Jekyll post directory, and search in my posts directly.</p>
<p>That's one of the reasons I like Jekyll, because it doesn't bury my content in a
database, it just let the posts live in simples markdown files, that I can browse
anytime I feel like it. So when I list my posts with a simple <code>ls</code> command, I
would like them to be sorted according to their name (which make sense to me)
rather than their date.</p>
<p>In order to achieve that, I take advantage of the fact the Jekyll ignores the
file that don't have the proper naming. So I just name my files as I want,
them add a symlink with the proper name.</p>
<div class="highlight"><pre><span></span><code>mv _drafts/my-post.md _posts/my-post.md
ln -s my-post.md _posts/2015-11-18-my-post.md
</code></pre></div>
<p>That's all, it's a simple trick, but I do appreciate that Jekyll let me do that.</p>
<h2>Jekyllkenny</h2>
<p>In the end, I started to write my own little script to automate a few things, like
syncing or publishing. Basically, a kind of <a href="http://octopress.org/">Octopress</a> but much
more simple, and taylored just for me.</p>
<p>If your Jekyll workflow meets mine, you can check it out on <a href="https://gitlab.com">Gitlab</a>
and give it a try. Here is the address:</p>
<p><a href="https://gitlab.com/arnaudr/jekyllkenny">https://gitlab.com/arnaudr/jekyllkenny</a></p>Getting Started With Jekyll2015-11-29T00:00:00+00:002015-11-29T00:00:00+00:00Arnaud Rebillouttag:arnaudr.io,2015-11-29:/2015/11/29/getting-started-with-jekyll/<p>Here's a brief guide to install <a href="http://jekyllrb.com/">Jekyll</a>
using the <a href="https://rubygems.org/">Gem</a> package manager,
and some basic configuration.</p>
<p>Here's a brief guide to install <a href="http://jekyllrb.com/">Jekyll</a>
using the <a href="https://rubygems.org/">Gem</a> package manager,
and some basic configuration.</p>
<h2>Introduction</h2>
<p>Before anything, I highly recommend that you take some time and read the
official Jekyll documentation: <a href="http://jekyllrb.com/docs/home/">http://jekyllrb.com/docs/home/</a>.
It's damn well written, trust me! It's so good that I must admit I stole pretty much everything
from them. Yes... This guide is mostly a quick summary of the official doc.
I go even further and quote their "types & hints" word for word.
So why do you keep on reading this post?</p>
<p>Probably because you're like me, you'd like to be finished before you're started. The less you read, the better.
Well it's up to you, but it won't be long until you HAVE TO go through the doc anyway, remember my word.
Jekyll is nice and all, but it's a blend of different technologies, and it's not easy to get the whole picture.
It's quick to get started, but as soon as you start customizing the thing, it's not easy anymore.
And if the web technologies are not the kind of water in which you usualy swim (like me), it's gonna be painful...</p>
<p>Alright, one last word before we start, why Jekyll? Why not Octopress or Ghost?</p>
<p><a href="http://octopress.org">Octopress</a> is "<em>an obsessively designed toolkit for Jekyll blogging</em>" (just quoting them).
My definition would be <em>a Jekyll helper</em>. If you want to go with it, here's my warnings:</p>
<ul>
<li>You will have to deal with Jekyll anyway, Octopress doesn't hide it all over. And by dealing with Jekyll,
you deal with the various languages and technologies it uses, such as
<a href="https://en.wikipedia.org/wiki/HTML">html</a>,
<a href="https://en.wikipedia.org/wiki/Cascading_Style_Sheets">css</a>,
<a href="http://sass-lang.com/">sass</a>,
<a href="http://yaml.org/">yaml</a>,
<a href="http://liquidmarkup.org">liquid</a>...
It's already complicated, you see. By using Octopress, you add one more piece to the puzzle.</li>
<li>Also, cross your fingers that Octopress is actively maintained, because Jekyll IS very active and moving fast.</li>
<li>BTW, the current Octopress version is 3. Try to remember that, OK? Most of the doc you will find on the web refers
to the version 2, it will only lead you astray...</li>
</ul>
<p>When I started this blog, I started with Octopress. But the more I used it, the more I felt it wasn't helping anything.
So in the end I dropped it, and started again from scratch, pure Jekyll this time. But maybe Octopress is not for me,
that's all. If you want to go with Octopress, here's the best tuto I found:
<a href="http://www.610yesnolovely.org/2014/12/31/setting-up-a-free-blog.html">Setting up a free blog</a>.</p>
<p>So what about <a href="https://ghost.org/">Ghost</a> now? Ghost is a blogging platform written in Javascript.
It has NOTHING TO DO with Jekyll, it's completely different from a technical point-of-view.
The only common point I can see between both is that you will write your blog using the
<a href="http://daringfireball.net/projects/markdown/">Markdown</a> language (but other markup languages are available though).
Ghost is good if you want to write your blog the <a href="https://wordpress.org/">WordPress</a> way (that is, using a web browser).</p>
<p>Jekyll is good for any of the following reasons:</p>
<ul>
<li>if you want write your blog using your usual text editor (I'm writing with <a href="http://www.vim.org/">Vim</a> at the moment).</li>
<li>if you want your blog to be very lightweight, clean and simple.</li>
<li>if you don't want your content to be stored in a database. Instead, each post is a markdown file.</li>
<li>if you like to make your hands dirty ;)</li>
</ul>
<h2>Jekyll installation</h2>
<p>Alright, after all this talking, let's come to the point. Installing Jekyll is straight-forward.
If you want to have the latest version, you should not install it through your package manager, but use Gem instead.
Jekyll is written in <a href="https://www.ruby-lang.org/">Ruby</a>, and <a href="https://rubygems.org/">Gem</a> is the Ruby Package Manager.</p>
<div class="highlight"><pre><span></span><code>apt-get install ruby
gem install jekyll
</code></pre></div>
<p>That's all :)</p>
<p>In a near future, when you will want to update Jekyll, you will type something like that.</p>
<div class="highlight"><pre><span></span><code>gem update
</code></pre></div>
<p>And when things will break for no apparent reason, you will cross your fingers and invoke the following spell.</p>
<div class="highlight"><pre><span></span><code>gem cleanup
</code></pre></div>
<h2>Creating a new blog</h2>
<p>This is again very easy.</p>
<div class="highlight"><pre><span></span><code>jekyll new blog
cd blog
jekyll serve
</code></pre></div>
<p>Now switch to your web browser. Jekyll comes with a built-in development server, perfect for preview.
By defaut, it serves on the local port 4000, so just enter this address in your web browser, and witness the miracle:</p>
<div class="highlight"><pre><span></span><code>http://localhost:4000
</code></pre></div>
<p>That's it, your blog is already ready! What's cool is that the server watches most of the input files,
so when you edit a post, you can just hit F5 in your web browser to see the changes.</p>
<p>By default, Jekyll only serves the localhost. So if you're running it on a remote machine, you need to tell it
to serve everyone with the <code>--host</code> option. The command looks like that:</p>
<div class="highlight"><pre><span></span><code>jekyll serve --host=0.0.0.0
</code></pre></div>
<h2>Playing with Dr. Jekyll</h2>
<p>Here are some of the things that you may try to get acquainted with Jekyll:</p>
<ul>
<li>Edit the file <code>_config.yml</code>. First warning here (look carefully at the comment at the beginning of the file):</li>
</ul>
<blockquote>
<p>For technical reasons, this file is <em>NOT</em> reloaded automatically when you use
'jekyll serve'. If you change this file, please restart the server process.</p>
</blockquote>
<p>You've been warned. Want another warning about this file?</p>
<blockquote>
<p>Do not use tabs in configuration files
This will either lead to parsing errors, or Jekyll will revert to the default settings. Use spaces instead.</p>
</blockquote>
<p>Why's that? Well, you see the extension of the file, <code>.yml</code>? It means that the file is written with the Yaml language.
And tabs are not allowed in Yaml.</p>
<ul>
<li>Write your first post. It happens in the <code>_posts</code> directory. But wait, beware of the filename!</li>
</ul>
<blockquote>
<p>How you name files in this folder is important.
Jekyll requires blog post files to be named according to the following format:</p>
<p>YEAR-MONTH-DAY-title.MARKUP</p>
</blockquote>
<p>If you don't do that, the file is simply ignored by Jekyll. Also, don't forget to start your
post with a Yaml Front Matter. Have a look at the example post provided by Jekyll and you should be fine.</p>
<ul>
<li>
<p>Add some defaults in <code>_config.yml</code>. For example, if you're the only writter of your blog, and you don't want
to fill the <code>author</code> field at the beginning of each post, you can add a default value for that. It looks like that:</p>
<h1>Defaults settings</h1>
<p>defaults:
-
scope:
path: ""
values:
author: "Your name"</p>
</li>
</ul>
<h2>When Jekyll becomes Mr. Hyde</h2>
<p>It's been so easy that you certainly want to go further, start to customize the css, install a theme, maybe install a
plugin, who knows... Well I won't go any further with you. It won't be easy anymore. I mean, for people like me who
are not dealing with the web development daily. But hopefully you know better than me, and you will enjoy the power
and flexibility that Jekyll provides... Good luck!</p>