pablobmhttps://blog.pablobm.com2024-02-16T00:00:00+00:00Pablo Brasero Morenohttps://www.pablobm.comRuby: `SyntaxError: void value expression`https://blog.pablobm.com/2024/02/16/ruby-void-value-expression/2024-02-16T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>I recently came across this error in a Ruby project:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>SyntaxError: foo.rb:123: void value expression
</pre></div>
</div>
</div>
<p>After some digging and simplifying, I isolated the issue to something like
the following:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">def</span> <span class="function">void_assignment</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> a =
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="keyword">begin</span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="keyword">return</span> <span class="integer">1</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>The method includes an assignment, but it is never realised: the code hits
<code>return</code> and the execution ends, leaving the <code>a</code> sort of "dangling", not
receiving its promised value. Ruby does not like this!</p>
<h2 id="tldr-fix">TLDR fix</h2>
<p>Remove the useless <code>return</code>:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">def</span> <span class="function">void_assignment</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> a =
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="keyword">begin</span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="integer">1</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>(Sure, there is the question of why having an assignment at all, but this
is just an extremely simplified example).</p>
<h2 id="what-is-a-void-value-expression">What is a "void value expression"?</h2>
<p>Ruby says <code>void value expression</code> because a <code>return</code>… does not return
anything, counterintuitively. Sure, it "returns" a value to the code that
called the current method, but it does not return anything within the method
where it lives. In other words, <strong>a <code>return</code> expression resolves to a
<em>void value</em></strong>.</p>
<p>So if you have this:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>a = <span class="keyword">return</span> <span class="integer">1</span>
</pre></div>
</div>
</div>
<p>Then <code>a</code> never gets a value: it is "void". Not even <code>nil</code>, which would be a
valid value in Ruby.</p>
<p>Sometimes the parser detects this in advance of running the code, raising
a syntax error and forcing you to fix it. It is a bit strange, because other
times it cannot detect it, runs the code normally, and there are no runtime
errors or anything. I am sure there is a good reason for declaring it an
error, but I do not know it.</p>
<h2 id="variations">Variations</h2>
<p>This error appears in other instances where you have a <code>return</code> and Ruby
expects a value. For example:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">return</span> <span class="predefined-constant">true</span> <span class="keyword">and</span> <span class="predefined-constant">false</span>
</pre></div>
</div>
</div>
<p>This may look like it should be equivalent to <code>return false</code>, but the word
<code>and</code> is subtly different from the operator <code>&&</code>. It resolves later than
<code>return</code>, while <code>&&</code> resolves before <code>return</code>. We say that <code>and</code> has "lower
precedence" than <code>return</code> and <code>&&</code> has "higher precedence".</p>
<p>You may remember, possibly from secondary school, how multiplications <code>×</code> had
to be calculated before additions <code>+</code>. This is what "precedence" is about.
Therefore the above code is equivalent to <code>(return true) and false</code>, where the
parentheses make this precedence clearer, highlighting how <code>false</code> is never
reached.</p>
<h2 id="other-ruby-implementations">Other Ruby implementations</h2>
<p>More interestingly, different Ruby implementations handle this differently.
My team actually came across this issue working with JRuby instead of MRI.
Have a look at the following method:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">def</span> <span class="function">void_assignment_with_if</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> a =
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="keyword">begin</span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="keyword">if</span> <span class="predefined-constant">true</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="keyword">return</span> <span class="integer">1</span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>The code above parses and runs with MRI, but fails with the syntax error in
JRuby. My guess is that JRuby's JIT compiler looks more closely into that sort
of situation, detecting that the <code>if</code> is masking a potential void value
expression.</p>
<p>But the following works!</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="keyword">def</span> <span class="function">void_assignment_with_if</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> a =
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="keyword">begin</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> <span class="keyword">if</span> <span class="predefined-constant">true</span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="keyword">return</span> <span class="integer">1</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="keyword">end</span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> puts <span class="string"><span class="delimiter">"</span><span class="content">AFTER</span><span class="delimiter">"</span></span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="keyword">end</span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>I'm sure there's a perfectly reasonable explanation for all this, but I am
not looking any deeper :-)</p>
<h2 id="why-doesnt-my-linter-detect-this">Why doesn't my linter detect this?</h2>
<p>It surprised me that <a href="https://github.com/rubocop/rubocop">RuboCop</a>
(and by extension <a href="https://github.com/standardrb/standard">StandardRB</a>) does
not flag this as an issue. If at least this was a "normal" syntax error,
RuboCop would not need a rule to detect it: its parser would probably throw an
error while analysing the file (before showing actual linter warnings) and that
would help. However this is a "post-parse" syntax error, so the parser that
RuboCop uses is not troubled by it.</p>
<p>My impression is that this must be a relatively known scenario for people
who know better than me, like the RuboCop maintainers. However I cannot see
why there would not be a rule for it. Who knows, maybe I came across something
"new".</p>
<p>But the best way to make sure is to ask people who know better. So I went and
did this in the form of a <a href="https://github.com/rubocop/rubocop/pull/12671">tentative PR for
RuboCop</a>.</p>
<p>People are busy and this may take a while to be looked into (I will bump it
a bit after some time), but ultimately I am hoping it will be enlightening,
and perhaps offer a new RuboCop rule to everyone's benefit.</p>
<p>Watch this space for updates!</p>
44https://blog.pablobm.com/2024/01/07/44/2024-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>A happy number. Yay…</li>
<li>Ruthenium.</li>
<li>The United Kingdom, calling.</li>
<li><code>,</code>.</li>
<li>To Tooting.</li>
</ul>
<p>Toot!</p>
“Unable to access CD Drive” on DOSBoxhttps://blog.pablobm.com/2023/04/18/unable-to-access-cd-drive-on-dosbox/2023-04-18T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>If you are seeing the error <code>Unable to access CD Drive</code> when trying to play a game on DOSBox, this guide may be able to help you. These caveats apply:</p>
<ul>
<li>I am referring to DOS games as distributed by <a href="https://www.gog.com/en/game/heroes_of_might_and_magic">GOG.com</a>. I do not know whether this can generalise to other game stores.</li>
<li>My distro is currently Debian 11 (Bullseye). This is typically a good indicator that it should work on other current distributions, but your mileage may vary.</li>
<li>I have been successful with <em>Heroes of Might and Magic</em>, <em>Heroes of Might and Magic 2: Gold</em> and <em>Alone in the Dark</em>. It may work with other games, but there is no guarantee.</li>
</ul>
<p>So with that out of the way, let's proceed. I will use <em>Heroes of Might and Magic</em> as an example.</p>
<h2 id="install-the-game-with-wine">Install the game with Wine</h2>
<p>First, you will have to download the "offline" version of the game, as available on your library on GOG.com after purchase:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/gog-offline-download.png" role="img" alt="Download dialog on GOG.com" />
</div>
</figure>
<p>You should now have a file called something like <code>setup_heroes_of_might_and_magic_1.2_(1.1)_(33754).exe</code>. You can install it using Wine:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ wine setup_heroes_of_might_and_magic_1.2_\(1.1\)_\(33754\).exe
</pre></div>
</div>
</div>
<p>This will bring up a graphical wizard tool that should install the game. At the end there will be a "Launch" button, but unfortunately it will not work. That is not a problem and we will get to actually running the game in a moment:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/gog-installer-completed.png" role="img" alt="Last step of the installer wizard" />
</div>
</figure>
<p>After the installer has run, you should be able to find the installed files in your Wine directory:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ find $HOME/.wine/ | grep -i heroes
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$HOME/.wine/drive_c/GOG Games/HoMM/HEROES.EXE
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>$HOME/.wine/drive_c/GOG Games/HoMM/HEROES.CFG
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>(...many files listed...)
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>$HOME/.wine/drive_c/ProgramData/Microsoft/Windows/Start Menu/Programs/GOG.com/Heroes of Might and Magic/Documents/Manual.lnk
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>$HOME/.wine/drive_c/ProgramData/Microsoft/Windows/Start Menu/Programs/GOG.com/Heroes of Might and Magic/Documents/Support.url
</pre></div>
</div>
</div>
<h2 id="situation-after-install">Situation after install</h2>
<p>If you try running the game with Wine or DOSBox (providing the path to <code>HEROES.EXE</code>) it will not work either. Wine will tell you that the game <code>is a DOS application, you need to install DOSBox</code>, while DOSBox will tell you <code>Unable to access CD Drive</code>.</p>
<p>OK, so let's get that CD Drive going now. The installation includes an ISO image of the CD, so we need to tell DOSBox about it. The file has the <code>.gog</code> extension and it can be found among the installed files:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ ls -l $HOME/.wine/drive_c/GOG\ Games/HoMM/homm1.gog
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>-rw-r--r-- 1 anjali anjali 457375744 Apr 1 22:25 "$HOME/.wine/drive_c/GOG Games/HoMM/homm1.gog"
</pre></div>
</div>
</div>
<p>(Note: in some cases, the file you need is not the one with the <code>.gog</code> extension. More detail below.)</p>
<p>This is the process:</p>
<ol>
<li>Open DOSBox on the game directory.</li>
<li>Mount the ISO image as a CD-ROM drive.</li>
<li>Run the game.</li>
</ol>
<p>So let's do just that.</p>
<h2 id="actually-running-the-game">Actually running the game</h2>
<p>First run DOSBox on the game directory:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ dosbox $HOME/.wine/drive_c/GOG\ Games/HoMM
</pre></div>
</div>
</div>
<p>By passing the game directory as argument, we get the <code>C:</code> drive mounted there. So in the example above, directory <code>C:\</code> within DOSBox is the same as folder <code>$HOME/.wine/drive_c/GOG\ Games/HoMM</code> in my Linux machine. This will mean fewer keystrokes for us in the next steps, as well as the safety of the DOS software not being allowed access all of our hard drive.</p>
<p>Now we can use the tool <code>imgmount</code> within DOSBox to mount the ISO image as drive <code>D:</code>, like so:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>C:\>imgmount d homm1.gog -t iso
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>MSCDEX installed.
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>Drive D is mounted as $HOME/.wine/drive_c/GOG Games/HoMM/homm1.gog
</pre></div>
</div>
</div>
<p>If that worked correctly and you got similar output, you should now be able to run the game from within DOSBox:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>C:\>HEROES.EXE
</pre></div>
</div>
</div>
<p>And the game should now launch succesfully.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/homm1-splash-screen.png" role="img" alt="Splash screen for Heroes of Might and Magic" />
</div>
</figure>
<h2 id="a-variation-different-directories-and-the-ins-file">A variation: different directories and the <code>.ins</code> file</h2>
<p>I was able to run <em>Alone in the Dark</em> with a couple of variations on this method.</p>
<p>An initial but minor detail is that this game is bundled with files split into two directories: <code>INDARK</code> and <code>JACK</code>, corresponding to the game proper and the bonus spin-off <a href="https://aloneinthedark.fandom.com/wiki/Jack_In_The_Dark"><em>Jack in the Dark</em></a>. However the ISO file is still in the root directory:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ ls -l
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>total 355M
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>drwxr-xr-x 2 bakithi bakithi 4.0K Apr 7 10:00 INDARK
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>drwxr-xr-x 2 bakithi bakithi 4.0K Apr 7 10:00 JACK
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>(...some other files and directories...)
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>-rw-r--r-- 1 bakithi bakithi 338M Apr 7 10:00 GAME.GOG
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>-rw-r--r-- 1 bakithi bakithi 793 Apr 7 10:00 GAME.INS
</pre></div>
</div>
</div>
<p>So when we run the game, we will need to <code>cd</code> into the <code>INDARK</code> directory and then actually run <code>INDARK.EXE</code>.</p>
<p>A more important detail is that mounting the ISO image <code>GAME.GOG</code> is not enough. Instead it is the file <code>GAME.INS</code> that has to be provided to <code>imgmount</code>. This file appears to be a sort of playlist that points to <code>GAME.GOG</code> while also providing info about audio tracks in the CD.</p>
<p>So with these two details in mind, the game can be run as follows. Start by running DOSBox on the game directory:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ dosbox $HOME/.wine/drive_c/GOG\ Games/Alone\ in\ the\ Dark
</pre></div>
</div>
</div>
<p>Then mount the ISO image, but via the <code>GAME.INS</code> file:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>C:\>imgmount d GAME.INS -t iso
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>MSCDEX installed.
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>Drive D is mounted as $HOME/.wine/drive_c/GOG Games/Alone in the Dark/GAME.INS
</pre></div>
</div>
</div>
<p>Finally, we go into the <code>INDARK</code> directory and run the game:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>C:\>cd INDARK
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>C:\INDARK>INDARK
</pre></div>
</div>
</div>
<p>And that should land you on the mansion of horrors.</p>
<p><strong>However</strong> note that the presence of a <code>.ins</code> file is not guarantee that this is the correct file to use. In the case of <em>Heroes of Might and Magic 2: Gold</em>, the file is present but I could not get the game to work correctly if I used the <code>.ins</code> file instead of the <code>.gog</code> file. Go figure.</p>
<h2 id="taking-a-shortcut">Taking a shortcut</h2>
<p>To avoid running three commands every time you want to play the game, you can use the argument <code>-c</code> to DOSBox, which allows you to run it with prepared commands ahead of the target program. In my example of <em>Heroes of Might and Magic</em>, it would be like this:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ dosbox "$HOME/.wine/drive_c/GOG Games/HoMM/HEROES.EXE" -c "imgmount d \"$HOME/.wine/drive_c/GOG Games/HoMM/homm1.gog\" -t iso"
</pre></div>
</div>
</div>
<p>And this is a variation for the case of <em>Alone in the Dark</em>:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ dosbox "$HOME/.wine/drive_c/GOG Games/Alone in the Dark/INDARK/INDARK.EXE" -c "imgmount d \"$HOME/.wine/drive_c/GOG Games/Alone in the Dark/GAME.INS\" -t iso"
</pre></div>
</div>
</div>
<p>Those lines are a mouthful, but you can use them to create shortcuts and run the games with short commands. Watch out for the spaces, escaping, etc!</p>
<h2 id="going-further">Going further</h2>
<p>The GOG install also provides configuration files for DOSBox, which look like this:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ ls -l *.conf
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>-rw-r--r-- 1 calle calle 281 Apr 1 22:25 dosboxHOMM1_client.conf
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>-rw-r--r-- 1 calle calle 10869 Apr 1 22:25 dosboxHOMM1.conf
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>-rw-r--r-- 1 calle calle 273 Apr 1 22:25 dosboxHOMM1_server.conf
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>-rw-r--r-- 1 calle calle 1010 Apr 1 22:25 dosboxHOMM1_single.conf
</pre></div>
</div>
</div>
<p>These include launchers for several situations, as well as tweaks for DOSBox such as default fullscreen mode, clock speed settings, etc. I have been experimenting with them a bit, and I have not been able to make them work as provided. My impression is that they have two problems:</p>
<ul>
<li>They assume a Windows host (instead of Linux), so they reference paths with backslashes (eg: <code>..\GAME.INS</code>) in contexts where a host path is required. I was able to make them work by changing those to forward slashes (eg: <code>../GAME.INS</code>).</li>
<li>They reference at least one feature (mount overlays) present in DOSBox v0.75, which is actually from the <a href="https://dosbox-staging.github.io/downloads/release-notes/0.75.0/">non-official fork</a> that GOG distribute with the game.</li>
</ul>
<p>But these files are not really necessary and I have not needed them to play the games. Still they may be interesting to you, so it is worth being aware of them.</p>
<h2 id="references">References</h2>
<p>I pieced this guide together after reading articles on the topic from <a href="https://www.freezenet.ca/guides/compatibility-and-emulation/how-to-play-a-16-bit-dos-games-that-require-a-cd-rom/">FreezeNet</a>, <a href="https://www.dosgamers.com/dos/dosbox-dos-emulator/mounting">DOSGamers</a>, the <a href="https://www.dosbox.com/wiki/GAMES:Heroes_of_Might_and_Magic">DOSBox Wiki</a>, and a forum post that I cannot find now (!) but was particularly useful.</p>
<p>Perhaps there's an easier way to do all this and I missed it, as I'm not a hard-core gamer. If you find it, please hit me up with it on the socials!</p>
43https://blog.pablobm.com/2023/01/07/43/2023-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>A prime number, twins with 41.</li>
<li>Techtenium.</li>
<li><a href="https://en.wikipedia.org/wiki/Licor_43">That stuff</a> that I drank too much of when I was a young whippersnapper.</li>
<li><code>+</code>.</li>
<li>To Friern Barnet.</li>
</ul>
<p>And beyond.</p>
Disable HSTS on Dokkuhttps://blog.pablobm.com/2022/11/12/disable-hsts-on-dokku/2022-11-12T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>HTTP Strict Transport Security, or in short <strong>HSTS</strong>, is a mechanism that forces web browsers to use secure connections (HTTPS) in order to access websites. It is a good idea and websites should use it. However there are some very niche cases where you do not want it. I manage one of those cases.</p>
<p>I run <a href="http://referrer-policy.info/">http://referrer-policy.info</a>, a website that allows you to test how your browser implements the <code>Referrer-Policy</code> header. It is a long story (see my article <a href="https://blog.pablobm.com/2020/04/14/referrer-policy.html">Notes on implementing a Referrer Policy</a>), but that website needs to be able to run both HTTP and HTTPS. If HSTS was active, it would disallow HTTP connections, and the site would lose utility. Therefore it has to be disabled.</p>
<h2 id="hsts-on-dokku">HSTS on Dokku</h2>
<p>I host the website in question on a VPS running <a href="https://dokku.com/">Dokku</a>. Dokku is configured by default to use HSTS, which again is a great idea in general, but it is not good for my special case.</p>
<p>Fortunately, this can be overriden. On Dokku, disabling HSTS involves two steps:</p>
<ol>
<li>Disabling the HSTS header.</li>
<li>Removing redirects to HTTPS.</li>
</ol>
<h2 id="do-you-need-https-in-the-first-place">Do you need HTTPS in the first place?</h2>
<p>The answer to this question is normally yes, but if that was your case you probably would not be reading this article.</p>
<p>If your site can run using HTTP only, simply do not enable HTTPS for your app, and Dokku will not force you to use it. HSTS will not be a problem here.</p>
<p>But if you do want your website to work both over HTTP and HTTPS, then you need to disable HSTS explicitly, which I will cover next.</p>
<h2 id="disabling-the-hsts-header">Disabling the HSTS header</h2>
<p>If you search for HSTS through the documentation, you will find a mention on the page about the <a href="https://dokku.com/docs/networking/proxies/nginx/#hsts-header">Nginx Proxy</a>.</p>
<p>Before following those instructions, let's first check that the HSTS header is indeed there, using <code>curl</code>:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>$ curl -i http://my-site.example/
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>HTTP/1.1 301 Moved Permanently
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>Server: nginx
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>Date: Thu, 20 Oct 2022 13:14:41 GMT
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>Content-Type: text/html
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>Content-Length: 162
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>Connection: keep-alive
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>Location: https://my-site.example:443/
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>Strict-Transport-Security: max-age=15724800; includeSubdomains
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span><html>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span><head><title>301 Moved Permanently</title></head>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span><body>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span><center><h1>301 Moved Permanently</h1></center>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><hr><center>nginx</center>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span></body>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span></html>
</pre></div>
</div>
</div>
<p>Yes, it is there. The header <code>Strict-Transport-Security: max-age=15724800; includeSubdomains</code> is the one. Let's remove it now:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ dokku nginx:set my-app hsts-max-age 0
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ dokku proxy:build-config my-app
</pre></div>
</div>
</div>
<p>Note that I am setting <code>hsts-max-age 0</code> instead of simply <code>hsts false</code>. This means that the HSTS header will still be there, but it will show a <code>max-age</code> of <code>0</code>. This tells browsers that the HSTS setting has expired and they should stop fulfilling it. By doing it this way, we get to "reeducate" browsers that accidentally saw the HSTS header before it was removed. It is not a bullet-proof solution though, and it may remain in browser caches for a bit.</p>
<p>Do not forget running the command <code>proxy:build-config</code>. Until recently, it was a bit hidden in the documentation and I missed it the first time. Fortunately <a href="https://github.com/dokku/dokku/pull/5430">this has been corrected now</a>.</p>
<p>But anyway, another hit with <code>curl</code> should show that the header now appears with a <code>max-age=0</code> parameter:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>$ curl -i http://my-site.example/
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>HTTP/1.1 301 Moved Permanently
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>Server: nginx
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>Date: Thu, 20 Oct 2022 13:14:41 GMT
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>Content-Type: text/html
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>Content-Length: 162
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>Connection: keep-alive
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>Location: https://my-site.example:443/
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>Strict-Transport-Security: max-age=0; includeSubdomains
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span><html>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span><head><title>301 Moved Permanently</title></head>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span><body>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span><center><h1>301 Moved Permanently</h1></center>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><hr><center>nginx</center>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span></body>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span></html>
</pre></div>
</div>
</div>
<p>But if you visit the app with your browser, you will still land on the HTTPS version. What gives?</p>
<h2 id="a-note-about-hsts-and-redirects">A note about HSTS and redirects</h2>
<p>When the browser sees the HSTS header, it makes a note to not use the HTTP site in the future. Any time you type an HTTP URL for the site, or you follow a link to it, the browser automatically will replace <code>http://</code> with <code>https://</code>, without ever hitting the HTTP site again. It is a client-side measure enforced by the browser.</p>
<p>A detail of the HSTS standard is that the header is only valid if served over HTTPS. If the browser request was made with HTTP, the header must be ignored (long story, but it is a safety feature). This means that for HSTS to work, we should make sure to redirect anyone who visits the HTTP site, and send them to the HTTPS version. Once working over HTTPS, the browser will see the header and enforce it.</p>
<p>So this redirect is what we need to remove. If you have a look at the <code>curl</code> outputs above, you will see that the HTTP status is <code>301 Moved Permanently</code>, with the header <code>Location: https://my-site.example:443/</code>. Dokku serves this by default when HTTPS is enabled.</p>
<h2 id="removing-the-redirect-to-https">Removing the redirect to HTTPS</h2>
<p>Removing this is a bit trickier. At the moment Dokku does not offer a simple configuration option to remove this redirect, but it does offer access to the Nginx configuration of individual apps, and this is all we need.</p>
<p>On the same page that the Dokku documentation describes HSTS, there is a section on <a href="https://dokku.com/docs/networking/proxies/nginx/#customizing-the-nginx-configuration">Customizing the nginx configuration</a>. It describes how each Dokku app uses an individual Nginx configuration, created from <a href="https://github.com/dokku/dokku/blob/master/plugins/nginx-vhosts/templates/nginx.conf.sigil">a template that can be found at their repo</a>. This template can be downloaded, added to our own apps, and altered to suit our needs. That is what we will do.</p>
<p>Start by downloading the template into your project's repo:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ curl -O https://raw.githubusercontent.com/dokku/dokku/master/plugins/nginx-vhosts/templates/nginx.conf.sigil
</pre></div>
</div>
</div>
<p>(Note that the location of the template may change in future versions. Check the documentation if that happens).</p>
<p>Now you have a <code>nginx.conf.sigil</code> file on your project. It must be on the root directory of the repo, and it will be picked up by Dokku on deployment, replacing the default configuration.</p>
<p>From this file, you need to remove the parts about the redirect. Personally, I prefer to comment them out for easier future reference. The following is a diff of the changes, as extracted from my project:</p>
<div class="language-diff code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="line comment">diff --git a/nginx.conf.sigil b/nginx.conf.sigil</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="line comment">index 21a0d01..2a76711 100644</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="line head"><span class="head">--- </span><span class="filename">a/nginx.conf.sigil</span></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="line head"><span class="head">+++ </span><span class="filename">b/nginx.conf.sigil</span></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="change"><span class="change">@@</span> -11,12 +11,24 <span class="change">@@</span></span> server {
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> {{ if $.NOSSL_SERVER_NAME }}server_name {{ $.NOSSL_SERVER_NAME }}; {{ end }}
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> access_log {{ $.NGINX_ACCESS_LOG_PATH }}{{ if and ($.NGINX_ACCESS_LOG_FORMAT) (ne $.NGINX_ACCESS_LOG_PATH "off") }} {{ $.NGINX_ACCESS_LOG_FORMAT }}{{ end }};
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> error_log {{ $.NGINX_ERROR_LOG_PATH }};
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span><span class="line insert"><span class="insert">+</span></span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span><span class="line insert"><span class="insert">+</span>{{/*</span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span><span class="line insert"><span class="insert">+</span>##### <custom> #####</span>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span><span class="line insert"><span class="insert">+</span>##</span>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span><span class="line insert"><span class="insert">+</span>## Conditional commented out for my-site.example, as we do</span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span><span class="line insert"><span class="insert">+</span>## want to allow both HTTP and HTTPS domains.</span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="line insert"><span class="insert">+</span>##</span>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span><span class="line insert"><span class="insert">+</span>## Also remember to disable HSTS on Dokku's own config as</span>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span><span class="line insert"><span class="insert">+</span>## described at https://dokku.com/docs/networking/proxies/nginx/#hsts-header</span>
<span class="line-numbers"><a href="#n18" name="n18">18</a></span><span class="line insert"><span class="insert">+</span>##</span>
<span class="line-numbers"><a href="#n19" name="n19">19</a></span><span class="line insert"><span class="insert">+</span>##### </custom> #####</span>
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> {{ if (and (eq $listen_port "80") ($.SSL_INUSE)) }}
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> include {{ $.DOKKU_ROOT }}/{{ $.APP }}/nginx.conf.d/*.conf;
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> location / {
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> return 301 https://$host:{{ $.PROXY_SSL_PORT }}$request_uri;
<span class="line-numbers"><a href="#n24" name="n24">24</a></span> }
<span class="line-numbers"><a href="#n25" name="n25">25</a></span> {{ else }}
<span class="line-numbers"><a href="#n26" name="n26">26</a></span><span class="line insert"><span class="insert">+</span>*/}}</span>
<span class="line-numbers"><a href="#n27" name="n27">27</a></span> location / {
<span class="line-numbers"><a href="#n28" name="n28">28</a></span>
<span class="line-numbers"><a href="#n29" name="n29">29</a></span> gzip on;
<span class="line-numbers"><strong><a href="#n30" name="n30">30</a></strong></span><span class="change"><span class="change">@@</span> -63,7 +75,7 <span class="change">@@</span></span> server {
<span class="line-numbers"><a href="#n31" name="n31">31</a></span> internal;
<span class="line-numbers"><a href="#n32" name="n32">32</a></span> }
<span class="line-numbers"><a href="#n33" name="n33">33</a></span> include {{ $.DOKKU_ROOT }}/{{ $.APP }}/nginx.conf.d/*.conf;
<span class="line-numbers"><a href="#n34" name="n34">34</a></span><span class="line delete"><span class="delete">-</span>{{<span class="eyecatcher"> end </span>}}</span>
<span class="line-numbers"><a href="#n35" name="n35">35</a></span><span class="line insert"><span class="insert">+</span>{{<span class="eyecatcher">/* end */</span>}}</span>
<span class="line-numbers"><a href="#n36" name="n36">36</a></span> }
<span class="line-numbers"><a href="#n37" name="n37">37</a></span> {{ else if eq $scheme "https"}}
<span class="line-numbers"><a href="#n38" name="n38">38</a></span> server {
</pre></div>
</div>
</div>
<p>If you apply the outlined changes and deploy, the redirect will be gone. Now you should be able to visit the HTTP and HTTPS versions of the app without issue. This may still require some browser caches to expire though, as they may still remember the redirect and apply it, without actually visiting the page.</p>
<h2 id="caveats">Caveats!</h2>
<p>There are two caveats that I have found, where things will not work as expected and your browser will still send you to the HTTPS site. They are not really specific to Dokku, but instead apply to HSTS in general.</p>
<p>The first is with <strong>Firefox in Private Browsing mode</strong> (aka. <em>incognito mode</em>). Since version 91, <a href="https://blog.mozilla.org/security/2021/08/10/firefox-91-introduces-https-by-default-in-private-browsing/">Firefox forces HTTPS in Private Browsing mode</a>. Only if the site does not support HTTPS at all, it will allow using HTTP. But if HTTPS is available, users will be sent there regardless of any configuration you attempt.</p>
<p>The second is with <strong><code>.dev</code> domains</strong>. Both <a href="https://ma.ttias.be/chrome-force-dev-domains-https-via-preloaded-hsts/">Chrome and Firefox force HSTS on <code>.dev</code> domains by default</a>. There is nothing that you can do to prevent this. Just know that you will have to use a different TLD for your app if you need to disable HSTS.</p>
42https://blog.pablobm.com/2022/01/07/42/2022-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>2 × 3 × 7.</li>
<li><a href="https://en.wikipedia.org/wiki/Pronic_number">Pronic</a>, <a href="https://en.wikipedia.org/wiki/Abundant_number">Abundant</a>, <a href="https://en.wikipedia.org/wiki/Sphenic_number">Sphenic</a>, <a href="https://en.wikipedia.org/wiki/42_(number)">etc</a>.</li>
<li>Molybdenum.</li>
<li><code>*</code>.</li>
</ul>
<p>And The Answer to… you know the rest.</p>
Configuring VCR to work with Selenium WebDriverhttps://blog.pablobm.com/2021/10/14/vcr-webdriver-errors/2021-10-14T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>If you are using VCR and Selenium Webdriver in your project, there are a couple of common errors that you might see.</p>
<figure style="max-width: 512px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img style="max-width: 256px;" src="/images/cassette.svg" alt="" />
</div>
<figcaption>
<p class="attribution">
<a href="https://thenounproject.com/maxim221/collection/cassetes/?i=788347">"Cassette"</a> icon by Maxim Kulikov from the Noun Project.
</p>
</figcaption>
</figure>
<h2 id="the-quick-fix">The quick fix</h2>
<p>Before I go into detail, this is the configuration I'm using for VCR on my current Rails project, which fixes the issues described below. Note the comments:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>require <span class="string"><span class="delimiter">"</span><span class="content">vcr</span><span class="delimiter">"</span></span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="constant">VCR</span>.configure <span class="keyword">do</span> |config|
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> config.cassette_library_dir = <span class="string"><span class="delimiter">"</span><span class="content">vcr_cassettes</span><span class="delimiter">"</span></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> config.hook_into <span class="symbol">:webmock</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="comment"># Avoid conflict with Selenium</span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> config.ignore_localhost = <span class="predefined-constant">true</span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="comment"># Allow downloading webdrivers for Selenium</span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> driver_hosts = <span class="constant">Webdrivers</span>::<span class="constant">Common</span>.subclasses.map { |driver| URI(driver.base_url).host }
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="comment"># Downloading the Firefox driver involves a redirect</span>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> driver_hosts += [<span class="string"><span class="delimiter">"</span><span class="content">github-releases.githubusercontent.com</span><span class="delimiter">"</span></span>]
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> config.ignore_hosts(*driver_hosts)
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>Now on to the actual issues.</p>
<h2 id="problem-1-error-talking-to-selenium-webdriver">Problem 1: error talking to Selenium WebDriver</h2>
<p>The first problem manifests as an error that looks a bit like this:</p>
<div class="language-text code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>====================================================
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>An HTTP request has been made that VCR does not know how to handle:
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> GET http://127.0.0.1:9516/something/something
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>There is currently no cassette in use. There are a few ways
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>you can configure VCR to handle this request:
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>[...The error message goes on...]
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>====================================================
</pre></div>
</div>
</div>
<p>The important bit here is the URL. Your code is trying to access something at <code>http://127.0.0.1:9516</code>, most probably while running a test. This is the local address used by Selenium to communicate with the browser in order to control it.</p>
<p>VCR captures all your HTTP requests, including some that you may not realise are taking place and you don't want to capture. This is one such example.</p>
<p>Typically you don't want VCR to capture requests to local addresses, as they are normally used by things that you do want to reach. To avoid capturing these, you can use the following setting with VCR:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span> <span class="comment"># Avoid conflict with Selenium</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> config.ignore_localhost = <span class="predefined-constant">true</span>
</pre></div>
</div>
</div>
<h2 id="problem-2-error-downloading-a-selenium-driver">Problem 2: error downloading a Selenium driver</h2>
<p>This is an example of a related but different problem:</p>
<div class="language-text code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>====================================================
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>An HTTP request has been made that VCR does not know how to handle:
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> GET https://chromedriver.storage.googleapis.com/LATEST_RELEASE_90.0.4430
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>There is currently no cassette in use. There are a few ways
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>you can configure VCR to handle this request:
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>[...etc...]
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>====================================================
</pre></div>
</div>
</div>
<p>In this case, the request is being made behind the scenes by the gem <a href="https://github.com/titusfortner/webdrivers"><code>webdrivers</code></a>. If you are using Ruby on Rails, this gem is installed by default as of version 6 (in older versions it was <a href="https://github.com/flavorjones/chromedriver-helper"><code>chromedriver-helper</code></a>).</p>
<p>This gem downloads drivers that Selenium can use to control your browser. This saves you having to download and install these drivers yourself. Of course since downloading something is an HTTP request, VCR also captures it, causing this error.</p>
<p>To fix this, we can tell VCR to ignore (ie: let through) requests to the host from where it's trying to download the driver. In the example above, you could do the following:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span> config.ignore_request <span class="keyword">do</span> |request|
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="comment"># Allow downloading Chromedriver</span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="keyword">next</span> URI(request.uri).host == <span class="string"><span class="delimiter">"</span><span class="content">chromedriver.storage.googleapis.com</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>This is not all though. In this example I'm assuming that your are using Chrome with Selenium, but you could be using a different browser. For example, if you are using Firefox, you'd see an error like this:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>====================================================
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>An HTTP request has been made that VCR does not know how to handle:
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> GET https://github.com/mozilla/geckodriver/releases/latest
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>There is currently no cassette in use. There are a few ways
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>you can configure VCR to handle this request:
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>[...etc...]
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>====================================================
</pre></div>
</div>
</div>
<p>For each variant of this issue, the URL will be suggestive of a specific browser. For each browser its driver is downloaded from a different location, so our VCR configuration needs to allow for any download URLs that we might need. For Firefox, the following would work. It involves two checks because there's a redirect to take into account:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span> config.ignore_request <span class="keyword">do</span> |request|
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> uri = URI(request.uri)
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="comment"># Don't interfere with the Webdrivers gem downloading GeckoDriver</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="keyword">next</span> <span class="predefined-constant">true</span> <span class="keyword">if</span> uri.to_s =~ <span class="regexp"><span class="delimiter">%r{</span><span class="content">https://github.com/mozilla/geckodriver/releases/</span><span class="delimiter">}</span></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> <span class="keyword">next</span> <span class="predefined-constant">true</span> <span class="keyword">if</span> uri.host == <span class="string"><span class="delimiter">"</span><span class="content">github-releases.githubusercontent.com</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> <span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>If you use more than one browser, you'll need to add matchers for their respective download URLs. If you want a comprehensive configuration that covers many browsers, this can get rather tiring rather soon.</p>
<p>Fortunately this issue was raised with the relevant gem maintainers in 2019 (<a href="https://github.com/titusfortner/webdrivers/issues/109">read the conversation</a>) and a solution was offered which is <a href="https://github.com/titusfortner/webdrivers/wiki/Using-with-VCR-or-WebMock">documented at the Webdrivers repository</a>. My configuration at the top of this article is based on that solution. It assumes that you are using Rails (or at least ActiveSupport), but you can read the GitHub link if that's not the case for you.</p>
<h2 id="but-why-no-cassette-in-use">But why "no cassette in use"?</h2>
<p>There's something else interesting here: why does VCR say that there's "no cassette in use" if our code sets up a cassette? How come that VCR doesn't allow these requests, recording them like any other?</p>
<p>For example, in the second case: since we are using VCR, it would be reasonable to expect that VCR handles the request, allowing the download, and storing the it in a cassette file just like with any other HTTP request.</p>
<p>The downside of this behaviour would be the creation of cassette files that are several megabytes in size, but otherwise it would be consistent with what we expect from VCR. Why doesn't this happen?</p>
<p>The reason for this not to happen is that these requests (be it to control Selenium or to download drivers) take place before your cassette is in operation. For example, if you are using these tools in a Rails system test, the Webdrivers gem performs the download before the test even starts. At that point, your calls to <code>VCR.use_cassette</code> haven't happened yet and there's no cassette in use, just like the error message says.</p>
41https://blog.pablobm.com/2021/01/07/41/2021-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>A prime number, twins with 43.</li>
<li>The sum of the first 6 primes: 2 + 3 + 5 + 7 + 11 + 13.</li>
<li>And also of these: 11 + 13 + 17.</li>
<li>Niobium.</li>
<li>Mozart's last symphony.</li>
</ul>
<p>Hopefully not my last.</p>
How to move commits to the correct branchhttps://blog.pablobm.com/2020/07/23/how-to-move-commits-to-correct-branch/2020-07-23T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>So you made some commits with Git, and then you realised that you were not on the correct branch. Perhaps you made the commits on <code>master</code>, when they were intended for a branch. Or you were on some other unintended branch, it doesn't matter. What to do?</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-rebase.svg" role="img" alt="" />
</div>
</figure>
<p>This is a common question when you are learning the ropes of Git. I hope this guide can get you out of that problem, while helping you understand Git better.</p>
<p>Since this is intended for beginners, I'm going to assume that your setup doesn't have anything out of the ordinary. I expect you'll have something like this:</p>
<ul>
<li>You only have one remote, and it is called <code>origin</code>.</li>
<li>There's only one "main" branch, and it is called <code>master</code>.</li>
</ul>
<h2 id="visualize-the-problem">0. Visualize the problem</h2>
<p>If you are not using a visual tool to work with Git, get one. For macOS, I really recommend <a href="https://git-fork.com/">Fork</a>, which also has a Windows version. For Linux, I have used <a href="https://wiki.gnome.org/Apps/Gitg">gitg</a> and it does the job well enough.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/gitg.png" alt="gitg" />
<img src="/images/fork.png" alt="Fork" srcset="/images/fork.png 1x, /images/fork@2x.png 2x" />
</div>
</figure>
<p>If for some reason you can't use any of these tools, you can use the <code>git log</code> command with the <code>--graph</code> option, which will render an ASCII graph. Here's an example of use that will show the last 20 commits:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>$ git log --all --decorate --oneline --graph -n 20
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>* 5c32b78566 (origin/master, origin/HEAD, master) Merge pull request #40280 from rails/no-rg-dep
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>|\
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>| * 6fca0f31f1 Try calling clock_gettime rather than testing the platform
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>* | 7cc563427d Merge pull request #40281 from kamipo/add_dependency_rexml
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>|\ \
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>| * | c23533ee0b rexml is no longer default gem in Ruby 3.0
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>|/ /
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>* | 94ea7281b8 Use master branch for `listen` and `redis-namespace`
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>* | 8ecb0e22d7 Fix typo s/inherit_from/inherit_all/ [ci skip]
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>* | 20423ef455 Merge pull request #39320 from trevorrjohn/i18n_key
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>|\ \
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>| * | 6fba3c3be0 Allow ActiveModel::Name fields to be overriden
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>* | | fc2a1ed24f Merge pull request #40240 from adrianna-chang-shopify/subscriber-attach-to-with-inherit-option
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>|\ \ \
<span class="line-numbers"><a href="#n16" name="n16">16</a></span>| |_|/
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>|/| |
<span class="line-numbers"><a href="#n18" name="n18">18</a></span>| * | 387aa8c373 Subscriber.attach_to with inherit_all option
<span class="line-numbers"><a href="#n19" name="n19">19</a></span>* | | 19eabad014 Merge pull request #40279 from the-spectator/corrects_require
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span>|\ \ \
<span class="line-numbers"><a href="#n21" name="n21">21</a></span>| * | | b314ab555e Move require `active_support/core_ext/string/filters` to `active_job/log_subscriber` from `active_job/logging`
<span class="line-numbers"><a href="#n22" name="n22">22</a></span>* | | | b6d86add73 Remove unused require
<span class="line-numbers"><a href="#n23" name="n23">23</a></span>|/ / /
<span class="line-numbers"><a href="#n24" name="n24">24</a></span>* | | 5abe09bba7 Merge pull request #40213 from dbussink/allow-setting-digest-class
<span class="line-numbers"><a href="#n25" name="n25">25</a></span>|\ \ \
<span class="line-numbers"><a href="#n26" name="n26">26</a></span>| * | | 0f6c9bc786 Add an ActiveSupport option to allow setting a digest class
<span class="line-numbers"><a href="#n27" name="n27">27</a></span>| | |/
<span class="line-numbers"><a href="#n28" name="n28">28</a></span>| |/|
<span class="line-numbers"><a href="#n29" name="n29">29</a></span>* | | 33cd36b898 Merge pull request #40278 from kddeisz/gitattributes
<span class="line-numbers"><strong><a href="#n30" name="n30">30</a></strong></span>|\ \ \
<span class="line-numbers"><a href="#n31" name="n31">31</a></span>| |/ /
<span class="line-numbers"><a href="#n32" name="n32">32</a></span>|/| |
<span class="line-numbers"><a href="#n33" name="n33">33</a></span>| * | da39688e26 Create a gitattributes file for templated apps
<span class="line-numbers"><a href="#n34" name="n34">34</a></span>|/ /
<span class="line-numbers"><a href="#n35" name="n35">35</a></span>* | fb852668df Merge pull request #40276 from ghiculescu/fix-bad-formatting
<span class="line-numbers"><a href="#n36" name="n36">36</a></span>|\ \
<span class="line-numbers"><a href="#n37" name="n37">37</a></span>| * | 3457d227c1 Fix formatting error in concerns doc
<span class="line-numbers"><a href="#n38" name="n38">38</a></span>|/ /
<span class="line-numbers"><a href="#n39" name="n39">39</a></span>* | f8dd68e131 Merge pull request #40275 from composerinteralia/approvals
<span class="line-numbers"><strong><a href="#n40" name="n40">40</a></strong></span>|\ \
</pre></div>
</div>
</div>
<p>Whichever tool you go for, run it now and have a look at the situation.</p>
<h2 id="did-you-push-the-changes">1. Did you push the changes?</h2>
<p>If you pushed the changes to <code>master</code> AND you work in a team, unfortunately there isn't a good solution.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-bad-origin-master.svg" role="img" alt="" />
</div>
</figure>
<p>I don't recommend that you do anything that hides the mistake. Instead, <strong>tell your team immediately</strong>. These changes may have repercussions in your project. Or not. It depends. In any case, it's only human to do this, and more common than you may think. I have done it! This is why repositories often have specific settings that block direct pushes to <code>master</code>. It's too easy to do, and humans will invariably do it by mistake sooner or later. Perhaps your team should think of enabling this setting if available.</p>
<p>If you pushed to a branch that other members of the team are also working on, also tell them. You all may agree that it can be "fixed". Or perhaps it's simpler to create commits that revert the changes, leaving things to be cleaned up later. Or never. It depends, and it's fine.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-bad-origin-branch.svg" role="img" alt="" />
</div>
</figure>
<p>If you pushed to a branch (including <code>master</code>) that only you are working on, then you can use this guide to rewrite history and pretend that this never happened.</p>
<p>If you didn't push, that's the best scenario. Use this guide to rewrite history. Nobody needs to know.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-bad-branch.svg" role="img" alt="" />
</div>
</figure>
<p>Finally: don't panic. Projects have ugly Git histories anyway. People make mistakes all the time. Still, I have spoken to people who felt very bad about making this mistake, and they thought that it would reflect badly on their skill in the eyes of employers. Don't worry: it won't. How you handle the situation is what counts here.</p>
<p>And by the way: nobody looks at whether the commit history of your pet GitHub project looks pretty.</p>
<h2 id="label-your-branch">2. Label your branch</h2>
<p>First, you made commits to the wrong branch. What should the branch have been called? Put a label (a branch name) on it now. Identify the commit with all the changes that you want to move, and create a new branch with a name that describes what it is:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git branch intended-branch
</pre></div>
</div>
</div>
<p>If you are used to creating branches with <code>git checkout -b</code>, you might find <code>git branch</code> strange. It does the same thing, but doesn't "move" you to the newly created branch.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-label-branch.svg" role="img" alt="" />
</div>
</figure>
<h2 id="reset-the-accidented-branch-to-its-desired-state">3. Reset the accidented branch to its desired state</h2>
<p>Now let's fix the accidented branch.
You had a branch and you added the wrong commits to it. Time to remove those commits from the branch. To do this, you <strong>reset</strong> the branch. This moves the branch label to a different place in the tree.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-reset.svg" role="img" alt="" />
</div>
</figure>
<p>First, make sure that you are on the branch that you want to reset:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git checkout accidentally-extended
</pre></div>
</div>
</div>
<p>Second, find out the location where you want to move it. Use your visual tool to find something that marks the commit where you want to move the label. This marker will be another branch label, or the SHA of the commit at that location.</p>
<p>When you have this location marker, you can use the command <code>git reset</code> to move the branch label where you currently are to that other location.</p>
<p>For example, if you didn't push the changes, you may have a remote branch label marking the place:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git reset --hard origin/accidentally-extended
</pre></div>
</div>
</div>
<p>Or if you go by the SHA of the destination commit:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git reset --hard a1b2c3d4e5f6
</pre></div>
</div>
</div>
<p>You may wonder what the option <code>--hard</code> stands for. Don't worry for now, as we don't want to make this too complicated. You simply need it here. I'll leave that one for you to research.</p>
<h2 id="push-the-branch-back-in-place">4. Push the branch back in place</h2>
<p>If you had commited to changes to the branch, and together with your team you decided that it's safe to rewrite the remote history, then you'll want to do this. Skip this step if you didn't push those changes.</p>
<p>Here you fix the remote branch by pushing the changes.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-push-force.svg" role="img" alt="" />
</div>
</figure>
<p>Assuming that your local branch is correctly in place now, push it. As before, you start by making sure that you are on the branch that you are going to push:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git checkout accidentally-extended
</pre></div>
</div>
</div>
<p>Now the actual push. You'll need the option <code>--force-with-lease</code> to do this, as you'll be changing the remote history and this is normally not desired.</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git push --force-with-lease origin accidentally-extended
</pre></div>
</div>
</div>
<p>Why <code>--force-with-lease</code> and not <code>--force</code>? The short explanation is that <code>--force-with-lease</code> makes sure that no new changes have happened in <code>origin</code> that you are not aware of. Again, there's only so much complication that we want to add here, but I'll just recommend that you always use <code>--force-with-lease</code> instead of <code>--force</code>.</p>
<h2 id="secure-your-changes-to-prevent-losing-them">5. Secure your changes to prevent losing them</h2>
<p>We are about to make a destructive change. Things can go wrong now. To prevent any issues, put a new label in your changes to avoid losing them. This is: create a new branch alongside the one you just created a couple of steps ago. This one will stay here, while the other one moves. It will work as a "backup copy" of sorts.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-mark-backup.svg" role="img" alt="" />
</div>
</figure>
<p>Again make sure that you are on the right location:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git checkout intended-branch
</pre></div>
</div>
</div>
<p>And now create a new branch label with a descriptive name:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git branch backup
</pre></div>
</div>
</div>
<h2 id="rebase-the-commits-onto-the-right-place">6. Rebase the commits onto the right place</h2>
<p>We will "cut" the branch with the commits, and we'll "transplant" it to a new location. It'll be a bit like gardening.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-rebase.svg" role="img" alt="" />
</div>
</figure>
<p>To do this, you need to know:</p>
<ol>
<li>The name of the branch to move. Eg: <code>intended-branch</code>.</li>
<li>The location from where it stems; typically a branch label or a SHA. Eg: <code>accidentally-extended</code>.</li>
<li>The location where we will transplant it; also a branch label or a SHA. Eg: <code>master</code>.</li>
</ol>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-ready-to-rebase.svg" role="img" alt="" />
</div>
</figure>
<p>We will use the <code>git rebase</code> command for this. This is how to use it:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git rebase --onto master accidentally-extended intended-branch
</pre></div>
</div>
</div>
<p>Note that <code>accidentally-extended</code> points to the commit <strong>right before</strong> the first commit that you want to transplant. In other words: you don't want to move the commit at <code>accidentally-extended</code>; you want to move the commits after it.</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/git-after-rebase.svg" role="img" alt="" />
</div>
</figure>
<p>If this command resolves cleanly: congratulations! You have rebased the branch. If it did not rebase cleanly, and it complains about conflicts… that can potentially get hairier.</p>
<p>Conflicts happen when the changes that you are trying to add (in this case via a rebase) are in conflict with other changes that have been added to the target branch in the mean time. Git can't tell automatically which changes should "win", and therefore relies on you to do it.</p>
<p>If you know how to resolve conflicts, you can try. If anything goes wrong, remember that you have that <code>backup</code> label that still has your changes. They haven't been lost.</p>
<p>If you tried to rebase but then didn't feel comfortable resolving the conflicts, you can abort the rebase with <code>git rebase --abort</code>. It will bring things back to where they were before the initial <code>git rebase</code> command.</p>
<p>Unfortunately, resolving conflicts is a whole new topic, and therefore it's out of scope for this guide.</p>
<h2 id="push-your-rebased-branch">7. Push your rebased branch</h2>
<p>If the rebase went well, have a look again with your visual tool of choice. Do things look correct now? If they do, don't forget to push your newly rebased branch:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ git push --force-with-lease origin intended-branch
</pre></div>
</div>
</div>
<p>Using <code>--force-with-lease</code> may not be necessary here, depending on your specific scenario.</p>
Docker `error while creating mount source path` on macOShttps://blog.pablobm.com/2020/05/30/docker-mkdir-error/2020-05-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>This week I was trying to get this Docker-based development environment set up for a project, on macOS. However I was getting this error:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>ERROR: for XYZ Cannot start service XYZ: error while creating mount source path '/Users/pablobm/Documents/foo/bar/baz': mkdir /Users/pablobm/Documents: file exists
</pre></div>
</div>
</div>
<p>After trying a few things, it turned out that the problem was with having the project files in <code>$HOME/Documents</code>. I moved them to a different location, and that did the trick.</p>
<p>It must be interfering with Catalina's security features or something. I find it odd that I don't get the usual "SoAndSo would like to access files in your Documents folder" dialog, but go figure.</p>
Notes on implementing a Referrer Policyhttps://blog.pablobm.com/2020/04/14/referrer-policy/2020-04-14T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<div class="update-notice">
<p><time>25 May 2020</time>: did more research, added more detail, changed my recommendation, and linked to the new test site.</p>
</div>
<h2 id="what-is-the-referer-header">What is the <code>Referer</code> header?</h2>
<p>To understand referrer policies, first you must understand the <code>Referer</code> header.</p>
<p>The <code>Referer</code> header is sent by browsers as part of HTTP requests. They are a friendly indicator of which website we came from that we ended up requesting a given URL.</p>
<p>So for example:</p>
<ol>
<li>You are visiting <code>http://example.com/foo/bar</code>.</li>
<li>You click on a link that takes you to <code>http://www.pablobm.com/</code>.</li>
<li>When the browser sends a request for <code>http://www.pablobm.com/</code>, it includes the header <code>Referer: http://example.com/foo/bar</code>, indicating where we come from.</li>
</ol>
<p>This is not the only scenario. This header is also sent when requesting assets (eg: images). For example:</p>
<ol>
<li>You are visiting <code>http://example.com/foo/bar</code>.</li>
<li>The page includes the image <code>http://example.com/image.jpg</code>.</li>
<li>When the browser sends a request for <code>http://example.com/image.jpg</code>, it includes the header <code>Referer: http://example.com/foo/bar</code>, indicating the page that references the image.</li>
</ol>
<p>And similarly, this can be sent in Ajax requests indicating the URL that originated them.</p>
<div class="please-note">
<p>Nitpick: note that <code>Referer</code> is misspelled (the middle <code>r</code> should be double). However <code>Referrer-Policy</code> is correctly spelled. This is because <code>Referer</code> came first (back in 1996), included the mistake, browsers adopted it, and now we can't simply change it as software that uses the original spelling would stop working. Going forward, I'll use the spelling "referrer", unless specifically referring (argh!) to the header verbatim.</p>
</div>
<h2 id="whats-this-for-and-can-it-be-a-problem">What's this for, and can it be a problem?</h2>
<p>Here are some uses for the <code>Referer</code> header:</p>
<ul>
<li>Identifying sites that are sending you traffic, which can be useful for your SEO strategy.</li>
<li>Finding sites that are linking to pages that don't exist on your site (404 errors), so that perhaps you can contact them to correct the problem.</li>
<li>Detecting and blocking <a href="https://simple.wikipedia.org/wiki/Hotlinking">hotlinking</a>: sites that link to your images (or other large resources) directly, while you bear the bandwidth costs.</li>
<li>Debugging issues: referrers can be included in error reports, helping diagnose some problems.</li>
</ul>
<p>However, this header may reveal information that you want secret. The classic scenario goes something like this:</p>
<ol>
<li>You use the "forgotten password" feature on website A.</li>
<li>You receive the usual password reset email including a link to website A, which you follow. The URL will contain a secret string, unique to your request for a new password.</li>
<li>The password reset page may include an image hosted on a third party site B (hotlinked).</li>
<li>The URL, including the secret string, will be sent to site B in the <code>Referer</code> header.</li>
<li>Someone listening in at site B could get this password reset URL, with the secret string, and reset the password before you do, effectively hijacking your account.</li>
</ol>
<p>This scenario may seem far-fetched, but it's perfectly legitimate, and the sort of issue that you need to bear in mind when securing your websites. And it's not the only scenario: there may be other details that you don't want leaked, and could potentially be accessed by third parties through a similar mechanism. For example: just the fact that you visited a specific site could be sensitive, if it is a site banned by a repressive government.</p>
<p>This concern may not relevant to many. For those for whom it is, there is a way to control the referrers sent as a result of visiting your website: setting a <em>Referrer Policy</em>.</p>
<h2 id="quick-summary-and-recommendation">Quick summary and recommendation</h2>
<p>If you have decided to define a referrer policy for your website, there are two aspects you should consider:</p>
<ol>
<li>Which specific policy to use.</li>
<li>How to set this policy in place.</li>
</ol>
<p>If you just want it in short, here's my current recommendation:</p>
<ol>
<li>Prefer <code><meta></code> tags over headers for wider compatibility.</li>
<li>Use the same policy across whole domains, as opposed to specific policies for individual pages.</li>
<li>Which policies are available depends on which browsers you want to support:
<ul>
<li>For modern browsers as well as old ones (Edge Legacy, IE11, old iOS and Android, etc): you may want to stick to <code>never</code> or <code>origin</code>.</li>
<li>If only supporting modern browsers: use one of the policies <a href="https://infosec.mozilla.org/guidelines/web_security#referrer-policy">recommended by Mozilla</a>: <code>no-referrer</code>, <code>same-origin</code>, <code>strict-origin</code>, or <code>strict-origin-when-cross-origin</code>.</li>
</ul>
</li>
</ol>
<p>If you want more detail, read on!</p>
<h2 id="what-policies-are-available">What policies are available?</h2>
<p>There are a few options when it comes to referrer policies, and you can see <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy">the list of standard policies at MDN</a>. They can look a bit complicated to wrap your head around but, in general, you can think of the options in two dimensions:</p>
<ul>
<li>What to share:
<ul>
<li>Send a full referrer (<code>https://example.com/foo/bar?a=b</code>).</li>
<li>Only send the so-called "origin" (protocol, domain and port: <code>https://example.com</code>).</li>
<li>Don't sent a referrer at all.</li>
</ul>
</li>
<li>With whom to share:
<ul>
<li>Other resources in the same origin.</li>
<li>From an HTTPS site to an HTTP site, or across sites with the same protocol.</li>
<li>Just any site.</li>
</ul>
</li>
</ul>
<p>Here's a summary table:</p>
<table class="data-table data-table--referrer-policies">
<thead>
<th>policy \ referrer</th>
<th>origin, path, query string</th>
<th>origin</th>
<th>nothing</th>
</thead>
<tbody>
<tr>
<th><code>unsafe-url</code></th>
<td>always</td>
<td></td>
<td></td>
</tr>
<tr>
<th><code>origin</code></th>
<td></td>
<td>always</td>
<td></td>
</tr>
<tr class="data-table__highlighted-row">
<th><code>no-referrer</code></th>
<td></td>
<td></td>
<td>always</td>
</tr>
<tr>
<th><code>origin-when-cross-origin</code></th>
<td>same origin</td>
<td>other origin</td>
<td></td>
</tr>
<tr class="data-table__highlighted-row">
<th><code>same-origin</code></th>
<td>same origin</td>
<td></td>
<td>other origin</td>
</tr>
<tr>
<th><code>no-referrer-when-downgrade</code></th>
<td>same or higher security</td>
<td></td>
<td>lower security</td>
</tr>
<tr class="data-table__highlighted-row">
<th><code>strict-origin</code></th>
<td></td>
<td>same or higher security</td>
<td>lower security</td>
</tr>
<tr class="data-table__highlighted-row">
<th><code>strict-origin-when-cross-origin</code></th>
<td>same origin</td>
<td>other origin, same or higher security</td>
<td>other origin, lower security</td>
</tr>
</tbody>
</table>
<p>As I write these lines, <a href="https://infosec.mozilla.org/guidelines/web_security.html#referrer-policy">Mozilla's InfoSec guidelines</a> recommend using one of these values only: <code>no-referrer</code>, <code>same-origin</code>, <code>strict-origin</code>, or <code>strict-origin-when-cross-origin</code>.</p>
<p>Additionally, some old browsers do not support these policies. Instead they implement an old version of the spec that only defines four of them: <code>default</code>, <code>always</code>, <code>origin</code>, and <code>never</code>. Affected browsers include old iOS and Android devices, IE11 and "Legacy" Edge (more on Edge later).</p>
<p>I'll refer to these old policies as, well, the "old policies" from now on. Of these, <code>default</code>, <code>always</code>, and <code>never</code> are considered deprecated, while <code>origin</code> made it to the current spec.</p>
<h2 id="how-do-i-implement-a-policy">How do I implement a policy?</h2>
<p>There are two main ways to implement a referrer policy: an HTTP header, or an HTML <code><meta></code> tag.</p>
<p>Technically there are other options, such as setting specific policies for individual links or images. Or doing like Google does, altering all outgoing links to point to an intermediate page that then redirects to the intended, final URL. I'm not going to cover any of that here though, as that was outside the scope of my research.</p>
<h3 id="referrer-policy-as-an-http-header">Referrer policy as an HTTP header</h3>
<p>The HTTP header is <code>Referrer-Policy</code>, followed by the name of the desired policy. For example:</p>
<div class="language-http code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>Referrer-Policy: strict-origin
</pre></div>
</div>
</div>
<p>Technically there could be several policies, comma-separated. This could be useful to provide a fallback policy for the old browsers, along with a "proper" one for modern browsers. However old browsers don't appear to recognise this header, defeating the purpose.</p>
<p>Also technically, you could send the same header twice, with different values. Again this can be a problem as your server-side framework may not support duplicate headers. For example, currently frameworks based on the <a href="https://github.com/rack/rack">Rack webserver interface</a> don't support this. This includes the popular Ruby on Rails.</p>
<h3 id="referrer-policy-as-a-meta-html-tag">Referrer policy as a <code><meta></code> HTML tag</h3>
<p>The HTML <code><meta></code> tag must have the attribute <code>name="referrer"</code> (correct spelling with 4 "r") and a <code>content</code> attribute with the name of the desired policy as value:</p>
<div class="language-html code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><meta</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">"</span><span class="content">referrer</span><span class="delimiter">"</span></span> <span class="attribute-name">content</span>=<span class="string"><span class="delimiter">"</span><span class="content">strict-origin</span><span class="delimiter">"</span></span> <span class="tag">/></span>
</pre></div>
</div>
</div>
<p>This appears to be supported by any browser aware of referrer policies, even if not all possible values will be understood. This is in opposition to using a header to specify the policy, which is only understood by modern browsers.</p>
<h2 id="declaring-more-than-one-policy">Declaring more than one policy</h2>
<p>So there are standard policies, understood by modern browsers, and old policies, understood by older browsers. Is there a way to provide two policies, so that both modern and old browsers have something to work with? This would be consistent with the standard as drafted as I write these lines, which says:</p>
<blockquote class="block-quote">
<p>11.1: (...) unknown policy values will be ignored, and when multiple sources specify a referrer policy, the value of the latest one will be used (...)</p>
<footer class="block-quote__cite">
<p>— <cite><a href="https://web.archive.org/web/20200518192738/https://w3c.github.io/webappsec-referrer-policy/">Referrer Policy, Editor's Draft (1 April 2020)</a></cite></p>
</footer>
</blockquote>
<p>Unfortunately, this doesn't appear to work. If you provide two policies, old browsers will use the second policy, regardless of whether they understand it or not. Modern browsers also always apply the second policy, but they understand both old and new values, so this is consistent with the spec.</p>
<p>As for what browsers do with policies they don't understand, here's a table:</p>
<table class="data-table data-table--referrer-policy-compatibility">
<thead>
<th>policy \ browser</th>
<th>Old iOS/Safari<sup><a href="#old-ios">1</a></sup></th>
<th>Old Android<sup><a href="#old-android">2</a></sup></th>
<th>Edge Legacy</th>
<th>IE11</th>
</thead>
<tbody>
<tr>
<th><code>unsafe-url</code></th>
<td><code>unsafe-url</code></td>
<td rowspan="7"><code>no-referrer</code></td>
<td rowspan="7"><code>no-referrer-when-downgrade</code></td>
<td><code>unsafe-url</code></td>
</tr>
<tr>
<th><code>no-referrer</code></th>
<td rowspan="3"><code>no-referrer</code></td>
<!-- rowspan -->
<!-- rowspan -->
<td><code>no-referrer</code></td>
</tr>
<tr>
<th><code>origin-when-cross-origin</code></th>
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
<td rowspan="5"><code>no-referrer-when-downgrade</code></td>
</tr>
<tr>
<th><code>same-origin</code></th>
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
</tr>
<tr>
<th><code>no-referrer-when-downgrade</code></th>
<td><code>no-referrer-when-downgrade</code></td>
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
</tr>
<tr>
<th><code>strict-origin</code></th>
<td rowspan="2"><code>no-referrer</code></td>
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
</tr>
<tr>
<th><code>strict-origin-when-cross-origin</code></th>
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
<!-- rowspan -->
</tr>
</tbody>
</table>
<p>So for example, if you provide two policies: <code>origin</code> and <code>strict-origin</code>, then IE11 will apply <code>no-referrer-when-downgrade</code>, while an old Android browser will apply <code>no-referrer</code>. This is the same as if you had only provided <code>strict-origin</code>, as the first policy is always ignored.</p>
<h2 id="checking-that-this-is-working">Checking that this is working</h2>
<p>So you have implemented a referrer policy after following the advice above. How do you know that it is correctly set up?</p>
<p>A simple method is using the network inspector in your browser. You can use it to verify which referrers are sent, as well as what policy applies to a given document.</p>
<p>For example, these are captures of the inspectors from Firefox 68 and Chrome 80 (actually Chromium on my Linux machine):</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/referrer-policy-firefox-inspector.png" alt="Examining the referrer policy on Firefox" />
<img src="/images/referrer-policy-chrome-inspector.png" alt="Examining the referrer policy on Chrome" />
</div>
<figcaption>
<p class="description">Examining the referrer policy on Mozilla Firefox and Google Chrome</p>
</figcaption>
</figure>
<p>You can see:</p>
<ol>
<li>Which referrer policy was given by the server (in this case with a header).</li>
<li>Which <code>Referer</code> value was sent by the browser, based on the previous policy.</li>
<li>Which policy applied to a specific request, again based on the previous policy.</li>
<li>As a bonus, on Firefox you can filter headers by name, helping you find them.</li>
</ol>
<p>Remember that no referrer is sent in certain situations where it doesn't make sense. For example, if you enter a URL manually on your address bar, there's no referrer as you didn't follow a link.</p>
<h2 id="other-considerations">Other considerations</h2>
<h3 id="the-referrer-policy-cant-be-changed-dynamically">The referrer policy can't be changed dynamically</h3>
<p>You can't change the referrer policy dynamically using JavaScript on the browser. The policy set in the initial page request, be it in the headers or in the <code><meta></code> tag of your original document, will be the one that will stick for as long as the page lives. For example, it's not possible to change it by altering the DOM and changing the <code><meta></code> tag. This is on purpose, as allowing it to change would open the door to altering it maliciously via <a href="https://en.wikipedia.org/wiki/Cross-site_scripting">Cross-site Scripting</a>.</p>
<h3 id="spas-and-turbolinks">SPAs and Turbolinks</h3>
<p>The previous point is particularly important if your site works as a <a href="https://en.wikipedia.org/wiki/Single-page_application">Single-Page Application</a>, or if it uses a technology such as <a href="https://github.com/turbolinks/turbolinks">Turbolinks</a> to speed up page load. In these cases, it won't be possible for different pages in your site to define different referrer policies. Only the one defined in the first page load will work for the rest of the session, until there's again a full-page reload.</p>
<p>For this reason, as well as to generally avoid accidents, I recommend using the same referrer policy across the whole domain.</p>
<h3 id="default-policies-set-by-frameworks">Default policies set by frameworks</h3>
<p>If you are seeing a referrer policy that you didn't expect, it might be that your framework sets a default value. For example, Ruby on Rails sends a header <code>Referrer-Policy: strict-origin-when-cross-origin</code> by default, I believe <a href="https://github.com/rails/rails/commit/428939be9f954d39b0c41bc53d85d0d106b9d1a1">since version 5.2.0</a>.</p>
<h3 id="edge-vs-edge-legacy">Edge vs. Edge Legacy</h3>
<p>Until this research, I thought that MS Edge was an "evergreen" browser. In other words, that it updated automatically and always stayed at the latest version. This turned out to be not exactly true, at least at the moment.</p>
<p>There are two families of MS Edge out there. The original one, which I have seen dubbed "Edge Legacy", includes versions up to 44 and uses Microsoft's own EdgeHTML rendering engine.</p>
<p>Then there's "current" Edge. If a Windows user deliberately downloads and installs Edge, they will get this, which is indeed evergreen. It uses Google's Blink rendering engine, just as Google Chrome does.</p>
<p>As a result of this requirement to download and install, current Edge has only a fraction of the market share that Edge Legacy enjoys: 0.26% vs. 2.18% according to <a href="https://gs.statcounter.com/browser-market-share#monthly-201903-202003">Statcounter Global Stats as of March 2020</a>. Fortunately this is expected to change in the future, as Microsoft are <a href="https://blogs.windows.com/msedgedev/2020/01/15/upgrading-new-microsoft-edge-79-chromium/">planning on including it with Windows updates</a>. Eventually.</p>
<p>Regarding referrer policies, Edge Legacy only supports old referrer policies. Current Edge supports the new policies.</p>
<p>And a last "fun" fact: Edge Legacy identifies itself as v18 in its <code>User-Agent</code>, even though the actual version can be as high as 44. From what I can tell, this refers to the version of the rendering engine EdgeHTML, as opposed to that of the browser.</p>
<h2 id="try-yourself">Try yourself!</h2>
<p>Writing this guide took me a lot of tedious, repetitive testing. My first version included some mistakes, and I'm sure this version has problems somewhere. So before taking all this from me at face value, how about you try out yourself?</p>
<p>To help with this, I created <a href="https://referrer-policy.info">https://referrer-policy.info</a>, a site providing many subdomains that implement different policies and combinations. Give it a whirl and tell me if you discover something I missed! You can point my mistakes out to me on Twitter at <a href="https://twitter.com/pablobm">@pablobm</a>.</p>
<div class="footnotes">
<ol>
<li id="old-ios"><p>For old iOS, I tested an old iPad mini running 9.3.6</p></li>
<li id="old-android"><p>For old Android, I tested the Android stock browser on an emulator running Android v22 (Lollipop)</p></li>
</ol>
</div>
Testing non-existencehttps://blog.pablobm.com/2020/01/09/testing-non-existence/2020-01-09T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>Given a test like this one:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>expect(page).to have_css(<span class="string"><span class="delimiter">"</span><span class="content">[data-search-results]</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>click_on <span class="string"><span class="delimiter">"</span><span class="content">Hide results</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>expect(page).not_to have_css(<span class="string"><span class="delimiter">"</span><span class="content">[data-search-resutls]</span><span class="delimiter">"</span></span>) <span class="comment"># Typo!</span>
</pre></div>
</div>
</div>
<p>Something like this may pass in your code… but it is not testing
what you think it is testing. There is a typo in the second expectation,
so of course it is not going to find <code>[data-search-resutls]</code>, regardless of
whether your app is working as intended or not.</p>
<p>To avoid this situation, I always put the thing to look for (in this case
the CSS selector) in a variable, and then I use this variable for both the
test of existence and the test of non-existence:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>results_selector = <span class="string"><span class="delimiter">"</span><span class="content">[data-search-results]</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>expect(page).to have_css(results_selector)
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>click_on <span class="string"><span class="delimiter">"</span><span class="content">Hide results</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>expect(page).not_to have_css(results_selector)
</pre></div>
</div>
</div>
<p>This way, a typo will make the first expectation fail, and the
issue will be made evident.</p>
<p>It is important that there is also a test of existence (first expectation in
this example). It does not matter if this is as an assertion/expectation in the
same test, or in a separate one. Having only a test that checks for
non-existence of something can be misleading for the same reason: your target
may not be present for completely different reasons from what you expect.</p>
40https://blog.pablobm.com/2020/01/07/40/2020-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>1111<sub>3</sub>.</li>
<li>Radio hits.</li>
<li>Zirconium.</li>
<li>Enki.</li>
<li>Ruby.</li>
<li>When life begins…</li>
</ul>
To mock, or not to mockhttps://blog.pablobm.com/2019/10/31/to-mock-or-not-to-mock/2019-10-31T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>During a mentoring session recently, I had a conversation about using mocks in unit tests: when and how they are appropriate, when I feel they enhance tests, and what I don't like about them. Here are my thoughts in writing for future reference.</p>
<p>First, I use several mocking-related terms here, and I go by <a href="http://xunitpatterns.com/Test%20Double.html">George Meszaros's definitions for those words</a>. Here they are in a snappier form as they appear on <a href="https://martinfowler.com/articles/mocksArentStubs.html">Martin Fowler's article on this very same topic</a> (which is a recommended read on this subject):</p>
<blockquote class="block-quote">
<p>Meszaros uses the term <strong>Test Double</strong> as the generic term for any kind of pretend object used in place of a real object for testing purposes (…) [He] defined five particular kinds of double:</p>
<ul>
<li><strong>Dummy</strong> objects are passed around but never actually used. Usually they are just used to fill parameter lists.</li>
<li><strong>Fake</strong> objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).</li>
<li><strong>Stubs</strong> provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.</li>
<li><strong>Spies</strong> are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.</li>
<li><strong>Mocks</strong> are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.</li>
</ul>
</blockquote>
<p>Once that's out of the way, let's get to the code that was subject of the conversation. It included a test that I wasn't convinced about, and looked a bit like this:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">QueueFlushWorker</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">clears half of the dead jobs</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> some_dead_jobs_stub = double(<span class="key">clear!</span>: <span class="predefined-constant">true</span>)
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> all_dead_jobs_stub = double(<span class="key">first</span>: some_dead_jobs_stub)
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> dead_queue_stub = double(<span class="key">size</span>: <span class="integer">4</span>, <span class="key">jobs</span>: all_dead_jobs_stub)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> allow(<span class="constant">Queue</span>).to receive(<span class="symbol">:read</span>).and_return(dead_queue_stub)
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> worker = <span class="constant">QueueFlushWorker</span>.new
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> worker.process
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> expect(all_dead_jobs_stub).to have_received(<span class="symbol">:first</span>).with(<span class="integer">2</span>)
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> expect(some_dead_jobs_stub).to have_received(<span class="symbol">:clear!</span>)
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>And tested code that looked a bit like this:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">class</span> <span class="class">QueueFlushWorker</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="keyword">def</span> <span class="function">process</span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> dead_queue = <span class="constant">Queue</span>.read(<span class="symbol">:dead</span>)
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> num_dead = dead_queue.size
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> dead_jobs = dead_queue.jobs.first(num_dead / <span class="integer">2</span>)
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> dead_jobs.clear!
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>The problem I had with that test is that it knows too much about what the subject of the test is doing under the hood. It knows of every method call being made, and almost the order in which the code will run. I feel it's essentially writing the same code again and, as a result, if the code is wrong the test will pass happily anyway. It doesn't test the behaviour as much as specifics of the implementation, which are mocked and do nothing.</p>
<p>In this scenario, I would have written the test like follows:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">QueueFlushWorker</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">clears half the dead jobs</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="integer">4</span>.times.each { <span class="constant">Queue</span>::<span class="constant">Dead</span>.add(<span class="constant">Queue</span>::<span class="constant">Job</span>.new) }
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> expect(<span class="constant">Queue</span>::<span class="constant">Dead</span>.count).to eq(<span class="integer">4</span>)
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> worker = <span class="constant">QueueFlushWorker</span>.new
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> worker.process
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> expect(<span class="constant">Queue</span>::<span class="constant">Dead</span>.count).to eq(<span class="integer">2</span>)
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>This is called <em>state verification</em>. We set up the test so that the state of the system is A, we exercise the code (ie: run the code we are testing), and then we check that the state of the system is now B. This doesn't involve knowing what exactly happened inside.</p>
<p>(Unfortunately, in the specific case we were working on, <code>count</code> would not return an updated count straightaway, so this test didn't work.)</p>
<p>Now, this doesn't mean that doubles should not be used. There are perfectly valid use cases for them and they can be a great tool. Consider the following spec:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">FileBrowserWithInjection</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> describe <span class="string"><span class="delimiter">"</span><span class="content">#files_with_extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">returns the files with the given extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> f1 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file1.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> f2 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file2.py</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> f3 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file3.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> files = [f1, f2, f3]
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> filesystem = double(<span class="key">files</span>: files)
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> browser = <span class="constant">FileBrowserWithInjection</span>.new(filesystem)
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> result = browser.files_with_extension(<span class="string"><span class="delimiter">"</span><span class="content">rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> expect(result).to match_array([f1, f3])
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>That's a lot of stubs, and it could feel suspicious. However I'm ok with this code because it's using dependency injection, and it's only using stubs to fake the thing that will be injected. Ultimately, it's the final result that is being checked, which still falls under state verification.</p>
<p>A quick break to explain <a href="https://en.wikipedia.org/wiki/Dependency_injection">Dependency Injection</a> if you are not familiar: it means that we tell object A to use object B internally, by providing B as an argument or similar. This way, A doesn't need to know what class is B, and can be fed anything that implements a pre-arranged interface. It's particularly handy for having several implementations of the same abstract concept. For example, a notifier class (the "client") can receive a Twitter adapter or a Slack adapter (the "service"), with the same interfaces, and the notifier doesn't need to know which one it is using to send a notification.</p>
<p>In this example, A is <code>FileBrowserWithInjection</code>, while B is something we call <code>filesystem</code> and is completely faked with stubs. We use doubles to create a fake filesystem that will be used internally. We don't care about the order methods are called, which arguments they receive, etc. This could be done instead with fake classes, like follows:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">FileBrowserWithInjection</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> <span class="constant">FakeFile</span> = <span class="constant">Struct</span>.new(<span class="symbol">:name</span>)
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="constant">FakeFilesystem</span> = <span class="constant">Struct</span>.new(<span class="symbol">:files</span>)
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> describe <span class="string"><span class="delimiter">"</span><span class="content">#files_with_extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">returns the files with the given extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> f1 = <span class="constant">FakeFile</span>.new(<span class="string"><span class="delimiter">"</span><span class="content">file1.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> f2 = <span class="constant">FakeFile</span>.new(<span class="string"><span class="delimiter">"</span><span class="content">file2.py</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> f3 = <span class="constant">FakeFile</span>.new(<span class="string"><span class="delimiter">"</span><span class="content">file3.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> filesystem = <span class="constant">FakeFilesystem</span>.new([f1, f2, f3])
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> browser = <span class="constant">FileBrowserWithInjection</span>.new(filesystem)
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> result = browser.files_with_extension(<span class="string"><span class="delimiter">"</span><span class="content">rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> expect(result).to match_array([f1, f3])
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>The doubles in the previous example just provided a handy shortcut to the fake classes used in this latest example.</p>
<p>Now have a look at this variant:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">FileBrowserWithGlobals</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> describe <span class="string"><span class="delimiter">"</span><span class="content">#files_with_extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">returns the files with the given extension</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> f1 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file1.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> f2 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file2.py</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> f3 = double(<span class="key">name</span>: <span class="string"><span class="delimiter">"</span><span class="content">file3.rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> files = [f1, f2, f3]
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> allow(<span class="constant">FileSystem</span>).to receive(<span class="symbol">:files</span>).and_return(files)
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> browser = <span class="constant">FileBrowserWithGlobals</span>.new
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> result = browser.files_with_extension(<span class="string"><span class="delimiter">"</span><span class="content">rb</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> expect(result).to match_array([f1, f3])
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>It's pretty much the same, but in this case the class under test (<code>FileBrowserWithGlobals</code>) knows that it's using <code>FileSystem</code> internally, and there's no changing that. As a result, we have to stub <code>FileSystem</code> with <code>allow</code>. This means our test knows a bit more about what's going on behind the hood than in the previous example.</p>
<p>Often there's no escaping this. I don't mean to say that we should use dependency injection everywhere. For example, you'll have code that calls ActiveRecord models directly, and that's fine. Dependency Injection is just another useful tool in your belt.</p>
<p>As for testing that things have been called, perhaps with specific arguments and in a specific order, that can be useful too sometimes. For example when testing a cache. Here's an example:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>describe <span class="constant">MemoryCache</span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> it <span class="string"><span class="delimiter">"</span><span class="content">only accesses the source once</span><span class="delimiter">"</span></span> <span class="keyword">do</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> source = double(<span class="key">foobar</span>: <span class="predefined-constant">nil</span>)
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> cache = <span class="constant">MemoryCache</span>.new(source)
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> cache.read(<span class="symbol">:foobar</span>)
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> cache.read(<span class="symbol">:foobar</span>)
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> expect(source).to have_received(<span class="symbol">:foobar</span>).once
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span><span class="keyword">end</span>
</pre></div>
</div>
</div>
<p>Using state verification, we can't tell easily if the source was hit once or twice. It's much easier to use a mock (a spy here actually!) and verify it at the end.</p>
<p>(OK, you could have a fake class that behaves as the source, and counts the number of hits to <code>foobar</code>, and do state verification on that; but that would be more work).</p>
<p>Having said all that, there's still a school of thought that approves of doing more mocking than I normally like. What I'm explaining here is not gospel, but it's the sort of things we should be thinking about when using these tools, and the trade-offs to consider.</p>
Map/Reduce with CouchDB: a visual primerhttps://blog.pablobm.com/2019/07/18/map-reduce-with-couchdb-a-visual-primer/2019-07-18T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<p>As part of a larger experiment, I recently played around a bit with CouchDB.
Soon I had to run some map/reduce queries and realised that CouchDB handles
these in a way that is significantly different from what many developers
like me expect.</p>
<p>For those of us who have done simple map/reduce work with JavaScript, Ruby,
Elixir, Python, etc, the model used by CouchDB can be surprising and confusing.
There's a lot of documentation out there, but while figuring it out I felt
I could use a visual explanation rather than walls of text.</p>
<p>Here's my attempt at providing such a visual explanation.
I hope that this guide will help those who are as confused as I used to be.</p>
<h2 id="visits-per-day-month-or-year">Visits per day, month or year</h2>
<p>The documents in this example will be from a hypothetical web analytics
database. To illustrate the map/reduce process, I'll create a view that will
allow us to query the total number of visits to the site, grouped by day, month
or year. The data looks like this:</p>
<div class="main-section table-like-svg">
<table>
<thead>
<tr>
<th>id</th>
<th>document</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>{ "uri": "/foo", "userAgent": "Edge",<br />"visitedAt": "2019-05-22T12:44:32Z" }</td>
</tr>
<tr>
<td>2</td>
<td>{ "uri": "/bar", "userAgent": "Firefox",<br />"visitedAt": "2019-05-30T08:05:07Z" }</td>
</tr>
<tr>
<td>3</td>
<td>{ "uri": "/foo", "userAgent": "Safari",<br />"visitedAt": "2019-06-03T14:12:45Z" }</td>
</tr>
<tr>
<td>4</td>
<td>{ "uri": "/bar", "userAgent": "Firefox",<br />"visitedAt": "2019-06-11T15:23:17Z" }</td>
</tr>
<tr>
<td>5</td>
<td>{ "uri": "/bar", "userAgent": "Firefox",<br />"visitedAt": "2019-06-11T01:55:41Z" }</td>
</tr>
</tbody>
</table>
</div>
<p>When thinking of map/reduce, we may think of a two-stage process. However, when
thinking in terms of CouchDB I prefer to think of it as four steps.</p>
<h3 id="map-the-necessary-detail-required-for-the-query">1. Map the necessary detail required for the query</h3>
<p>The process starts with the map step. When mapping on CouchDB, we <em>emit</em> key/value pairs:</p>
<ul>
<li>The keys will be used to group and filter results. In this case, I'll use the
<code>visitedAt</code> timestamp, as I want to group by day, month or year.</li>
<li>The values will be later combined to create the final results. In this case I
want to count visits, and each document represents a single visit. Therefore,
for each input record I'll emit a value of <code>1</code>, a literal number one
signifying a single visit.</li>
</ul>
<p>The concept of "grouping" is governed by the emitted keys, and needs a bit
more of clarification. They keys won't just be the literal <code>visitedAt</code> strings.
Instead I will split those date strings into <code>[year, month, day]</code> arrays,
which will allow me to group results by three different granularity levels:
day, month, or year.</p>
<p>This is a way to implement the map step for this query:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">// The map function</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="keyword">function</span> (document) {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> const [date, time] = document.visitedAt.split(<span class="string"><span class="delimiter">"</span><span class="content">T</span><span class="delimiter">"</span></span>);
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> const [year, month, day] = date.split(<span class="string"><span class="delimiter">"</span><span class="content">-</span><span class="delimiter">"</span></span>);
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> emit([year, month, day], <span class="integer">1</span>);
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>}
</pre></div>
</div>
</div>
<p>This map step can be visually represented as follows:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--20-map.svg" alt="Map each document into a three items: its original id, a key as array of date components, and the integer value 1" />
</div>
</figure>
<p>The result is a list of visits, each represented by a date and
each counting as a single (1) visit. Dates are represented as a three-element
array, which will become useful in the next step.</p>
<h3 id="group-intermediate-results-by-key">2. Group intermediate results by key</h3>
<p>Before going into the reduce step, CouchDB sorts the results of the map step
into groups.</p>
<p>If your query included the <code>group_level</code> parameter, then this is used at this
stage. It will dictate what portion of the keys (the ones generated in the
map step) will be used as grouping key. For example:</p>
<ul>
<li><code>group_level=3</code>: group by the three items in the key (year, month and day).</li>
<li><code>group_level=2</code>: group by the first two items in the key (year and month).</li>
<li><code>group_level=1</code>: group by the first item in the key (the year only).</li>
<li><code>group_level=0</code>: group all rows together into a single group (their date
won't matter). This is the default.</li>
</ul>
<p>This would be an example of grouping with <code>group_level=2</code>:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--30-grouping.svg" alt="Results from the map step, grouped by month (group level 2)" />
</div>
</figure>
<p>However, this is not entirely what happens. At this stage, it's important to
understand the distributed nature of CouchDB, as this is not abstracted from
us. We will have to accomodate for it during the map/reduce process.</p>
<p>At the grouping step, the groups will be formed independently in each node
of the CouchDB cluster. Again with an example of <code>group_level=2</code>, and assuming
that your cluster has two nodes, the grouping will look more like this:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--40-grouping-by-cluster.svg" alt="Results from the map step, grouped by cluster, then by month (group level 2)" />
</div>
</figure>
<p>This separation of results into nodes will be come important later, in the
re-reduce stage.</p>
<h3 id="reduce-within-each-db-node">3. Reduce within each DB node</h3>
<p>At this point, we have groups of "equivalent" records. Equivalent because
each record represents a single hit, and each group only contains hits on the
same time segment (day, month or year). This is similar to what <code>GROUP BY</code>
would get us in a SQL query.</p>
<p>Also similarly to <code>GROUP BY</code>, now we have to consolidate each group into a
single result. In this case, these new results will represent the time
segment of the group and the count of records in the group.</p>
<p>In more common flavours of map/reduce, the reduce function is called once
for each record. In CouchDB however, it receives groups of inputs.
Specifically, the reduce function appears to take two arguments:</p>
<ul>
<li>A list of ids and keys from the map results.</li>
<li>A list of values from the map results.</li>
</ul>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--50-reduce-arguments.svg" alt="First argument is a list of id/key pairs. Second argument is a list of values." />
</div>
</figure>
<p>Therefore, a reduce function could be written like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">// First attempt at reduce</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="keyword">function</span>(keys, values) {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="keyword">return</span> values.length;
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>}
</pre></div>
</div>
</div>
<p>The <code>keys</code> argument won't be used in this case. The <code>values</code> argument will just
be a list of ones <code>[1, 1, 1, ..., 1]</code>, and I want its length (or the sum, which
is the same thing in this case).</p>
<p>After this step, our results will be reduced, but only within
each node of the CouchDB cluster:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--60-reduce-in-clusters.svg" alt="Groups reduced within each cluster" />
</div>
</figure>
<p>The process is not complete. There's still one step to go.</p>
<h3 id="re-reduce-into-the-final-results">4. Re-reduce into the final results</h3>
<p>The "sibling" results across the nodes now have to be consolidated into the
final results for the whole database. CouchDB calls this step "re-reduce".</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--70-re-reduce.svg" alt="Groups re-reduced across clusters, leading to the final results" />
</div>
</figure>
<p>For the re-reduce step, CouchDB calls <strong>the reduce function again</strong>, but with
different arguments:</p>
<figure class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/couchdb-map-reduce-steps--80-re-reduce-arguments.svg" alt="First argument is null; second is a list of values from the same group; third is a boolean true." />
</div>
</figure>
<p>This time there are no keys (the first argument is null), the values (second
argument) are results from the reduce step, and there's a third argument. I'm
calling this new argument <code>isRereduce</code>, and its value will be <code>false</code> during
the reduce step and <code>true</code> during re-reduce.</p>
<p>So it turns out that the previous reduce function was not correct, because it
needed to serve two different steps. We'll have to use the boolean third
argument to tell which of two behaviours to implement:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">// Reduce and re-reduce</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="keyword">function</span>(_keys_, values, isRereduce) {
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="keyword">if</span> (isRereduce) {
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> <span class="comment">// `sum` is provided by CouchDB. It adds up all values in an array</span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="keyword">return</span> sum(values);
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> } <span class="keyword">else</span> {
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="keyword">return</span> values.length;
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> }
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>}
</pre></div>
</div>
</div>
<p>A keen eye may realise that this function could be simpler: we could just do
<code>sum</code> in both cases. However this would work only because of the specific
example we are working with here. For the sake of explaining the difference
between reduce and re-reduce, I have preferred to do it this way instead.</p>
<h3 id="final-notes">Final notes</h3>
<p>The row-counting in this example is very common for a reduce (and
re-reduce). For this reason, CouchDB provides a built-in function to perform
it, called <code>_count</code>. We can set the <code>reduce</code> property of the CouchDB view
to the string value <code>"_count"</code> instead of providing the full function, and
the result will be the same.</p>
<p>This is what the design document can look like now, using this shortcut:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="key"><span class="delimiter">"</span><span class="content">views</span><span class="delimiter">"</span></span>: {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="key"><span class="delimiter">"</span><span class="content">entries_by_date</span><span class="delimiter">"</span></span>: {
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="key"><span class="delimiter">"</span><span class="content">map</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">function (document) { const date = document.date.split(</span><span class="char">\"</span><span class="content">T</span><span class="char">\"</span><span class="content">)[0].split(</span><span class="char">\"</span><span class="content">-</span><span class="char">\"</span><span class="content">); emit(date, 1); }</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="key"><span class="delimiter">"</span><span class="content">reduce</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">_count</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> }
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> }
<span class="line-numbers"><a href="#n8" name="n8">8</a></span>}
</pre></div>
</div>
</div>
<p>Something that can trip you up: make sure not to name your functions! Provide
your map and reduce functions as anonymous functions. In other words: start them
with <code>function(...) {</code> and not with <code>function map(...)</code>. Otherwise, you'll get
an error like this one:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>Compilation of the map function in the 'entries_by_date' view failed: Expression does not eval to a function.
</pre></div>
</div>
</div>
<p>In order to carry out my research, I created a small project with a Docker
image running CouchDB, as well as some Ruby scripts to populate and query it.
You can find them at <a href="https://gitlab.com/pablobm/couchdb-research">https://gitlab.com/pablobm/couchdb-research</a>,
along with instructions on how to run it yourself.</p>
The Store without Ember Datahttps://blog.pablobm.com/2019/01/15/store-without-ember-data/2019-01-15T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Developing with Ember.js, we are used to dealing with the Store service which
many of us associate with Ember Data. It was while I was doing some research on
how Ember integrates with data layers that I realised that the Store is part
of an integration glue that exists in Ember even when Ember Data is not
present. Moreover, you can create a light integration with a data layer
by simply hooking into Ember's Store interface.</p>
<p>Here's a little guide for you to experiment with the possibilities.</p>
<h2 id="embers-minimal-store-interface">Ember's minimal Store interface</h2>
<p>First, to make sure we are not using Ember Data accidentally, let's remove it
from our app. Simply remove the appropriate line from your <code>packages.json</code> and
rerun <code>npm install</code> (or <code>yarn install</code>):</p>
<div class="language-diff code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="line comment">diff --git a/package.json b/package.json</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="line comment">index 5ba44f8..c536b45 100644</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="line head"><span class="head">--- </span><span class="filename">a/package.json</span></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="line head"><span class="head">+++ </span><span class="filename">b/package.json</span></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="line change"><span class="change">@@</span> -32,7 +32,6 <span class="change">@@</span></span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-cli-shims</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^1.2.0</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-cli-sri</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^2.1.0</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-cli-uglify</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^2.0.0</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span><span class="line delete"><span class="delete">-</span> <span class="key"><span class="delimiter">"</span><span class="content">ember-data</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">~3.1.0</span><span class="delimiter">"</span></span>,</span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-export-application-global</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^2.0.0</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-load-initializers</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^1.0.0</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-maybe-import-regenerator</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^0.1.6</span><span class="delimiter">"</span></span>,
</pre></div>
</div>
</div>
<p>So now we know there's no Ember Data. Next, let's say we have declared a route
that looks like follows:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">/// app/router.js</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">users</span><span class="delimiter">'</span></span>, { <span class="key">path</span>: <span class="string"><span class="delimiter">'</span><span class="content">/users/:user_id</span><span class="delimiter">'</span></span> });
</pre></div>
</div>
</div>
<p>As well as this template:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{{!--- app/templates/users.hbs ---}}
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="tag"><h1></span>User #{{model.id}}: {{model.name}}<span class="tag"></h1></span>
</pre></div>
</div>
</div>
<p>If we do not provide a route module, Ember will provide one for us, using a
default implementation. This will see the <code>:user_id</code> parameter last in the
path, and from there it will figure out that we are expecting to retrieve a
model called <code>user</code> , and its ID will be given in the URL. Roughly it will
work like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">/// app/routes/users.js</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="reserved">import</span> Route from <span class="string"><span class="delimiter">'</span><span class="content">@ember/routing/route</span><span class="delimiter">'</span></span>;
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="reserved">export</span> <span class="keyword">default</span> Route.extend({
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> model(params) {
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> <span class="keyword">return</span> <span class="local-variable">this</span>.store.find(<span class="string"><span class="delimiter">'</span><span class="content">user</span><span class="delimiter">'</span></span>, params.user_id);
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> }
<span class="line-numbers"><a href="#n8" name="n8">8</a></span>});
</pre></div>
</div>
</div>
<p>Note that this uses <code>this.store.find()</code> instead of <code>this.store.findRecord()</code>!
Why is this? Well, there's a difference between Ember Data's interface
and Ember's protocol to interact with a data layer.</p>
<p>Ember can integrate automatically with a data layer in this small way, providing
a default route module and figuring out the model name and record id
from the path. From here, a clever data layer can provide the appropriate integration
hooks, and Ember will use them. In this case the hook is a <code>store</code> service which
provides a <code>find</code> method.</p>
<p>But why <code>find</code> and not <code>findRecord</code>? Because Ember and Ember Data are
independent. Long time ago it was established that this method would be called
<code>find</code>, and initially Ember Data used <code>find</code> instead of <code>findRecord</code>.
Eventually Ember Data moved on to a new interface, bringing the methods that
we use nowadays. However Ember didn't need to follow suit, and anyway there
were already other data layers that were implementing this interface already
and there was no point on breaking their integrations. Ember Data does provide
a <code>find</code> method which simply translates to <code>findRecord</code> internally, and all
works as expected.</p>
<h2 id="the-default-store">The default store</h2>
<p>But anyway, as I was saying, somewhere in our app we have the following code:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.store.find(<span class="string"><span class="delimiter">'</span><span class="content">user</span><span class="delimiter">'</span></span>, params.user_id);
</pre></div>
</div>
</div>
<p>When Ember Data is not present, Ember provides a default store. This store
assumes that we have defined our models (in this case, a <code>user</code> model) in
files living at <code>app/models</code>. This would be an example:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// app/models/user.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="reserved">export</span> <span class="keyword">default</span> {
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> find(id) {
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> <span class="keyword">return</span> <span class="keyword">new</span> Promise(<span class="keyword">function</span>(resolve) {
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> resolve({
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="key">id</span>: <span class="integer">1</span>,
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="key">email</span>: <span class="string"><span class="delimiter">"</span><span class="content">pablo@thoughtbot.com</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> });
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> });
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> }
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>}
</pre></div>
</div>
</div>
<p>(To be precise, Ember doesn't care about the location and name of this file,
or even if there's a file. The important part is that the object above
is registered as <code>model:user</code> in the dependency injection container, but
that's a story for another day.)</p>
<p>This initial model is not very useful, returning a hard-coded object. This
second approach would work with a simple REST API:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">/// app/models/user.js</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="reserved">export</span> <span class="keyword">default</span> {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> find(id) {
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="keyword">return</span> fetch(<span class="error">`</span><span class="regexp"><span class="delimiter">/</span><span class="content">users</span><span class="delimiter">/</span></span><span class="predefined">$</span>{id}<span class="error">`</span>);
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> },
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>}
</pre></div>
</div>
</div>
<p>This approach is promising, but it has one big issue: we cannot use
dependency injection in objects that we instantiate this way. For
example, if instead of <code>fetch</code> we wanted to use the <code>ajax</code> service, or
if we wanted to grab configuration details from another service, we'd
be out of luck.</p>
<h2 id="a-custom-store">A custom store</h2>
<p>To allow our custom models to play with the injection container, the
simplest way might be to do just like Ember Data does, and provide
our own store service.</p>
<p>This is an example of a custom store service:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// app/services/store.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="reserved">import</span> Service, { inject } from <span class="string"><span class="delimiter">'</span><span class="content">@ember/service</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="reserved">import</span> { pluralize } from <span class="string"><span class="delimiter">'</span><span class="content">ember-inflector</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="reserved">export</span> <span class="keyword">default</span> Service.extend({
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="key">ajax</span>: inject(),
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> find(model, id) {
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> const resourcePath = pluralize(model);
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="keyword">return</span> <span class="local-variable">this</span>.ajax.request(<span class="error">`</span><span class="regexp"><span class="delimiter">/</span><span class="content">${resourcePath}</span><span class="delimiter">/</span></span><span class="predefined">$</span>{id}<span class="error">`</span>);
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> },
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>});
</pre></div>
</div>
</div>
<p>Then, to get this custom store injected by default in our routes, we
can register it with the container in an initializer:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">/// app/instance-initializers/setup-store.js</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="reserved">export</span> <span class="keyword">function</span> <span class="function">initialize</span>(application) {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> application.inject(<span class="string"><span class="delimiter">'</span><span class="content">route</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">store</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">service:store</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>}
</pre></div>
</div>
</div>
<p>And voila: your routers, default or otherwise, will be using your
custom store.</p>
<p>This is the same approach used by Ember Data which, at the time of writing
these lines, does exactly this: it injects its own store into routes,
controllers, etc, which is then used transparently by us developers.
See <a href="https://github.com/emberjs/data/blob/v3.6.0/addon/setup-container.js#L69"><code>addon/setup-container.js</code></a>
on the Ember Data source code to check for yourself.</p>
39https://blog.pablobm.com/2019/01/07/39/2019-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>3 + 5 + 7 + 11 + 13</li>
<li>Yttrium</li>
<li>Steps</li>
<li>Italy</li>
<li>Only one left to end the decade</li>
</ul>
Pluralization (and singularization)https://blog.pablobm.com/2018/08/07/pluralization-and-singularization/2018-08-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>I found myself <a href="https://en.wiktionary.org/wiki/yak_shaving">yak-shaving</a> pretty deeply
into a problem today, wondering why Ember was refusing to pluralize the word "beta"
as "betas".</p>
<p>Ultimately I wound up at the <a href="https://github.com/emberjs/ember-inflector/blob/e2d73b2/addon/lib/system/inflections.js">list of pluralization rules</a>
for <code>ember-inflector</code>. Among them, I found the culprit:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">// ...</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>inflect.plural(<span class="regexp"><span class="delimiter">/</span><span class="content">(buffal|tomat)o$</span><span class="delimiter">/</span><span class="modifier">i</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">\1</span><span class="content">oes</span><span class="delimiter">'</span></span>)
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>inflect.plural(<span class="regexp"><span class="delimiter">/</span><span class="content">([ti])um$</span><span class="delimiter">/</span><span class="modifier">i</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">\1</span><span class="content">a</span><span class="delimiter">'</span></span>)
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>inflect.plural(<span class="regexp"><span class="delimiter">/</span><span class="content">([ti])a$</span><span class="delimiter">/</span><span class="modifier">i</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">\1</span><span class="content">a</span><span class="delimiter">'</span></span>) <span class="comment">// <-- This bad fellow here is the problem</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>inflect.plural(<span class="regexp"><span class="delimiter">/</span><span class="content">sis$</span><span class="delimiter">/</span><span class="modifier">i</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">ses</span><span class="delimiter">"</span></span>)
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>inflect.plural(<span class="regexp"><span class="delimiter">/</span><span class="content">(?:([^f])fe|([lr])f)$</span><span class="delimiter">/</span><span class="modifier">i</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">\1</span><span class="content">\2</span><span class="content">ves</span><span class="delimiter">'</span></span>)
<span class="line-numbers"><a href="#n7" name="n7">7</a></span><span class="comment">// ...</span>
</pre></div>
</div>
</div>
<p>According to that rule, any words ending in "-ta" or "-ia" will remain unchanged when
pluralized. Initially, I was a bit confused by that, but then I did some looking up
to find examples of such words. Many of them are plural Latin words, such as
"blastomata", "bacteria", "branchiata", "media", "data", "trivia", etc. I can see how I
would not try to re-pluralize them in English.</p>
<p>However, there are many other words that we use in English that are caught in that rule
and I would pluralize, such as "phobia", "fashionista" (and "emberista", "pythonista"),
"magnolia", "pitta", "paranoia"… and of course "beta" and "delta". So annoying!</p>
<p>Fortunately, the library does offer a way to add your own pluralization rules, so
I ended up just doing that for the words I needed.</p>
<p>It's worth noting that this is not just an Ember gotcha. The <code>ember-inflector</code> package is
a direct port from the Ruby <code>ActiveSupport::Inflector</code>, with
<a href="https://github.com/rails/rails/blob/589dd0f/activesupport/lib/active_support/core_ext/integer/inflections.rb">the same default pluralization rules</a>,
so you get the same results if you try this in, say, a Ruby on Rails codebase:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>include <span class="constant">ActiveSupport</span>::<span class="constant">Inflector</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>pluralize(<span class="string"><span class="delimiter">"</span><span class="content">beta</span><span class="delimiter">"</span></span>) <span class="comment"># => "beta"</span>
</pre></div>
</div>
</div>
<p>And it's equally solvable by configuring the library:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>inflections <span class="keyword">do</span> |i|
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> i.plural <span class="string"><span class="delimiter">"</span><span class="content">beta</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">betas</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="keyword">end</span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>pluralize(<span class="string"><span class="delimiter">"</span><span class="content">beta</span><span class="delimiter">"</span></span>) <span class="comment"># => "betas"</span>
</pre></div>
</div>
</div>
<p>If you are asking "what are these plural and inflection things?", I recommend that you
read this article by Vaidehi Joshi, of the <a href="https://www.codenewbie.org/basecs">base<sub>cs</sub> podcast</a>
fame:
<a href="http://vaidehijoshi.github.io/blog/2015/09/01/inflections-everywhere-using-activesupport-inflector/">Inflections Everywhere: Using ActiveSupport Inflector</a>.
It explains why our frameworks need to know some grammar, how they go about it, and
why I'm not going to get "betas" added as a special case in these default rule sets.</p>
Is a number within a range?https://blog.pablobm.com/2018/06/17/number-within-a-range/2018-06-17T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>The other day I saw this on a pull request:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>start(appointment) <= now && end(appointment) >= now
</pre></div>
</div>
</div>
<p>I find this a bit jarring to read and avoid it in my code. Instead I prefer something like the following:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>start(appointment) <= now <= end(appointment)
</pre></div>
</div>
</div>
<p>This predicate, closer to the mathematical predicate, reads better to me. It's easier to see that <code>now</code> needs to be in between two limits. This syntax is allowed in some languages such as Python.</p>
<p>But many other languages don't allow this, so I go for the next best thing:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>start(appointment) <= now && now <= end(appointment)
</pre></div>
</div>
</div>
<p>And now… it was only while writing this that I realised: sometimes there is another great alternative, modelled after the following mathematical predicate:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>now ∈ [start(appointment), end(appointment)]
</pre></div>
</div>
</div>
<p>This can be translated directly to some languages too, such as this Ruby example:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>(start(appointment)..end(appointment)).include?(now)
</pre></div>
</div>
</div>
Map, then reducehttps://blog.pablobm.com/2018/02/27/map-then-reduce/2018-02-27T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<div class="update-notice">
<p>Added the <em>Appendix</em> section on <time>6 March 2018</time>.</p>
</div>
<p>I've been thinking about <code>map</code> and <code>reduce</code> lately. Nothing too erudite though. Something that I have realised is that it's sometimes too easy to conflate the two into just the <code>reduce</code>, when instead it might be clearer to have a very dumb <code>map</code> followed by a very dumb <code>reduce</code>.</p>
<p>Here's a simple example:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>[<span class="string"><span class="delimiter">'</span><span class="content">a</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">b</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">c</span><span class="delimiter">'</span></span>]
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> .reduce((acc, k) => ({...acc, [k]: []}), {});
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="comment">// => { a: [], b: [], c: [] }</span>
</pre></div>
</div>
</div>
<p>It's not obvious at first, but that <code>reduce</code> is doing a the job of a <code>map</code> as well as its own:</p>
<ol>
<li>Map: it generates new data for each element of the list (the empty arrays <code>[]</code>).</li>
<li>Reduce: it aggregates the data into a new piece of data (the resulting object).</li>
</ol>
<p>These can be separated as follows:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>[<span class="string"><span class="delimiter">'</span><span class="content">a</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">b</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">c</span><span class="delimiter">'</span></span>]
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> .map(k => [k, []])
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> .reduce((acc, [k,v]) => ({...acc, [k]: v}), {});
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="comment">// => { a: [], b: [], c: [] }</span>
</pre></div>
</div>
</div>
<p>This is longer to write, but I think it makes the <code>reduce</code> more readable by turning it into a common idiom, a "pairs to object" reduction if you will. With that in mind you can focus separately on the mapping function and possibly understand better what's being produced.</p>
<p>Functional programming tools can help me explain better. For example, using <a href="http://ramdajs.com/">Ramda</a> I could implement the above as follows:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>const createLists = R.compose(
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> R.fromPairs,
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> R.map(k => [k, []])
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>);
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>createLists([<span class="string"><span class="delimiter">'</span><span class="content">a</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">b</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">c</span><span class="delimiter">'</span></span>]);
<span class="line-numbers"><a href="#n6" name="n6">6</a></span><span class="comment">// => { a: [], b: [], c: [] }</span>
</pre></div>
</div>
</div>
<p>With the use of <code>R.fromPairs</code> in this example, we have turned the "pairs to object" idiom into a single, self-describing function invocation. Now that's easy to read. Compare to the Ramda version of the initial code:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>const createLists = R.reduce((acc, k) => ({...acc, [k]: [] }), []);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>createLists([<span class="string"><span class="delimiter">'</span><span class="content">a</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">b</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">c</span><span class="delimiter">'</span></span>]);
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="comment">// => { a: [], b: [], c: [] }</span>
</pre></div>
</div>
</div>
<p>I think now the separation between <code>map</code> and <code>reduce</code> becomes more apparent, and so does the benefit of enforcing it.</p>
<h2 id="appendix">Appendix</h2>
<p>After I first published this post, I discussed this topic with my friend <a href="http://rosario.io/">Rosario</a>. He reminded me that <code>map</code>, <code>filter</code>, <code>forEach</code>, and all those fellows are just special cases of <code>reduce</code>. For example:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>const reduce = (f, init, coll) => coll.reduce(f, init);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>const map = (f, coll) => reduce((a, b) => a.concat(f(b)), [], coll);
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>const filter = (f, coll) => reduce((a, b) => f(b) ? a.concat(b) : a, [], coll);
</pre></div>
</div>
</div>
<p>Thinking about this, I reached another conclusion: don't use <code>reduce</code> unless you must. It's possible that your library has already a function that will implement what you need. Some other special case of <code>reduce</code>, same as Ramda's <code>R.fromPairs</code> filled my reduction needs above. Use that instead. It will be easier to read and understand, both by means of saving you a few braces and brackets as well as by providing a more descriptive name.</p>
<p>And if you don't have such a reducer at hand, create one. A descriptively named function, tailored to your use case. Wrap the <code>reduce</code> in a named piece of code and use that instead of sticking it in a longer chain that may be already difficult enough to follow. Like that long sentence I just wrote.</p>
Ember Data: URIs for singular resourceshttps://blog.pablobm.com/2018/01/30/ember-data-singular-resources/2018-01-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<div class="update-notice">
<p><time>11 June 2018</time>: updated with some errors fixed. I did a terrible job the first time around.</p>
</div>
<div class="please-note">
<p><span class="please-note__note">NOTE:</span> I use the term "URI" here as I consider
it the correct one for the context, despite the API using the term "URL".
Maybe I'm just being pedantic, dunno…</p>
<p>For info on the difference between URL and URI, check out <a href="https://danielmiessler.com/study/url-uri">https://danielmiessler.com/study/url-uri</a>.</p>
</div>
<p>Coming from Ruby on Rails and other similar frameworks, I am used to the concept of "singular resources": resources that we can request from the API without providing a specific id, because it is implied in the request. So for example, these resource URIs:</p>
<ul>
<li><code>/users/123</code> is not a singular resource, because the URI is explicit as to which user resource we want: the one with id 123.</li>
<li><code>/user</code> is a singular resource, referring to the details of a user that is implied: probably the currently authenticated user.</li>
</ul>
<p>Ember Data doesn't support singular resources by default: you cannot use <code>findRecord</code> or similar to retrieve one of these out of the box. Fortunately, the provided adapters are easy to override to support these singular resources. Let's see how to do this.</p>
<p>First, a refresh: Ember Data ships by default with two adapters, <code>JSONAPIAdapter</code> (the default) and <code>RESTAdapter</code>, which implement two common protocols to communicate with JSON APIs. If your API communicates differently, both adapters provide hooks for you to override for your own particular case.</p>
<p>If you are going to define your own adapters, you should start by defining an <code>ApplicationAdapter</code>. This will be a common adapter for all your models that you can then override for each specific case. It can just be empty. This is just to provide a common ground for adapters specific to each model. Something like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment">/// app/adapters/application.js</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="reserved">import</span> JSONAPIAdapter from <span class="string"><span class="delimiter">'</span><span class="content">ember-data/adapters/json-api</span><span class="delimiter">'</span></span>;
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="reserved">export</span> <span class="keyword">default</span> JSONAPIAdapter.extend({
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>});
</pre></div>
</div>
</div>
<p>This example is actually equivalent to the Ember Data default as it is simply an extension of <code>JSONAPIAdapter</code>. I could have dispensed with the <code>extend</code> call since it's empty, but I like to have it, as I know I'll be adding my custom code. Also, I could have used <code>RESTAdapter</code> instead, if appropriate for my specific API.</p>
<p>Now, let's say that we have an Ember Data model <code>User</code>, and we want to fetch a singular resource of this type from the API. To do this, we'll create a custom adapter so that it retrieves this resource when we do the following:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>const singularUser = <span class="local-variable">this</span>.store.queryRecord(<span class="string"><span class="delimiter">'</span><span class="content">user</span><span class="delimiter">'</span></span>, {<span class="key">singular</span>: <span class="predefined-constant">true</span>});
</pre></div>
</div>
</div>
<p>The first thing to note is that I am using <code>queryRecord</code> instead of <code>findRecord</code>. You should use <code>findRecord</code> when you know the id of the resource you are requesting. This allows the store to retrieve this record from cache if appropriate. In this case, we do not know the id, and Ember Data's interface specifies that <code>queryRecord</code> is the correct API to use.</p>
<p>Also, we provide the <code>{singular: true}</code> argument. With this we signal to our custom adapter that we want the singular resource. Note that this is not something that Ember Data understands by default, but instead it is a convention between our custom adapter and its client code. I have seen codebases using other words instead of <code>singular</code>, such as <code>me</code> or <code>current</code>. Which one to use is up to you.</p>
<p>Now that we have clarified how we'll use our custom adapter, we'll build it. We can start by extending our <code>ApplicationAdapter</code> by overriding the <code>buildURL</code> hook. This is a method that the adapters use and is meant to be customised if we need to. It has several parameters, of which the following are useful to us:</p>
<ul>
<li><code>modelName</code>: a string specifying the model type we are requesting. In the example above, it will be <code>'user'</code>.</li>
<li><code>requestType</code>: this will be <code>'findRecord'</code>, <code>'findAll'</code>, <code>'queryRecord'</code>, etc. It tells us to know what type of request the client code requires.</li>
<li><code>query</code>: for the <code>'query'</code> and <code>'queryRecord'</code> request types, it will be the argument provided by the client code. In the case we are trying to implement, the value will be <code>{singular: true}</code>.</li>
</ul>
<p>With these three parameters, we can tell when our code is requesting a <code>'queryRecord'</code> for a "singular" resource, for any model type. Here's an implementation:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// app/adapters/application.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="reserved">import</span> JSONAPIAdapter from <span class="string"><span class="delimiter">'</span><span class="content">ember-data/adapters/json-api</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="reserved">export</span> <span class="keyword">default</span> JSONAPIAdapter.extend({
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> buildURL(modelName, id, snapshot, requestType, query) {
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="keyword">if</span> (requestType === <span class="string"><span class="delimiter">'</span><span class="content">queryRecord</span><span class="delimiter">'</span></span> && query && query.singular) {
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="keyword">return</span> <span class="string"><span class="delimiter">'</span><span class="content">/</span><span class="delimiter">'</span></span> + modelName;
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> } <span class="keyword">else</span> {
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="keyword">return</span> <span class="local-variable">this</span>._super(...<span class="local-variable">arguments</span>);
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> }
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> }
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>});
</pre></div>
</div>
</div>
<p>So when the client code is requesting a singular resource, return the URI for it. Otherwise, just call <code>this._super</code> and do as the adapter would normally do. Normally, this will return three types of URIs, depending on the request. For user resources, they will be these:</p>
<ul>
<li><code>/user</code> when given the agreed argument <code>{singular: true}</code> and a <code>queryRecord</code> request type.</li>
<li><code>/users/:id</code> when given an id and a <code>findRecord</code> request type.</li>
<li><code>/users</code> on a <code>findAll</code> request type.</li>
<li>And other combinations, but the main ones are the above.</li>
</ul>
<p>If you try the code, it should work now. There will be an interesting hitch though: your requests to the singular resource will also include the query parameter <code>?singular=true</code>. This is the default behaviour of <code>queryRecord</code>: sending the keys and values of the second argument as query params. In most cases, this is probably ok with your API as it will just ignore it. However, if you would rather not see that unseemly addition to the request, you can just do some more adapter overriding.</p>
<p>To do this, I can think of a couple of methods that could be overriden. My preference is to go for <code>sortQueryParams</code>, which is normally used to change the order of the query parameters (which is rarely necessary). We'd be cheating a bit here, because this method is intended to change the order and we are instead modifying the query altogether, by removing a key/value pair. I think it's still acceptable.</p>
<p>Having said that, this is a possible implementation:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// app/adapters/application.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="reserved">import</span> JSONAPIAdapter from <span class="string"><span class="delimiter">'</span><span class="content">ember-data/adapters/json-api</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="reserved">export</span> <span class="keyword">default</span> JSONAPIAdapter.extend({
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="comment">// ...</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> sortQueryParams(query) {
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> let newQuery = Object.assign({}, query);
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="keyword">delete</span> newQuery.singular;
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="keyword">return</span> newQuery;
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> },
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>});
</pre></div>
</div>
</div>
<p>I'm using <code>Object.assign</code> to make a copy of the original simply because I favour the "immutable" style of programming. I could just delete the key from the input argument and return it again. Do it the way it you like best.</p>
<p>Anyway, that's it. The adapters and serializers that Ember Data bundles by default have an extensive number of hooks that you can take advantage of, and their documentation has improved a lot over time. I recommend that you have a look at it.</p>
<p>Still, I actually learned all this by reading Ember Data's source code, before these hooks were so well documented. It's very easy to read, and a few <code>console.log</code> calls in the right places will show you what's actually going on when you interact with the library. Go try yourself, it's a better way to learn. If you find something that is not well documented, that's your chance to contribute ;-)</p>
38https://blog.pablobm.com/2018/01/07/38/2018-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<ul>
<li>Strontium</li>
<li>2<sup>2</sup> + 3<sup>2</sup> + 5<sup>2</sup></li>
</ul>
<p>I don't have a Ramanujan at hand, so I'm going to have to leave it at that…</p>
Keep it simple (express-session vs cookie-session)https://blog.pablobm.com/2017/12/10/keep-it-simple-express-session-vs-cookie-session/2017-12-10T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>If you are starting out a new webapp using <a href="http://expressjs.com/">Express</a>
on Node.js, do not use <a href="https://github.com/expressjs/session">express-session</a>
unless you really know what you are doing. Like: <strong>really</strong> know what you are doing
and why. In any other case, use <a href="https://github.com/expressjs/cookie-session">cookie-session</a>.</p>
<figure class="post-illustration post-illustration--side" style="max-width: 200px;">
<div class="post-illustration__images">
<img src="/images/http-cookie.png" />
</div>
<figcaption>
<p class="attribution">Original image by <a href="https://commons.wikimedia.org/wiki/File:Vector_Oreo.svg">Robbgodshaw</a></p>
<p class="license"><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/deed.en">CC BY-SA 3.0 License</a></p>
</figcaption>
</figure>
<p>The paragraph above may sound disparaging, but that's not where I'm going here.
There are perfectly good use cases for express-session. However it requires more
setting up than it may appear at first: you will have to integrate it with
a database or similar. It may appear to work as a drop-in at first, but this
is only because it defaults to using a memory store that won't work on production.</p>
<p>I bring this up because, recently, I was helping out a person who was learning
the ropes of web development and was using Express. Sessions were not working
correctly, expiring at random. Eventually I realised that they were using
express-session, thinking that it would just work after adding the package.
Dropping it in favour of cookie-session solved the issue.</p>
<p>Keep it simple. If you are just going through the first few iterations of
your new project, working towards an MVP, there's a lot of stuff you do not
need. It is tempting to see packages such as express-session, which are very
popular on Github and are very flexible, and you may think that your project should
use just that. However you should be wary of adding anything that has more
options than you need, and could add complexity that will slow you down in the end.</p>
Third party files on a custom Debian Live installerhttps://blog.pablobm.com/2017/11/19/third-party-files-on-a-custom-debian-live/2017-11-19T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Back in early 2016, I ran into a problem while customising a Debian
Live CD that I maintain. I wanted it to include a piece of
third-party, proprietary freeware, in a way that didn't break the terms
of the license, while keeping the process automated. Ultimately, I found
a solution that, while slightly overcomplicated, did the job
pretty nicely. I documented it at on an article titled
<a href="http://blog.pablobm.com/post/142965880780/large-files-on-a-custom-debian-installer"><em>Large files on a custom Debian installer</em></a>.</p>
<p>Earlier this year I revisited this problem, and I could find a much
simpler solution that involves fewer moving parts. This is the summary:</p>
<ol>
<li>Create a directory <code>config/includes.chroot/tmp</code> and put there the
files you require.</li>
<li>Make it so that your VCS ignores this directory (eg: add it to <code>.gitignore</code> if you use Git).</li>
<li>Write a config hook that accesses these files at
<code>/live-build/config/includes.chroot/tmp</code>, running installers,
copying them to the right location, or generally whatever it is
you need to do with them.</li>
</ol>
<p>And that's pretty much it. For a bonus, you could have a script that runs as
part of the build process and downloads the files for you. That would
help others who use your scripts later (or yourself, when you forget the
details a few weeks down the line).</p>
<p>For an example of this, check out the pull request I created in my project.
To run the buid I use a Makefile, where I added a dependency to show a message
with instructions when the third-party files are not in place:
https://github.com/pablobm/w2c3-livecd/pull/5</p>
Rebooting machines with Ansiblehttps://blog.pablobm.com/2017/08/05/rebooting-machines-with-ansible/2017-08-05T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>There are a few resources online explaining how to reboot a machine using
Ansible which didn’t work for me. My task would always time out and I had no
idea why. Finally I figured it out.</p>
<p>The tasks I was using looked roughly like these:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>---
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>- name: Restart machine
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> shell: shutdown -r now "Maintenance restart"
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> async: 0
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> poll: 0
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>- name: Wait for server to come back
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> local_action:
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> module: wait_for
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> host: {{ inventory_hostname }}
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> state: started
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> become: false
</pre></div>
</div>
</div>
<p>The problem here is the use of <code>inventory_hostname</code>. In my inventory, I was
referring to my machines by the name they had on my <code>.ssh/config</code>. This works
well when invoking Ansible, whose CLI integrates well with OpenSSH. However it
doesn’t work for modules, or at least it doesn’t for <code>wait_for</code> which I use
above.</p>
<p>After trying some alternatives, I eventually settled for having all the network
information on my inventory. This is, declaring <code>ansible_host</code> (and possibly
<code>ansible_port</code>) for each entry, instead of relying on <code>.ssh/config</code>. Then I
would use <code>ansible_host</code> in the <code>wait_for</code> task to indicate the host.</p>
<p>After some additional tweaking, currently I have a <code>reboot</code> role whose main task
looks like this:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>---
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>- name: Restart machine
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> shell: sleep 2 && shutdown -r now "Maintenance restart"
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> async: 1
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> poll: 0
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> ignore_errors: true
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>- pause:
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> seconds: 5
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>- name: Waiting for server to come back
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> local_action:
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> module: wait_for
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> host: '{{ ansible_host }}'
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> port: '{{ ansible_port }}'
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> state: started
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> delay: 10
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> timeout: 60
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> become: false # as this is a local operation
</pre></div>
</div>
</div>
<p>Why <code>sleep 2</code>, <code>async: 1</code> and <code>poll: 0</code>? I have no idea. I have tried a few
things and this is the one that appears to work reliably for me. For now, I’m
sticking with it, until I understand all this a bit better.</p>
Timetable for (most) any concert in Londonhttps://blog.pablobm.com/2017/07/28/timetable-for-most-any-concert-in-london/2017-07-28T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>I find it funny, how concerts in London tend to run like clockwork. So much so that they rarely stray from this schedule:</p>
<table>
<thead>
<tr>
<th style="text-align: right">Time</th>
<th style="text-align: left"> </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">19:00</td>
<td style="text-align: left">Second support starts</td>
</tr>
<tr>
<td style="text-align: right">19:30</td>
<td style="text-align: left">Second support ends</td>
</tr>
<tr>
<td style="text-align: right">19:45</td>
<td style="text-align: left">Main support starts</td>
</tr>
<tr>
<td style="text-align: right">20:30</td>
<td style="text-align: left">Main support ends</td>
</tr>
<tr>
<td style="text-align: right">21:00</td>
<td style="text-align: left">Headliner starts</td>
</tr>
<tr>
<td style="text-align: right">22:30</td>
<td style="text-align: left">Headliner ends</td>
</tr>
</tbody>
</table>
<p>Doors normally open at 7, so it's not unusual that you have missed most of the first act by the time you enter the venue. Fortunately, some venues do open earlier (6pm) if there's going to be a second support.</p>
<p>In any case: super handy to plan around.</p>
`WARNING: server ‘gpg-agent’ is older than us`https://blog.pablobm.com/2017/05/30/warning-server-gpg-agent-is-older-than-us/2017-05-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>I ran into the following error this morning, using the
<a href="https://www.passwordstore.org/">Pass</a> utility:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>$ pass -c foo/bar
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>gpg: starting migration from earlier GnuPG versions
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>gpg: WARNING: server 'gpg-agent' is older than us (2.0.30 < 2.1.21)
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>gpg: error: GnuPG agent version "2.0.30" is too old.
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>gpg: Please make sure that a recent gpg-agent is running.
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>gpg: (restarting the user session may achieve this.)
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>gpg: migration aborted
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>gpg: decryption failed: No secret key
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>There is no password to put on the clipboard at line 1.
</pre></div>
</div>
</div>
<p>From what I’ve seen online, you can also see this problem using GPG by means
other than through Pass. In any case, I couldn’t find a fix, and I was starting
to be worried I couldn’t access my passwords any more, or at least not easily.</p>
<p>Fortunately, I could figure out what was going on by reading the error closely.
Looks like there was a version of <code>gpg-agent</code> running that was different from
the one expected by GnuPG. Also, some migration was expected to occur between
the old and the new version. Therefore I had to shut down the old version so
that this migration could take place:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ gpgconf --kill gpg-agent
</pre></div>
</div>
</div>
<p>Then I ran my command again. This time the underlying <code>gpg</code> command runs its
migration successfully and then allows Pass to resume as if nothing had
happened:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ pass -c foo/bar
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>gpg: starting migration from earlier GnuPG versions
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>gpg: porting secret keys from '/Users/pablobm/.gnupg/secring.gpg' to gpg-agent
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>gpg: migration succeeded
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>Copied foo/bar to clipboard. Will clear in 45 seconds.
</pre></div>
</div>
</div>
Detect if a number has decimalshttps://blog.pablobm.com/2017/05/28/detect-if-a-number-has-decimals/2017-05-28T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>You have a float variable, and you want a quick boolean expression to tell if the value has any decimals. To put it differently, to tell whether a float variable is currently holding an integer value or not.</p>
<p>Divide by 1, check if the remainder is 0:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>number % <span class="integer">1</span> == <span class="integer">0</span> <span class="comment"># => true for integer, false for rational</span>
</pre></div>
</div>
</div>
<p>Which makes perfect sense if you think of it. If you divide 9.5 apples among 5 children, the remainder is how many you couldn't split: 4.5. If you divide the apples among 1 children, in a strict sense there's still 0.5 you can't "split". The versions with negative operands follow from there.</p>
Messing up at workhttps://blog.pablobm.com/2017/04/30/messing-up-at-work/2017-04-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Many years ago, I wrote (bad) code that sent a single marketing email,
repeatedly, to addresses in a subscribers list. Imagine your inbox filling up
with copies of the same email because some idiot got their code wrong. On the
bright side, I realised of what I had done quickly enough that most recipients
only received 4 copies of the email (I think). At the time, I had 3 years of
experience as a backend web developer.</p>
<h2 id="how-did-you-even-manage-that">How did you even manage that?</h2>
<p>These were the ingredients:</p>
<ul>
<li>A web application that, among other things, sent marketing emails.</li>
<li>At an admin’s request, a background process was created that sent all
emails.</li>
<li>The process would loop through a subscribers list and send an email to each
entry.</li>
<li>The background processing library would re-run a job if an exception was
raised at any point during its execution.</li>
<li>At least one address in the subscribers list was malformed enough to cause
the email library to raise an exception.</li>
</ul>
<p>So there you are. Email sending job runs, fails halfway through the list, tries
again from the very beginning. As a result, all addresses that got an email will
get another one. Neat (not).</p>
<h2 id="what-should-have-happened-instead-technically">What should have happened instead (technically)</h2>
<p>Some techniques that would avoided this spring to mind:</p>
<ul>
<li>Make sure to rescue exceptions when sending each email.</li>
<li>Create one background process for each email in the list, so that one
failure doesn’t affect the rest.</li>
<li>Rescue exceptions for the whole thing just in case.</li>
</ul>
<p>And of course, write specs/tests to make sure things are working the way you
expect. All pretty reasonable, really.</p>
<h2 id="whose-fault-was-it">Whose fault was it?</h2>
<p>It’s natural to feel bad when you mess up building software. However, software
engineering is fraught with difficulty and cannot be one person’s job. When this
happened, I had written all code myself as I was pretty much the Software
Development Department at that job. There were no standup meetings, pair
programming or code review because there wasn’t anyone I could have them with. I
was told what was needed, and I implemented and deployed it. This made me a
single point of failure.</p>
<p>When individuals become a single point of failure, the mistake has already been
made. Humans are not perfect and will make mistakes.</p>
<h2 id="aftermath">Aftermath</h2>
<p>Fortunately for me, my line manager reacted pretty well and was understanding.
Other people, directly affected by this, were less impressed, but I didn’t need
to worry about that too much.</p>
<p>Now, it’s easy to invoke the ghost of Imposter’s Syndrome, and say that we
needen’t worry about our ability and should simply keep going. It worked out for
me at the time, but I wonder what could have happened, or what has and will
still happen to other people in similar circumnstances but in a less favourable
environment.</p>
<p>Also: a moment to check my privilege. I’m a white male. Consider how many of my
lot out there may have screwed up in a similar fashion with no repercussions.
Consider how many in a different demographic may have been penalised after a
similar event, because of a bias unconscious or otherwise.</p>
<h2 id="what-this-can-learn-us">What this can learn us</h2>
<h3 id="its-not-your-fault">It’s not your fault</h3>
<p>If you get in trouble for something like this, start looking for a new job: that
environment is not conducive to your growing up as a professional or a person.
Having said that, I understand this is easier said than done. Not everyone
enjoys circumstances where making this jump is comfortable, or even possible,
regardless of their ability.</p>
<p>You might beat yourself up about it. Don’t. Share it with other people, both
your loved ones and peers in the industry (who may be one and the same!). A
local tech meetup can be a good place to exchange experiences and find that
other people also have their own botch-up stories.</p>
<h3 id="find-out-how-others-would-deal-with-this">Find out how others would deal with this</h3>
<p>Experiences like this make for a good interview question, for both sides of the
conversation:</p>
<ul>
<li>As an interviewer: tell us about a time you messed up. What did you learn
from it?</li>
<li>As an interviewee: tell me about a time when somebody messed up here. How
was that deal with?</li>
</ul>
<h3 id="fix-the-process-not-the-people">Fix the process, not the people</h3>
<p>If something like this happens in your watch, ask yourself: how did the
development process fail? What should be changed to avoid it repeating? What can
the team learn from this experience?</p>
<h2 id="in-closing">In closing</h2>
<p>You could read this as “There ain’t no such thing as individual failure in a
software development team”. TANSTAIFIASDT. Catchy!</p>
Your VPN can be an attack vectorhttps://blog.pablobm.com/2017/02/12/your-vpn-can-be-an-attack-vector/2017-02-12T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>If you use a VPN, be aware that it may not be filtering inbound ports.
Effectively, this opens up your computer to port scanning and attacks on
vulnerable network services. In this scenario, it doesn’t matter if you are
behind a NAT: the VPN virtually grants you a public IP address reachable by all.</p>
<p>A few months ago I had a simple HTTP server open on port 8000 of one of my
machines. At some point, I noticed that my logs listed requests from machines
outside my network. These are some examples:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>78.225.110.184 - - [21/Oct/2016 22:08:12] code 400, message Bad request syntax ('\xca\xe6f\x89\xc4\xa8\xbc\xc6\x8d^\x9b\x14\xa1X\xb3x\xa3\xf9o`9\x0c\xd6\xdcY_\xee\x1d\xec4\xe9\x8d4\xa5\xb7\x98{6\xb5\x18\xe0J\xee\x1d\xfcFWy\x1650\xa4H\x10\xe8\xb0\xa0\xc7RS \xd1\x1b\xe6\xbf2[\xa8\xb1\x9c$\xc5&4\xf4\x7f\x06\xa8x\xf0K\x17\xaf\xdbe\xf3M\xa9\xd5\x7f~\x9f_ \x0c\x92\r\xd5`\x97D"y\xb5\xf6"\x1f\x13:\t\x0b\x05*\xee\x0f\xd2\xab\xdf\xeb0\xa4\xa41\xf2\x9d\xdb%I\xbd\x8bh\x19\xf0M\xc0\x1b\xf5\x86E\x9eF\xcc\xed\xce1\xaa%"D\'\xf4\xad\xee\xc3\r\x8f\xa0\xb1\xe0Ji8\x0b\xf6\x999[71\xc0\xbf\xc4\xc0\xc4\xee\x9b\x8c\xae\x8bH3\xd1*\xa6T\x18\xd26NK\x8e\x94\xcc_\x95\xc9.\xfd\xa87\xe3\x1a\xb6\xed\x8b\xf0A\x83N\x0f\x1e?\t\xcd\x15\x08\x0bJ\x99\xd4\xfa\xbb\x18\xbc\x7f\x0fW\xccy\xdfG\xb6\x03\x03\x96\x8e\xcd\xab\xb0v2\xa3\x0f\xd9*q>\t\t\xb0\xac\xf3\x07\x80\x13E&\xa6\t')
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>23.125.107.154 - - [22/Oct/2016 20:37:09] code 404, message File not found
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>23.125.107.154 - - [22/Oct/2016 20:37:09] "GET /w00tw00t.at.ISC.SANS.test0:) HTTP/1.1" 404 -
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>123.151.42.61 - - [23/Oct/2016 10:26:37] "GET http://www.baidu.com/ HTTP/1.1" 404 -
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span>218.93.206.27 - - [23/Oct/2016 16:41:20] code 400, message Bad request version ('0\xf6\xdb\x00\xbd\x00\x00p\xc00\xc0,\xc02\xc0.\xc0/\xc0+\xc01\xc0-\x00\xa3\x00\x9f\x00\xa2\x00\x9e\xc0(\xc0$\xc0\x14\xc0')
</pre></div>
</div>
</div>
<p>There were more than those, most of them looking very much like HTTP
vulnerability probes of all sorts. All these requests had me baffled for a
while. I was convinced that my network had been breached somehow; maybe a router
misconfiguration. Eventually I realised that the source of these requests was my
VPN.</p>
<p>I use <a href="http://www.ipredator.se">IPredator</a> often. They provide a pretty reliable
service I’m happy with. However there are details like this one that are not
that obvious and can bring new trouble that you didn’t expect. Security is
annoyingly difficult to get right!</p>
A simple asset pipeline with Broccoli.jshttps://blog.pablobm.com/2017/01/22/a-simple-asset-pipeline-with-broccolijs/2017-01-22T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<div class="update-notice">
<p>Updated on <time>15 October 2017</time> to support Babel 6.</p>
</div>
<p>I've been doing some research on how to set up an asset
pipeline using Broccoli, which is part of the toolset
provided by Ember.js. <a href="http://broccoli.js">The official website</a>
shows a good example of use, but I wanted to do something
a bit more advanced. Here's the result.</p>
<p>At the end of this text, we'll have an asset pipeline
able to read these inputs:</p>
<ul>
<li>ES6-flavoured JavaScript modules</li>
<li>JavaScript packages from NPM</li>
<li>Sass files</li>
</ul>
<p>And generate these outputs:</p>
<ul>
<li>A single JavaScript file, in the dialect more commonly
understood by contemporary browsers</li>
<li>A single CSS file</li>
</ul>
<p>I will be using Yarn instead of NPM, because it will
create fewer headaches down the road. Also, it's 2017,
happy new year!</p>
<h2 id="basic-setup">Basic setup</h2>
<p>Broccoli works as a series of filters that can be applied
to directory trees. The pipeline is defined on a file named
<code>Brocfile.js</code>, which at its minimum expression would look
something like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>module.exports = <span class="string"><span class="delimiter">'</span><span class="content">src/html</span><span class="delimiter">'</span></span>;
</pre></div>
</div>
</div>
<p>A "Brocfile" is expected to export a Broccoli "node", which
is a sequence of transforms over a directory tree. The
simplest possible example would be just a string
representing a filesystem path, so the above does the job.
We could read it as "the output of this build is a copy of
the contents of the <code>src/html</code> directory".</p>
<p>Note that I say Broccoli "nodes". There's a lot of
literature out there referring to Broccoli nodes as
Broccoli "trees". It's the same thing, but "node" seems
to be the currently accepted nomenclature, while "tree"
is deprecated.</p>
<h2 id="running-a-build">Running a build</h2>
<p>We have a very simple Brocfile. Let's run it and see its
result. We need the Broccoli CLI and libraries for this,
so let's first create a Node project, then add the required
dependencies:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn init -y
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ yarn add broccoli broccoli-cli
</pre></div>
</div>
</div>
<p>Then we add the following entry to our <code>package.json</code>:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>"scripts": {
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> "build": "rm -rf dist/ && broccoli build dist"
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>}
</pre></div>
</div>
</div>
<p>Now we can run the build process any time with this
command:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn run build
</pre></div>
</div>
</div>
<p>When we run this command, we run a Broccoli build. Since
we are not doing much at the moment, it will simply copy
the contents of the <code>src/html</code> directory into <code>dist</code>. If
<code>dist</code> exists already, our build script deletes it first,
as Broccoli would refuse to write into an existing one.</p>
<p><strong>Did you get an error?</strong> No problem, that's probably
because you didn't have a <code>src/html</code> directory to read
from. Create one and put some files on it. Then you'll be
able to confirm that the build process is doing what it is
expected to do.</p>
<div class="please-note">
<p><span class="please-note__note">NOTE:</span> working with Node/NPM, it's common to see examples
that install a CLI tool (broccoli-cli in this case)
globally using <code>npm install -g PACKAGE_NAME</code>. Here
we avoid this by installing it locally to the project
and then specifying a command that uses it in the <code>scripts</code>
section of <code>package.json</code>. These commands are aware of
CLI tools in our local <code>node_modules</code>, allowing us to
keep eveything tidier, and locking the package version of
the CLI tool along with those of other packages.</p>
</div>
<h2 id="using-plugins">Using plugins</h2>
<p>Most transforms we can think of will be possible using
Broccoli plugins. These are modules published on NPM that
allow us to transpile code, generate checksums, concatenate
files, and generally do all the sort of things we need to
produce production-grade code.</p>
<p>Now, in the first example above we referred to a Broccoli
node using the string <code>src/html</code>, meant to represent the
contents of the directory of the same name. While this
will work, using a string this way is now discouraged.
Current advice is to instead use broccoli-source, which is
the first of the plugins that we will use in this
walkthrough. Let's install it:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn add broccoli-source
</pre></div>
</div>
</div>
<p>Now we can <code>require</code> it into our Brocfile and use it. I'm
going to use variables in this example to start giving
this pipeline some structure:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> source = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-source</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="keyword">var</span> WatchedDir = source.WatchedDir;
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="keyword">var</span> inputHtml = <span class="keyword">new</span> WatchedDir(<span class="string"><span class="delimiter">'</span><span class="content">src/html</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="keyword">var</span> outputHtml = inputHtml;
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>module.exports = outputHtml;
</pre></div>
</div>
</div>
<p>If we run the build, we'll get exactly the same result as
before. We needed more code to get the same thing, but this
prepares us for things to come, and follows best practices.</p>
<h2 id="the-development-server">The development server</h2>
<p>In the previous example, we referred to the input HTML as a
<code>WatchedDir</code>. This suggests that, similarly to other build
tools, Broccoli includes a development server that will
"watch" the input files, running a build automatically when
we save any changes. Let's create a command for this on our
<code>packages.json</code> file, adding a new entry to the <code>scripts</code>
section:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>"scripts": {
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> "build": "rm -rf dist/ && broccoli build dist",
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> "serve": "broccoli serve"
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>},
</pre></div>
</div>
</div>
<p>Now we can start the development server with:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn run serve
</pre></div>
</div>
</div>
<p>Assuming you have a file called <code>index.html</code> in your
<code>src/html</code> directory, you should see it at the URL
http://localhost:4200. If the file changes, you can simply
refresh the page and the changes will appear without you
having to explicitly run the build.</p>
<h2 id="adding-a-css-pre-processor">Adding a CSS pre-processor</h2>
<p>So far this isn't very exciting. The development server is
just showing copies of the HTML files in our project.
Let's add a proper transform.</p>
<p>For this we can use a CSS pre-processor. For example, we
can install the Sass plugin:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>yarn add broccoli-sass
</pre></div>
</div>
</div>
<p>Require it at the start of our Brocfile:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> sass = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-sass</span><span class="delimiter">'</span></span>);
</pre></div>
</div>
</div>
<p>And add it to our pipeline on the same file:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> inputStyles = <span class="keyword">new</span> WatchedDir(<span class="string"><span class="delimiter">'</span><span class="content">src/styles</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="keyword">var</span> outputCss = sass([inputStyles], <span class="string"><span class="delimiter">'</span><span class="content">index.scss</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">index.css</span><span class="delimiter">'</span></span>, {});
</pre></div>
</div>
</div>
<p>This example will:</p>
<ul>
<li>read files from <code>src/styles</code>.</li>
<li>start processing from the file <code>index.scss</code>, which must
be in the first node given in the first argument.</li>
<li>leave the result in a file called <code>index.css</code> in the
output location.</li>
</ul>
<p>There's a problem now. We have an HTML pipeline and a
SaSS pipeline. We have to merge the two into a single
result.</p>
<h2 id="merging-broccoli-nodes">Merging Broccoli nodes</h2>
<p>When you have several sources of code, to be treated in
different ways, you get separate Broccoli nodes. Let's
merge the ones we have into a single one. Of course for
this we need a new plugin:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn add broccoli-merge-trees
</pre></div>
</div>
</div>
<p>Now we can perform the merge and export the result:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> MergeTrees = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-merge-trees</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="comment">// ...process nodes...</span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>module.exports = <span class="keyword">new</span> MergeTrees(
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> outputCss,
<span class="line-numbers"><a href="#n7" name="n7">7</a></span> outputHtml,
<span class="line-numbers"><a href="#n8" name="n8">8</a></span>);
</pre></div>
</div>
</div>
<p>Now ensure that your HTML points to the produced CSS, which
in the above example we have called <code>index.css</code>. Reload
the develpment server and check the results.</p>
<h2 id="from-modern-js-to-one-that-browsers-understand">From modern JS to one that browsers understand</h2>
<p>All that was quite easy. Dealing with JavaScript took some
more figuring out for me, but eventually I got there.
Here's my take on it.</p>
<p>We are going to transform some ES6 files into a more
browser-friendly flavour of JavaScript. For this, we need
<a href="https://babeljs.io/">Babel</a>, and there's a Broccoli plugin that provides it for
us. We start by installing the appropriate package, as well as as
a Babel plugin that provides the transform we need:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn add broccoli-babel-transpiler babel-preset-env
</pre></div>
</div>
</div>
<p>And now we alter our <code>Brocfile.js</code> to look like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="keyword">var</span> babelTranspiler = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-babel-transpiler</span><span class="delimiter">'</span></span>);
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="comment">// ...etc...</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="keyword">var</span> BABEL_OPTIONS = {
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="key">presets</span>: [
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> [<span class="string"><span class="delimiter">'</span><span class="content">env</span><span class="delimiter">'</span></span>, {
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> <span class="key">targets</span>: {
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="key">browsers</span>: [<span class="string"><span class="delimiter">'</span><span class="content">last 2 versions</span><span class="delimiter">'</span></span>],
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> },
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> }],
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> ],
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>};
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="keyword">var</span> inputJs = <span class="keyword">new</span> WatchedDir(<span class="string"><span class="delimiter">'</span><span class="content">src/js</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n16" name="n16">16</a></span><span class="keyword">var</span> outputJs = babelTranspiler(inputJs, BABEL_OPTIONS);
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>
<span class="line-numbers"><a href="#n18" name="n18">18</a></span><span class="comment">// ...etc...</span>
<span class="line-numbers"><a href="#n19" name="n19">19</a></span>
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span>module.exports = <span class="keyword">new</span> MergeTrees(
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> outputCss,
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> outputHtml,
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> outputJs,
<span class="line-numbers"><a href="#n24" name="n24">24</a></span>);
</pre></div>
</div>
</div>
<p>The <code>BABEL_OPTIONS</code> argument can be used to tell Babel what platforms
its output should target. In this case, we specify that we want code
compatible with the last 2 versions of current browsers. You can find the list of supported browsers at <a href="https://github.com/ai/browserslist#browsers">https://github.com/ai/browserslist#browsers</a>.</p>
<p>Write some JavaScript that uses modern features of the language,
and put it in <code>src/js</code>, then check the
results. Remember to restart the dev server and reference
the JS files from your HTML. The output will consist of
files of the same name as those in the input, but converted
to JavaScript compatible with current browsers.</p>
<div class="please-note">
<p><span class="please-note__note">NOTE:</span> in previous versions
of this guide, we didn't need <code>BABEL_OPTIONS</code>, as Babel's default
behaviour was good enough for us. Since version 6 of Babel, we need
to be more explicit at to what exactly we want, and this new argument
is now required.</p>
</div>
<h2 id="local-javascript-modules">Local JavaScript modules</h2>
<p>The one thing Babel is not doing there is handling
module imports. If your project is split into several
modules, and you use <code>import</code> in them, these lines will
have been transpiled into <code>require</code> lines but these won't
actually work on a browser. Browsers can't handle
JavaScript modules natively, so we will need a new step
that will concatenate all files into a single one, while
respecting these module dependencies.</p>
<p>I have figured out a couple of ways of doing this, so I'll
explain the one I like best. First we are goint to need
a new Broccoli plugin:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn add broccoli-watchify
</pre></div>
</div>
</div>
<p>Watchify is a wrapper around Browserify. In turn,
Browserify reads JavaScript inputs, parses them, finds
any <code>require</code> calls, and concatenates all dependencies
into larger files as necessary.</p>
<p>Let's update the lines of our Brocfile that dealt with JS
to look as follows:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> babelTranspiler = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-babel-transpiler</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="keyword">var</span> watchify = require(<span class="string"><span class="delimiter">'</span><span class="content">broccoli-watchify</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="comment">// ...</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span><span class="keyword">var</span> inputJs = <span class="keyword">new</span> WatchedDir(<span class="string"><span class="delimiter">'</span><span class="content">src/js</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n7" name="n7">7</a></span><span class="keyword">var</span> transpiledJs = babelTranspiler(inputJs, BABEL_OPTIONS);
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="keyword">var</span> outputJs = watchify(transpiledJs);
</pre></div>
</div>
</div>
<p>The <code>watchify</code> transform assumes that you will have a file
<code>index.js</code> that is the entry point of your JavaScript code.
This will be its starting point when figuring out all
dependencies across modules. The final product, a single
JavaScript file with all required dependencies
concatenated, will be produced with the name
<code>browserify.js</code>.</p>
<p>Note that imports are expected to use <strong>relative paths</strong> by
default. This is, the following won't work as it uses an
absolute path:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="reserved">import</span> utils from <span class="string"><span class="delimiter">'</span><span class="content">utils</span><span class="delimiter">'</span></span>;
</pre></div>
</div>
</div>
<p>But this will (assuming the module <code>utils</code> lives in the
same directory as the one doing the import):</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="reserved">import</span> utils from <span class="string"><span class="delimiter">'</span><span class="content">./utils</span><span class="delimiter">'</span></span>;
</pre></div>
</div>
</div>
<p>That is the default behaviour. If you use different
settings, you can pass some options in. For example, say
that you want Browserify to:</p>
<ul>
<li>Use a file called <code>app.js</code> as entry point</li>
<li>Put the results in a file called <code>index.js</code></li>
</ul>
<p>To achieve this, you invoke it with these options:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> outputJs = watchify(transpiledJs, {
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="key">browserify</span>: {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="key">entries</span>: [<span class="string"><span class="delimiter">'</span><span class="content">app.js</span><span class="delimiter">'</span></span>]
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> },
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="key">outputFile</span>: <span class="string"><span class="delimiter">'</span><span class="content">index.js</span><span class="delimiter">'</span></span>,
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>});
</pre></div>
</div>
</div>
<h2 id="using-modules-from-npm">Using modules from NPM</h2>
<p>The best thing about Browserify though, is that it can
pull NPM modules into your project. For example, say you
want to use jQuery. First you have to fetch it from NPM:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ yarn add jquery
</pre></div>
</div>
</div>
<p>Then you would import it in a module in your own code:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="reserved">import</span> <span class="predefined">$</span> from <span class="string"><span class="delimiter">'</span><span class="content">jquery</span><span class="delimiter">'</span></span>;
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span><span class="comment">// ...</span>
</pre></div>
</div>
</div>
<p>And finally you tell the Watchify plugin where it can find
it, passing an option pointing to your local <code>node_modules</code>
as a valid place to pull modules from:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="keyword">var</span> outputJs = watchify(transpiledTree, {
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="key">browserify</span>: {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="key">entries</span>: [<span class="string"><span class="delimiter">'</span><span class="content">index.js</span><span class="delimiter">'</span></span>],
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="key">paths</span>: [__dirname + <span class="string"><span class="delimiter">'</span><span class="content">/node_modules</span><span class="delimiter">'</span></span>],
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> },
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>});
</pre></div>
</div>
</div>
<p>In this example, jQuery will be pulled into the final file,
where your code can use it freely.</p>
<div class="please-note">
<p><span class="please-note__note">NOTE:</span> even though by default it expects <code>index.js</code> as
entry file, I have noticed sometimes watchify (or
browserify, or the plugin, or something), doesn't work
correctly if we pass options and don't specify the
<code>entries</code> value. Therefore, I recommend always including
it.</p>
</div>
<h2 id="a-complete-example">A complete example</h2>
<p>I have a GitHub repo that I'm using to experiment with
build tools. At the time of writing, there are two working
Broccoli examples that you can check out. I may add others
in the future, as well as examples with other tools.</p>
<p>Check it out at <a href="https://github.com/pablobm/build-tools-research">pablobm/build-tools-research</a>. I hope you find it useful.</p>
37https://blog.pablobm.com/2017/01/07/37/2017-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>Permutable, lucky, unique prime</li>
<li>Hexagon and Star</li>
<li>Human warmth</li>
<li>Rubidium</li>
<li>My turn</li>
</ul>
I choose you, Ember.jshttps://blog.pablobm.com/2016/11/28/i-choose-you-emberjs/2016-11-28T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>There are too many JavaScript frameworks out there, and I am not a JavaScript
expert. I have neither the time nor the inclination to put all them to the test
and come up with rational, experience-based arguments as to which one is
superior, be it in general or for a specific task. Similarly, if I look for
opinions of experts, I am always going to find articles praising one over the
others, in all sorts of situations and contexts. Therefore, when the time came
for me to choose one of them to use, I had to rely on different metrics.</p>
<p><strong>My choice was Ember.js</strong>, and these are the reasons for my decision.</p>
<h2 id="a-community-effort">A community effort</h2>
<p><strong>Ember is a fully community-directed effort.</strong> Several companies, large and
small, with competing interests are involved in its development. This is in
contrast with React.js, developed mainly by Facebook, or Angular.js, developed
mainly by Google.</p>
<p>I do have a small, yet significant concern that these frameworks will make
progress in accordance with the desires of their primary backers, rather than
those of the community. In other words: this opens the door for behaviour,
features, or simply general direction that benefit the use cases of a single
company, against those of their users outside of it. This is a concern I do not
have with Ember.</p>
<p>(There’s also the matter of <a href="http://lu.is/blog/2016/10/31/reacts-license-necessary-and-open/">whether React’s license is open source or
not</a>, good or
bad, but I’m not going to go in there).</p>
<h2 id="the-development-process">The development process</h2>
<p>In a related point, <strong>Ember’s development process is extraordinarily open and
deliberate</strong>. Every substantial change requires going through an <a href="https://github.com/emberjs/rfcs">RFC
process</a> where the initiator must not only
present their case, but also list possible drawbacks and alternatives. This
greatly increases the chances of all possible use cases being considered for the
benefit of all users.</p>
<h2 id="convention-over-configuration">Convention over Configuration</h2>
<p><strong>Ember follows the principle of <a href="https://en.wikipedia.org/wiki/Convention_over_Configuration">Convention over
Configuration</a>.</strong>
Some time ago, I tried out Angular.js, and one of the sensations I had was that
I didn’t know where stuff went; how to properly organise a project. As a
long-time user of Ruby on Rails, I appreciate a framework being opinionated. It
removes from my mind these small worries, while letting me focus on how to work
on the actual problems that the project was created to solve. It enables new
members of the project to quickly understand it, as the structure will be
similar to that of other projects with the same framework. It provides tested,
stable solutions for common problems, supported by established best practices in
the community.</p>
<p>I have met people who felt otherwise, and wanted more control over small and
purely technical details of the software they were building. I think that is not
control, but an illusion of it. At the end of the day, you will end up having to
build the same foundations that an opinionated framework would have provided in
the first place: you are bound to reinvent the wheel… poorly.</p>
<h2 id="stability-without-stagnation">Stability without Stagnation</h2>
<p><strong>Ember commits to provide Stability without Stagnation.</strong> The JS community is
too used to having to reinvent itself every year, throwing away yesterday’s
tools and rebuilding everything at the whim of today’s new fad. Ember promises
to offer clearly-defined periods of backwards compatibility combined with clear
upgrade paths. In the words of co-creator Tom Dale:</p>
<blockquote class="block-quote">
<p>The Ember community works hard to introduce new ideas
with an eye towards migration. We call this “stability
without stagnation”, and it’s one of the cornerstones of
the Ember philosophy.</p>
<footer class="block-quote__cite">
<p>— Tom Dale, <cite><a href="https://github.com/emberjs/rfcs/pull/15">Ember RFC #15 (The Road to Ember 2.0)</a></cite></p>
</footer>
</blockquote>
<p>An example of this is the introduction of <a href="http://emberjs.com/blog/2016/02/25/announcing-embers-first-lts.html">LTS (Long Term Support)
releases</a>.
Development teams have better things to do than upgrading their framework every
six weeks (the length of Ember’s release cycle). To avoid this, LTS releases
allow teams to stick to versions for longer, while still getting support (ie:
bugs being addressed) and increased attention to upgrade paths.</p>
<h2 id="not-everything-is-rosy">Not everything is rosy</h2>
<h3 id="before-order-there-was-chaos">Before order, there was chaos</h3>
<p><strong>Before Stability without Stagnation,</strong> there was a fair amount of confusion.
Not sure when this changed, but at some point before version 1.13 order was
restored, or rather formally instituted for the first time. Until then, changes
were happening too rapidly and without that much fanfare. Each new version could
bring a breaking change that would affect your app. It affected me a bit, but I
was lucky to arrive towards the end of this previous era, and with small apps
that were relatively simple to upgrade, so I made it through unscathed.</p>
<p>Lessons were learned and good overcame evil, but some of the fallout remains.
One manifestation of this is that I must <strong>filter Ember-related searches to
content from 2015 onwards</strong>. Anything older than that is very likely to refer to
abandoned practices and interfaces. Remember to do this when you search for
resources.</p>
<h3 id="documentation-can-improve">Documentation can improve</h3>
<p><strong>Ember’s official documentation wasn’t always up to scratch.</strong> Fortunately,
this has also been addressed with the creation of the <a href="http://emberjs.com/blog/2016/05/19/introducing-subteams.html#toc_learning">Learning
sub-team</a>,
tasked not only with improving the documentation, but also generally make Ember
easier for users old and new.</p>
<p>There’s still some way to go here though. I regularly have conversations with
Ember users where we agree that certain idioms are not well explained, and
sometimes there’s confusion as to the best way to implement certain common
patterns, or make known parts of Ember work together. Still, I have seen great
improvement in the last year, and I have high hopes in this venue.</p>
<h3 id="ember-is-large">Ember is large</h3>
<p>As built on my laptop right now (using Ember 2.9.1), <strong>the framework portion of
Ember.js weighs 655.62 KB (175.49 KB gzipped)</strong>. that’s a big download for the
JS of your website, and about 3x-4x the size of React’s according to comparisons
online.</p>
<p>In its defence, Ember does more than React, and when using the latter you’d need
to incorporate a large number of third-party packages in order to get the
functionality that you’d normally get by default with Ember. And sure, you may
not need some pieces, but it does help your process if you don’t have to think
about which plugin to choose when you are faced with a decision on how to fill a
functionality gap that Ember would have already provided.</p>
<p>Not only that: Ember is en route to provide a solution for this problem too. It
won’t be tomorrow, but it will arrive. There is an active effort, formalised in
the form of several RFCs, to reduce the weight of your Ember builds. One such
feature will be present in Ember 2.10 (slated for November 2016): lazy engines.
This is the ability to split your application into smaller applications that
will only be loaded if the user actually visits them, all with minimal
configuration on the developer’s part. More is expected to come, such as dead
code elimination or removal of old APIs (within the constraints of Stability
without Stagnation).</p>
<h2 id="drinking-the-kool-aid">Drinking the Kool-Aid</h2>
<p>This is the Kool-Aid I have decided to drink. From the information available to
me, it does appear to be the best, most sensible choice. I am hopeful it will
be. Of course I can be wrong, but I have to work with what I have.</p>
Palm-mute on GarageBandhttps://blog.pablobm.com/2016/10/19/palm-mute-on-garageband/2016-10-19T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Playing around with GarageBand recently, I had a metal guitar track (a software
instrument, not recorded), and I wondered how to make it sound like it was palm
muted.</p>
<p>There are many tutorials on YouTube on many GarageBand-related topics, but I
couldn’t find anything on this. Finally, I stumbled across it mostly by
accident, so here it is for anyone who may be having the same problem.</p>
<p>In short: use the “modulation” function, bringing it to a high value.</p>
<p>For a quick example, create a guitar track and bring up the “musical typing”
tool (Cmd+K). Press 8 to set the highest level of “Modulation” and then press
any note. You should hear the muffled guitar sound characteristic of palm
muting. If you now press 3 to bring the “Modulation” setting back to normal and
try again with a note, the string will ring unmuted.</p>
<figure style="max-width: 490px" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/palm-muted-low-E.png" alt="Musical typing tool, modulation set at highest value and pressing a low E note" />
</div>
<figcaption>
<p>Musical typing tool, modulation set at highest value and pressing a low E note</p>
</figcaption>
</figure>
<p>To use this in a track:</p>
<ol>
<li>Open the editor/piano roll (press E)</li>
<li>Expand the MIDI draw (icon with three lines on the top left corner of the
editor)</li>
<li>Under “Controller”, select “Modulation” on the dropdown (I think it’s the
default option anyway).</li>
<li>And now the part that requires some patience: adjust the modulation value
for the track so that it’s near the top when you want to apply palm muting, and
lower when you don’t.</li>
</ol>
<figure style="max-width: 566px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/modulation-controls-to-get-palm-muting.png" alt="Modulation adjusted at different values as the track progresses" />
</div>
<figcaption>
<p>Modulation adjusted at different values as the track progresses</p>
</figcaption>
</figure>
<p>What does this have to do with modulation as a sound concept? Beats me. I guess
GarageBand uses the function for other, misc uses where the original sense
doesn’t work. I dunno, I don’t really know anything about sound engineering.</p>
The “p” sound in Englishhttps://blog.pablobm.com/2016/08/30/the-p-sound-in-english/2016-08-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>For some time, I had noticed some people taking my name and spelling it as
“Bablo”. This was more common with, say, couriers than with cafe baristas,
probably because the former tend to be from countries where English is a first
language, whereas the latter tend to be from Southern European countries. Or at
least that’s my perception, in my specific bubble in London. I could be wrong. I
would love to see some figures on that.</p>
<p>For a time, I made an effort to emphasise that misheard “p”, but people kept
getting it wrong. I would increase my degree of emphasis, and ultimately I would
end up sounding really silly… and still getting my name misspelled by couriers.</p>
<p>It was only the other day, playing with the voice control of a 4th generation
Apple TV that it dawned on me, what I had been doing wrong all this time. I was
spelling out some login credentials, letter after letter, and Siri refused to
acknowledge a “p”, insisting that I was giving her (it?) a “b”, no matter how
much force I put on my labial stop. And the person next to me, an Irish
national, agreed with Siri. Finally I had an opportunity to unravel this
mystery.</p>
<p>Long story short: <strong>the Spanish “p” is different from the English “p”</strong>. It took
me 12 years in the UK to realise.</p>
<p>Wikipedia starts providing a clue, then confuses the reader, then a proper read
shows the difference. The page for <a href="https://en.wikipedia.org/wiki/Spanish_phonology">Spanish
phonology</a> describes the “p” as
“labial stop” as expected (by myself anyway), while the <a href="https://en.wikipedia.org/wiki/English_phonology">English
phonology</a> page has it as
“labial plosive/affricate fortis”. Promising, whatever that meant… Or it appears
to be until both links (click on the “p” phoneme on the table) lead to the same
page: <a href="https://en.wikipedia.org/wiki/Voiceless_bilabial_stop">Voiceless bilabial
stop</a>. W00t?</p>
<p>It is in the examples section of the page that the Spanish phoneme is described
as a “plain p” or /p/, whereas the Engish one appears as “aspirated p” or /ph/.
And that was exactly what I learned spelling out letters to Siri with the
assistance of an English-speaking human. I had to pronounce it in a way that, to
me, sounded more like a “pf”, with a very light “f”, for Siri (and the human) to
accept. Not only that, further down the password there was also a “t” that Siri
would get as “d”, and the problem was pretty much the same: I had to pronounce a
bit more like “tch”.</p>
<p>You see, I speak English well. <a href="https://www.youtube.com/watch?v=5035TY5RSpg">I learn it from a
book</a>.</p>
Up-to-date Firefox on Debian Livehttps://blog.pablobm.com/2016/07/28/up-to-date-firefox-on-debian-live/2016-07-28T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>The computers where I install Debian Live are used to access a relatively
limited set of websites. The default browser on the builds is Iceweasel,
Debian’s rebranding of Firefox which came to be circa 2004-2006 due to esoteric
legal reasons. The version number is 38, which is an
<a href="https://www.mozilla.org/en-US/firefox/organizations/faq/">ESR</a>: a long-term
support version scheduled by Mozilla to receive security updates for a extended
period of time.</p>
<p>Now the problem: one of the websites we use recently started complaining that
our browser was too old. Well, it actually is old. Firefox/Iceweasel 38 was
first released in May 2015, and a new ESR cycle (version 45) has started since
then. Websites maintainers tend to recognise this, and this website appears to
require the latest ESR. Everything appears to work correctly, but it does show a
warning and I am concerned it could actually start breaking at any moment.</p>
<p>So what’s a Debian Live admin to do? I found that Debian do publish up-to-date
browser packages (or as up-to-date as the latest ESR anyway) in their
repositories. However, it’s not in the obvious, default repositories but in the
ones for security updates, which need to be added to my <code>sources.list</code>. These
are normally active by default in new installs of Debian, but for some reason
they are not on Debian Live builds.</p>
<p>But also to my surprise, while looking for a solution I found that <a href="https://lwn.net/Articles/676799/">the whole
Iceweasel/Firefox drama has finally come to an
end</a>. Things have changed over the years, and
finally Mozilla has changed its licensing policies, getting Debian to finally
agree to bundle Firefox, actually branded as Firefox, in its distribution. These
new packages are already available, also on the repositories for security
updates.</p>
<p>This means that I can have Firefox instead of Iceweasel, and it can be version
45, which fits my requirements.</p>
<p>To have my Debian Live ISOs bundle this updated Firefox, I have to add the
source line at a file in <code>config/archives/</code>. This is a valid example:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># config/archives/security.list.chroot
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>deb http://security.debian.org/ jessie/updates main
</pre></div>
</div>
</div>
<p>If you were including Iceweasel in your builds, that should get you version 45.
Now, if you want Firefox you can reference it (as <code>firefox-esr</code>) instead in your
package lists. For instance:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># config/package-lists/firefox.list.chroot
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>firefox-esr
</pre></div>
</div>
</div>
<p>The rest is just running a build and installing the result in your computers.</p>
Upgrading to Ember 2.xhttps://blog.pablobm.com/2016/06/21/upgrading-to-ember-2x/2016-06-21T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>As part of my effort to learn Ember.js, I built a relatively simple app that
used the TFL API to fetch bus arrival times at nearby stops. It can also do
tube, DLR and overground, and eventually I may include other means of transport.
It’s called <a href="http://london-waits.herokuapp.com">London Waits</a>.</p>
<p>All this was built in Ember 1.13, and only now I decided to upgrade it to Ember
2. It wasn’t trivial, but in the process I gathered some notes that may help
others.</p>
<h2 id="ember-20-initial-upgrades">Ember 2.0: initial upgrades</h2>
<p>I started by upgrading Ember proper to 2.0.x. This can be done with Bower:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.0.x --save
</pre></div>
</div>
</div>
<p>There was immediately a problem. When I run the app, it complained that it
required a more recent version of jQuery:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>Error: Assertion Failed: Ember Views require jQuery between 1.7 and 2.1
</pre></div>
</div>
</div>
<p>I did as told:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install jquery#2.1.x --save
</pre></div>
</div>
</div>
<p>This got my tests passing again, so I proceeded to upgrade Ember Data. Now,
before version 2.3, Ember Data had both an npm module and a Bower component, and
both were required. I’m not entirely sure, but I think the npm module provided
CLI tools, whereas the Bower package provided the library proper. In fact, try
upgrading the npm package only first:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ npm install ember-data@2.0.x --save-dev
</pre></div>
</div>
</div>
<p>My tests pass after that, but the debug messages in the console reveal that I’m
still using an old version of Ember Data:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>DEBUG: -------------------------------
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>DEBUG: Ember : 2.0.3
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>DEBUG: Ember Data : 1.13.15
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>DEBUG: jQuery : 2.1.4
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>DEBUG: -------------------------------
</pre></div>
</div>
</div>
<p>This is fixed by installing the Bower package. Pay special attention to the fact
that the version comes after a ‘#’, not an ‘@’, and that the dependency is saved
with runtime packages <code>--save</code>, instead of development packages <code>--save-dev</code>.
Easy to miss!</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember-data#2.0.x --save
</pre></div>
</div>
</div>
<p>My tests were passing and I was on Ember 2. Yay!</p>
<h2 id="ember-21-problems-with-error-routes">Ember 2.1: problems with error routes</h2>
<p>Carrying on with the upgrade, I went for Ember 2.1. It should be just a matter
of upgrading the package, right?</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.1.x --save
</pre></div>
</div>
</div>
<p>However, one of my tests stopped working: one that ensured that the app rendered
an error route correctly. After some online research, I discovered that this
indeed stopped working in the run up to 2.1, and there seems to be no official
support. I was rather bummed by this, but fortunately I found a workaround in <a href="https://github.com/emberjs/ember.js/issues/12791#issuecomment-218561153">a
thread discussing this issue on
GitHub</a>.</p>
<p>I generalised the workaround into a helper and published it as <a href="https://gist.github.com/pablobm/5ec0fee8a877badaca2c11bf4751e857">a GitHub
Gist</a>,
although the example provided has some additional changes that we’ll get to
later in this writeup. Right now, for the code just as it was upgraded from
1.13, the example should look like this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// tests/acceptance/errors-test.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="reserved">import</span> Ember from <span class="string"><span class="delimiter">'</span><span class="content">ember</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="reserved">import</span> { module, test } from <span class="string"><span class="delimiter">'</span><span class="content">qunit</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="reserved">import</span> startApp from <span class="string"><span class="delimiter">'</span><span class="content">london-waits/tests/helpers/start-app</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="reserved">import</span> errorStateWorkaround from <span class="string"><span class="delimiter">'</span><span class="content">london-waits/tests/helpers/error-state-workaround</span><span class="delimiter">'</span></span>;
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>module(<span class="string"><span class="delimiter">"</span><span class="content">Acceptance | errors</span><span class="delimiter">"</span></span>, {
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> <span class="function">beforeEach</span>: <span class="keyword">function</span>() {
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="local-variable">this</span>.application = startApp();
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> errorStateWorkaround.setup(err => {
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="comment">// Return `true` if `err` is the error</span>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="comment">// we expect, and `false` otherwise</span>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> });
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> },
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> <span class="function">afterEach</span>: <span class="keyword">function</span>() {
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> errorStateWorkaround.teardown();
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> Ember.run(<span class="local-variable">this</span>.application, <span class="string"><span class="delimiter">'</span><span class="content">destroy</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> },
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span>});
<span class="line-numbers"><a href="#n21" name="n21">21</a></span>
<span class="line-numbers"><a href="#n22" name="n22">22</a></span>test(<span class="string"><span class="delimiter">"</span><span class="content">Something that lands an error</span><span class="delimiter">"</span></span>, <span class="keyword">function</span>(assert) {
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> <span class="comment">// Do something that would get the user to</span>
<span class="line-numbers"><a href="#n24" name="n24">24</a></span> <span class="comment">// an error route or substate</span>
<span class="line-numbers"><a href="#n25" name="n25">25</a></span>
<span class="line-numbers"><a href="#n26" name="n26">26</a></span> andThen(<span class="keyword">function</span>() {
<span class="line-numbers"><a href="#n27" name="n27">27</a></span> <span class="comment">// Assert that the error has ocurred as expected</span>
<span class="line-numbers"><a href="#n28" name="n28">28</a></span> });
<span class="line-numbers"><a href="#n29" name="n29">29</a></span>});
</pre></div>
</div>
</div>
<p>After that, I got my tests passing. Now it’s time to upgrade Ember Data:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ npm install ember-data@2.1.x --save-dev
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ bower install ember-data#2.1.x --save
</pre></div>
</div>
</div>
<p>And that’s all for 2.1.</p>
<h2 id="ember-22-liquid-fire-needs-upgrading">Ember 2.2: liquid-fire needs upgrading</h2>
<p>A simpler one now. My app was using the liquid-fire addon at version 0.21.2.
This was not compatible with Ember 2.2. The error I was getting was:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>TypeError: renderNode.state is undefined
</pre></div>
</div>
</div>
<p>Anyway, I upgraded liquid-fire along with Ember itself. These are the lines to
run, and the ones for Ember Data 2.2:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.2.x --save
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ npm install liquid-fire --save-dev
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>$ npm install ember-data@2.2.x --save-dev
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>$ bower install ember-data#2.2.x --save
</pre></div>
</div>
</div>
<h2 id="ember-23-ember-data-is-not-in-bower-any-more">Ember 2.3: Ember Data is not in Bower any more</h2>
<p>For Ember 2.3, just install Ember 2.3 (d’oh):</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.3.x --save
</pre></div>
</div>
</div>
<p>The <a href="http://emberjs.com/blog/2016/01/15/ember-2-3-released.html">release notes for Ember
2.3</a> advise that we
upgrade ember-qunit, which is a Bower package. I wasn’t getting any errors on my
tests, but did it nonetheless:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember-qunit --save
</pre></div>
</div>
</div>
<p>As for the <a href="http://emberjs.com/blog/2016/01/12/ember-data-2-3-released.html">release notes for Ember Data
2.3</a>, these
inform us that the Bower package is not required any more, as Ember Data is now
a full-fledged Ember addon. Also, ember-cli-shims (npm) needs to be upgraded to
0.1.0. Manually remove any references to ember-data from <code>bower.json</code>:</p>
<div class="language-diff code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="line comment">diff --git a/bower.json b/bower.json</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="line comment">index 214ef1e..8f74eb7 100644</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="line head"><span class="head">--- </span><span class="filename">a/bower.json</span></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="line head"><span class="head">+++ </span><span class="filename">b/bower.json</span></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="line change"><span class="change">@@</span> -4,7 +4,6 <span class="change">@@</span></span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2.3.x</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-cli-shims</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">ember-cli/ember-cli-shims#0.0.3</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-cli-test-loader</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">ember-cli-test-loader#0.1.3</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span><span class="line delete"><span class="delete">-</span> <span class="key"><span class="delimiter">"</span><span class="content">ember-data</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2.2.x</span><span class="delimiter">"</span></span>,</span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-load-initializers</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">ember-cli/ember-load-initializers#0.1.5</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-qunit</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^0.4.20</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-qunit-notifications</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0.0.7</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n13" name="n13">13</a></span><span class="line change"><span class="change">@@</span> -19,7 +18,6 <span class="change">@@</span></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> <span class="key"><span class="delimiter">"</span><span class="content">Faker</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">~2.1.3</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> },
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> <span class="key"><span class="delimiter">"</span><span class="content">resolutions</span><span class="delimiter">"</span></span>: {
<span class="line-numbers"><a href="#n17" name="n17">17</a></span><span class="line delete"><span class="delete">-</span> <span class="key"><span class="delimiter">"</span><span class="content">ember-data</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2.2.x</span><span class="delimiter">"</span></span>,</span>
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> <span class="key"><span class="delimiter">"</span><span class="content">ember</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2.3.x</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> <span class="key"><span class="delimiter">"</span><span class="content">jquery</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2.1.x</span><span class="delimiter">"</span></span>,
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> <span class="key"><span class="delimiter">"</span><span class="content">ember-qunit</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">^0.4.20</span><span class="delimiter">"</span></span>
</pre></div>
</div>
</div>
<p>And upgrade the required packages at the command line:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ npm install ember-data@2.3.x --save-dev
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ bower install ember-cli-shims#0.1.0 --save
</pre></div>
</div>
</div>
<h2 id="an-easy-one">2.4: an easy one</h2>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.4.x --save
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ npm install ember-data@2.4.x --save-dev
</pre></div>
</div>
</div>
<p>Piece of cake.</p>
<h2 id="changes-to-selectors-in-tests">2.5: changes to selectors in tests</h2>
<p>So I upgraded Ember as usual:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember#2.5.x --save
</pre></div>
</div>
</div>
<p>And tests started failing :-(</p>
<p>I’m not sure where exactly this comes from, but the semantics of jQuery
selectors seem to have changed subtly between Ember 2.4 and 2.5, at least in
acceptance tests. This affected all my acceptance tests because the <code>click()</code>
helper started to fail.</p>
<p>My HTML contains something like this:</p>
<div class="language-html code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><a</span> <span class="attribute-name">href</span>=<span class="string"><span class="delimiter">"</span><span class="content">...</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="tag"><p></span>Foo<span class="tag"></p></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="tag"><p></span>Bar<span class="tag"></p></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="tag"></a></span>
</pre></div>
</div>
</div>
<p>And my acceptance tests would do this:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>click(<span class="string"><span class="delimiter">'</span><span class="content">:contains("Bar")</span><span class="delimiter">'</span></span>);
</pre></div>
</div>
</div>
<p>Before 2.5, that would have sufficed to follow the link, but now the helper must
be triggering the <code>click</code> event differently, so it’s necessary to be more
specific. This works:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>click(<span class="string"><span class="delimiter">'</span><span class="content">a:contains("Bar")</span><span class="delimiter">'</span></span>);
</pre></div>
</div>
</div>
<p>And since Ember Data is a proper Ember addon, I thought I would upgrade it as
such from now on:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ ember install ember-data@2.5.x
</pre></div>
</div>
</div>
<h2 id="finally">2.6: finally!</h2>
<p>The last step was easy again:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>$ bower install ember --save
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>$ ember install ember-data
</pre></div>
</div>
</div>
<h2 id="epilogue-updating-ancillary-files">Epilogue: updating ancillary files</h2>
<p>After all the above, I got the app working on Ember(+Data) 2.6. Although this
should be enough, there are still some differences with a proper 2.6 app as
generated with <code>ember new</code>. These differences are in the files generated with
the application. I decided to update these too to avoid any potential problems
in the future.</p>
<p>What I did was generating a new app, then comparing the generated files with
those in mine. There are a few differences, for example in <code>app/app.js</code>:</p>
<div class="language-diff code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="line change"><span class="change">@@</span> -1,6 +1,6 <span class="change">@@</span></span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> import Ember from 'ember';
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span><span class="line delete"><span class="delete">-</span>import Resolver from '<span class="eyecatcher">.</span>/resolver';</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span><span class="line delete"><span class="delete">-</span>import loadInitializers from 'ember<span class="eyecatcher">-</span>load-initializers';</span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span><span class="line insert"><span class="insert">+</span>import Resolver from '<span class="eyecatcher">ember</span>/resolver';</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span><span class="line insert"><span class="insert">+</span>import loadInitializers from 'ember<span class="eyecatcher">/</span>load-initializers';</span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> import config from './config/environment';
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> var App;
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span><span class="change"><span class="change">@@</span> -25,7 +25,7 <span class="change">@@</span></span> Ember.onerror = function(error) {
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> App = Ember.Application.extend({
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> modulePrefix: config.modulePrefix,
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> podModulePrefix: config.podModulePrefix,
<span class="line-numbers"><a href="#n14" name="n14">14</a></span><span class="line delete"><span class="delete">-</span> Resolver</span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span><span class="line insert"><span class="insert">+</span> Resolver<span class="eyecatcher">: Resolver</span></span>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> });
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> loadInitializers(App, config.modulePrefix);
</pre></div>
</div>
</div>
<p>Also there are new files, such as <code>tests/helpers/module-for-acceptance.js</code>,
which is used in acceptance tests.</p>
<p>Differences too in <code>package.json</code> and <code>bower.json</code>. I updated the packages that
were behind, leaving alone those for which my version was ahead. All in all, I
tried to reduce differences with a 2.6 app as much as possible. This included:</p>
<ul>
<li>Updating old packages</li>
<li>Leaving alone packages that I had added myself</li>
<li>Updating packages added by myself that had compatibility issues or
deprecation notices</li>
<li>Removing some packages that are not necessary any more</li>
<li>Adding new packages</li>
<li>Removing the <code>resolutions</code> section from <code>bower.json</code></li>
</ul>
<p>This was a slow and annoying process, but it may spare me new problems in the
future. Also, I avoided including the addon <code>ember-welcome-page</code> because it’s
only useful in new installs, and expected to be removed by the developer after
work starts.</p>
<p>After all these changes, my tests failed again, again because of a subtlety,
this time in the test helper <code>moduleForAcceptance</code>. In two of my tests, I had a
utility function defined as part of the test module:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// tests/acceptance/my-test-test.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>module(<span class="string"><span class="delimiter">"</span><span class="content">Acceptance - my test</span><span class="delimiter">"</span></span>, {
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="comment">// ...</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> myUtilityFunction() {
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="comment">// do useful stuff...</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> },
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="comment">// ...</span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>});
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>test(<span class="string"><span class="delimiter">"</span><span class="content">Test case</span><span class="delimiter">"</span></span>, <span class="keyword">function</span>(assert) {
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="comment">// ...</span>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="local-variable">this</span>.myUtilityFunction();
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> <span class="comment">// ...</span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>});
</pre></div>
</div>
</div>
<p>Turns out this doesn’t work when using <code>moduleForAcceptance</code> instead of
<code>module</code>. I had to define the utility function outside and bind it to the module
at <code>beforeEach</code>:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment">/// tests/acceptance/my-test-test.js</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="keyword">function</span> <span class="function">myUtilityFunction</span>() {
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="comment">// do useful stuff...</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>}
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>moduleForAcceptance(<span class="string"><span class="delimiter">"</span><span class="content">Acceptance - my test</span><span class="delimiter">"</span></span>, {
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="comment">// ...</span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> beforeEach() {
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="local-variable">this</span>.myUtilityFunction = myUtilityFunction;
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> },
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="comment">// ...</span>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>});
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>test(<span class="string"><span class="delimiter">"</span><span class="content">Test case</span><span class="delimiter">"</span></span>, <span class="keyword">function</span>(assert) {
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> <span class="comment">// ...</span>
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> <span class="local-variable">this</span>.myUtilityFunction();
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> <span class="comment">// ...</span>
<span class="line-numbers"><a href="#n18" name="n18">18</a></span>});
</pre></div>
</div>
</div>
<p>This got me back on green tests.</p>
<p>After all this, I still have a deprecation notice coming from liquid-fire. There
doesn’t seem to be anything I can do about that, so I’ll just leave it there and
wait until a new release of the addon fixes it.</p>
Implicit `index` routes in Ember.jshttps://blog.pablobm.com/2016/05/29/implicit-index-routes-in-emberjs/2016-05-29T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>TLDR: Ember.js implicitly creates an <code>index</code> child route for you when you nest
under a route</p>
<p>This appears to be not so well known on the Ember community, and I don’t think
it’s documented anywhere. I should have a go at clarifying this on the official
docs (through a pull request on the project), but for the moment I’ll dump it
here.</p>
<p>This is how it works. Say you declare a route in <code>router.js</code>:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">lines</span><span class="delimiter">'</span></span>);
</pre></div>
</div>
</div>
<p>This declaration will get you a route, called <code>lines</code>. Nothing new here. Now
let’s declare the same route, but with nesting:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">lines</span><span class="delimiter">'</span></span>, <span class="keyword">function</span>(){});
</pre></div>
</div>
</div>
<p>This is subtly different. We added nesting but left it empty, so there’s no
reason to think that the result would be any different. However, there is indeed
a difference. This will get you not one but two routes: <code>lines</code> and
<code>lines.index</code>. Specifically, it’s the same as declaring the following:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">lines</span><span class="delimiter">'</span></span>, <span class="keyword">function</span>(){
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">index</span><span class="delimiter">'</span></span>, { <span class="key">path</span>: <span class="string"><span class="delimiter">'</span><span class="delimiter">'</span></span> });
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>});
</pre></div>
</div>
</div>
<p>This actually makes a lot of sense. If we think as <code>lines</code> as a “trunk” route
and <code>lines.index</code> as a “leaf”, it turns out that trunk routes cannot be
“landed”. These routes have an <code>outlet</code> that needs to be filled out. If we try
to land on a trunk route, for example using <code>transitionTo</code>, Ember will redirect
us to the <code>index</code> leaf route under them, and the outlet will get its content. In
other words, these two transitions are equivalent, assuming that <code>lines</code> is a
route with nesting:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.transitionTo(<span class="string"><span class="delimiter">'</span><span class="content">lines</span><span class="delimiter">'</span></span>);
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="local-variable">this</span>.transitionTo(<span class="string"><span class="delimiter">'</span><span class="content">lines.index</span><span class="delimiter">'</span></span>);
</pre></div>
</div>
</div>
<p>These implicit <code>index</code> routes are implemented with an empty route module and an
empty template. We don’t notice any of this, but we can verify it by using the
Ember Inspector, where we can see these routes, along with <code>loading</code> and <code>error</code>
subroutes:</p>
<figure style="max-width: 500px" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/ember-implicit-index-routes.png" alt="Implicit routes showing up on the Ember Inspector" />
</div>
<figcaption>
<p class="description">Implicit routes showing up on the Ember Inspector</p>
</figcaption>
</figure>
<p>This can go several levels deep. If we explicitly declare an <code>index</code> route with
nesting, Ember will declare yet another <code>index</code> under it:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">lines</span><span class="delimiter">'</span></span>, <span class="keyword">function</span>(){
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="local-variable">this</span>.route(<span class="string"><span class="delimiter">'</span><span class="content">index</span><span class="delimiter">'</span></span>, { <span class="key">path</span>: <span class="string"><span class="delimiter">'</span><span class="delimiter">'</span></span> }, <span class="keyword">function</span>() {
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> });
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>});
</pre></div>
</div>
</div>
<p>If we try to transition to <code>lines</code>, Ember will take us to <code>lines.index</code> and then
in turn to <code>lines.index.index</code>. This can go on for as long as necessary, until a
leaf route is found where we can land safely.</p>
Large files on a custom Debian installerhttps://blog.pablobm.com/2016/04/18/large-files-on-a-custom-debian-installer/2016-04-18T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<div class="update-notice">
<p>On <time>19 November 2017</time> I wrote about a better, simpler way of doing this. Check it out at <a href="http://blog.pablobm.com/post/167656438120/third-party-files-on-a-custom-debian-live">Third party files on a custom Debian Live installer</a>.</p>
</div>
<p>Following with my work on a custom Debian distribution, there was a third-party
package I wanted to include in it. Unfortunately, this was proprietary software,
it wasn’t included in Debian non-free, and it was a pretty large download (about
1GB).</p>
<p>Debian Live doesn’t seem to have a way to handle a case like this, or at least
one that doesn’t involve significant drawbacks.</p>
<h2 id="initial-approach">Initial approach</h2>
<p>A first approach would be to download the package and store it with the build
scripts, handy to be installed during the build process. The problem with this
is that I keep my build scripts under source control in
<a href="https://github.com/pablobm/w2c3-livecd">GitHub</a>. Adding a 1GB file to a git
repository is generally not a good idea, less so when that repo weighs under 1MB
otherwise (this is including all commits in its history). There’s also the
matter of whether the software license would allow uploading it to GitHub or a
similar service, which legally could constitute unauthorised distribution.</p>
<p>What would be better is to only keep a reference (a URL) to this package,
downloading it during the build process. Here the problem is that the package
will have to be downloaded every time we run a build; a large download that
significantly slows down the process (already slow) and makes it more cumbersome
to iterate quickly.</p>
<p>Debian Live keeps a <code>cache</code> directory where it keeps the Debian packages that it
downloads during a build. It should be possible to download our large package
there, and avoid re-downloading if it exists already… except that the <code>cache</code>
directory is not accessible from the chroot jail where custom scripts are run.
It’s from one of these scripts that I would run downloaded file (if it is an
installer) or otherwise put it in the correct location, so access from these is
necessary. Well, bummer.</p>
<p>I also tried keeping a separate cache directory next to where the chroot hook
scripts are kept (that would be <code>config/hooks/cache</code>), but those scripts don’t
appear to be run off that location, and again that file hierarchy doesn’t appear
to be accessible.</p>
<h2 id="my-solution">My solution</h2>
<p>At the end I went for something a bit more involved. I altered the build scripts
to add the following:</p>
<ul>
<li>At the start of the build, download the large file to <code>cache</code>, unless it
already exists there</li>
<li>From <code>cache</code>, run a simple http server that can serve the large file. For
example, you can use Python’s built-in <code>SimpleHTTPServer</code>)</li>
<li>Let the build run as normal</li>
<li>On a chroot hook, use <code>wget</code> or <code>curl</code> to download the large file from the
local http server into the chroot jail</li>
<li>Run any additional setup steps for the large file</li>
</ul>
<p>In other words, although the chroot jail doesn’t allow us to copy files from the
external filesystem, it doesn’t stop us from accessing files that are served
over HTTP from that external filesystem.</p>
<h2 id="the-scripts">The scripts</h2>
<p>This what my build script looked like after adding this feature:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>#!/usr/bin/env sh
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>set -e
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>BASE_DIR=$(dirname "$0")/..
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>SCRIPTS_DIR="$BASE_DIR/scripts"
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>CACHE_DIR="$BASE_DIR/cache"
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>CONFIG_DIR="$BASE_DIR/config"
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>DOWNLOADS_DIR="$CACHE_DIR/downloads"
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>LOCAL_FILES_DIR="$CONFIG_DIR/files"
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>PID_FILE_PATH=$(mktemp)
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>mkdir -p "$DOWNLOADS_DIR"
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>DOWNLOADER_PATH="$SCRIPTS_DIR/downloader"
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>SERVER_PATH="$SCRIPTS_DIR/server"
<span class="line-numbers"><a href="#n16" name="n16">16</a></span>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>DOWNLOADS_LIST_PATH="$CONFIG_DIR/downloads.list"
<span class="line-numbers"><a href="#n18" name="n18">18</a></span>
<span class="line-numbers"><a href="#n19" name="n19">19</a></span>"$DOWNLOADER_PATH" -d "$DOWNLOADS_DIR" -l "$LOCAL_FILES_DIR" "$DOWNLOADS_LIST_PATH"
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span>"$SERVER_PATH" -P "$PID_FILE_PATH" "$DOWNLOADS_DIR"
<span class="line-numbers"><a href="#n21" name="n21">21</a></span>
<span class="line-numbers"><a href="#n22" name="n22">22</a></span>lb build noauto "${@}" 2>&1 | tee build.log
<span class="line-numbers"><a href="#n23" name="n23">23</a></span>
<span class="line-numbers"><a href="#n24" name="n24">24</a></span>PID=$(cat "$PID_FILE_PATH")
<span class="line-numbers"><a href="#n25" name="n25">25</a></span>kill "$PID"
</pre></div>
</div>
</div>
<p>For the final implementation, I divided the files I wanted to copy into two
categories: large ones I needed to download from the Internet, and smaller ones
that I was ok with adding to the repository. On this second group there can be
file checksums, customised config files, and probably other things.</p>
<p>There is a “downloader” script, referenced as <code>$DOWNLOADER_PATH</code> that reads a
list of files (<code>$DOWNLOADS_LIST_PATH</code>). For each entry, if it appears to look
like a URL, a file is downloaded from the Internet. Other entries are expected
to be files residing at <code>$LOCAL_FILES_DIR</code>. All files are copied to
<code>$DOWNLOADS_DIR</code>, renamed as per indicated in each entry of the list.</p>
<p>A server script (<code>$SERVER_PATH</code>) is then run, and we take note of its pid. This
way, we can cleanly shut it down after the build is done.</p>
<p>This is the downloader script:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>#!/usr/bin/env sh
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>while getopts ':d:l:' opt; do
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> case $opt in
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> d)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> DOWNLOADS_DIR="$OPTARG"
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> ;;
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> l)
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> LOCAL_FILES_DIR="$OPTARG"
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> ;;
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> esac
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>done
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>shift $((OPTIND-1))
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>DOWNLOADS_LIST_PATH="$*"
<span class="line-numbers"><a href="#n16" name="n16">16</a></span>
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>cat "$DOWNLOADS_LIST_PATH" | while read -r NAME URL; do
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> DEST_PATH="$DOWNLOADS_DIR/$NAME"
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> if echo $URL | grep -qE '^https?://' ; then
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> wget -c --output-document "$DEST_PATH" "$URL"
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> else
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> cp "$LOCAL_FILES_DIR/$URL" "$DEST_PATH"
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> fi
<span class="line-numbers"><a href="#n24" name="n24">24</a></span>done
</pre></div>
</div>
</div>
<p>As mentioned before, it’s fed a list of files (<code>$DOWNLOADS_LIST_PATH</code>) that may
refer to URLs or local files (under <code>$LOCAL_FILES_DIR</code>). Whatever is downloaded,
it will be copied to <code>$DOWNLOADS_DIR</code>. Using wget’s <code>-c</code> option, we ensure that
failed downloads are continued, and existing files are not re-downloaded.</p>
<p>This is an example of a file list:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>big-file https://example.com/big-file
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>small-file small_file_kept_locally
</pre></div>
</div>
</div>
<p>For each entry, the first word before the space is the name that the
downloaded/copied file will receive at the cache directory. The second word is
either the URL to retrieve it from, or the name of the file in the local
filesystem.</p>
<p>This script runs the http server:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>#!/usr/bin/env sh
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>while getopts ':P:' opt; do
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> case $opt in
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> P)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> PID_FILE_PATH="$OPTARG"
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> ;;
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> esac
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>done
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>shift $((OPTIND-1))
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>WORKING_DIR="$*"
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>cd "$WORKING_DIR"
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>python -mSimpleHTTPServer 12345 &
<span class="line-numbers"><a href="#n16" name="n16">16</a></span>bg
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>echo $! > "$PID_FILE_PATH"
</pre></div>
</div>
</div>
<p>Nothing much to see on it, apart from the option used to specify a pid file.
This will be used to shut down the server cleanly at the end.</p>
<p>Finally there’s the chroot hook that will retrieve the files “over the fence” of
the chroot jail and actually install them. It will be something like this:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>#!/bin/sh
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>set -e
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>WORKING_DIR=$(mktemp -d /tmp/tmp.XXXXXXXXXXXXX)
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>cd "$WORKING_DIR"
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>wget http://localhost:12345/big-file
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>wget http://localhost:12345/small-file
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span># Do something with the downloaded big-file and small-file
<span class="line-numbers"><a href="#n12" name="n12">12</a></span># ...
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span># Finally we clean after ourselves
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>rm -rf "$WORKING_DIR"
</pre></div>
</div>
</div>
Hash syntax in Ruby 2.2https://blog.pablobm.com/2016/03/17/hash-syntax-in-ruby-22/2016-03-17T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Twice in the last three weeks, I have stumped into a relatively recent addition
to Ruby’s syntax: a new way to define symbol keys in a Hash literal.</p>
<p>Since Ruby 2.2, the following syntax is legal:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{ "foo": "bar" }
</pre></div>
</div>
</div>
<p>The weird thing here is that the above literal doesn’t mean what I thought it
meant. This is what I expected it to be equivalent to:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{ "foo" => "bar" }
</pre></div>
</div>
</div>
<p>And this is what it actually is equivalent to:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{ :foo => "bar" }
</pre></div>
</div>
</div>
<p>I have to admit that both meanings are perfectly reasonable in their own way. To
me, it was annoying because I copied over some JSON into my code and forgot to
change the colons <code>:</code> to hash rockets <code>=></code>. My tests started failing and it took
me a bit to realise what was going on. An older Ruby would have given me a
syntax error and I would have realised straightaway.</p>
<p>Another reason I find it surprising is because, since the introduction of the
colon-based syntax for hashes in Ruby 1.9 (eg: <code>{foo: "bar"}</code>), I have had some
opportunities to see mixes of colons and hash bangs such as this one:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> foo: "bar",
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> 1 => "baz",
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> "uno" => "dos",
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>}
</pre></div>
</div>
</div>
<p>I was hoping that, eventually, Ruby would get a syntax compatible with JSON’s,
so that I could just copy the latter into the former without the problem
described above, just as you can in JavaScript. The syntaxes of both were not
compatible but didn’t have any conflicts, and Ruby just needed extending the
colon syntax to values other than symbols. However, the change in 2.2 eliminates
this possibility, as it introduces an element that means different things in
each format.</p>
<p>I have no idea of what the exact line of thought was in Ruby’s core team when
they decided to introduce this (I haven’t been able to find a relevant link).
However, I did find a <a href="http://stackoverflow.com/questions/27759791/new-way-of-creating-hashes-in-ruby-2-2-0">defence of it on
StackOverflow</a>.</p>
<p>The argument goes that symbols need to be defined with quotes when they have
certain characters—eg: <code>:"strange-symbol"</code>. Therefore it makes sense that these
two keys both are interpreted as symbols:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>{
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> normal_symbol: "foo",
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> "weird-symbol": "bar",
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>}
</pre></div>
</div>
</div>
<p>Which ok, I admit also makes sense, but it sure is weird to me.</p>
Memory upgrade for an MSI Wind U135https://blog.pablobm.com/2016/02/21/memory-upgrade-for-an-msi-wind-u135/2016-02-21T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>I own an inexpensive netbook that I take with me when I travel. Specifically, an
MSI Wind U135. It’s been serving me well for almost 5 years, and the other day I
decided to upgrade it a bit, by adding some memory.</p>
<p>After some online research, it was clear I could only upgrade from the builtin
1GB to a total of 2GB, adding a 1GB memory module. What was not so clear though
is what specific model of memory I needed. This netbook uses DDR3 SDRAM, but
it’s not as easy as just buying that:</p>
<ul>
<li>Within the DDR3 SDRAM specification there are 6 different sub-specs
operating at different frequencies.</li>
<li>For each frequency, there are a further 2-4 modes of operation.</li>
<li>In total, there appears to be 21 different types of DDR3 SDRAM.</li>
<li>Online resources warn that the computer will not work with the incorrect
type of memory.</li>
<li>There’s no indication anywhere of what specific type of memory my netbook
uses.</li>
</ul>
<p>On the BIOS setup menu, it lists the “System Bus Speed” as 667MHz. The tool
<code>dmidecode</code> confirms this, but doesn’t add anything to it<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>$ sudo dmidecode --type memory
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span># dmidecode 2.12
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>SMBIOS 2.6 present.
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>Handle 0x0029, DMI type 16, 15 bytes
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>Physical Memory Array
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> Location: System Board Or Motherboard
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> Use: System Memory
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> Error Correction Type: None
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> Maximum Capacity: 4 GB
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> Error Information Handle: Not Provided
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> Number Of Devices: 2
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>Handle 0x002B, DMI type 17, 28 bytes
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>Memory Device
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> Array Handle: 0x0029
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> Error Information Handle: Not Provided
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> Total Width: 64 bits
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> Data Width: 64 bits
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> Size: 1024 MB
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> Form Factor: DIMM
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> Set: None
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> Locator: A1_DIMM0
<span class="line-numbers"><a href="#n24" name="n24">24</a></span> Bank Locator: A1_BANK0
<span class="line-numbers"><a href="#n25" name="n25">25</a></span> Type: <OUT OF SPEC>
<span class="line-numbers"><a href="#n26" name="n26">26</a></span> Type Detail: Synchronous
<span class="line-numbers"><a href="#n27" name="n27">27</a></span> Speed: 667 MHz
<span class="line-numbers"><a href="#n28" name="n28">28</a></span> Manufacturer: A1_Manufacturer0
<span class="line-numbers"><a href="#n29" name="n29">29</a></span> Serial Number: A1_SerNum0
<span class="line-numbers"><strong><a href="#n30" name="n30">30</a></strong></span> Asset Tag: A1_AssetTagNum0
<span class="line-numbers"><a href="#n31" name="n31">31</a></span> Part Number: Array1_PartNumber0
<span class="line-numbers"><a href="#n32" name="n32">32</a></span> Rank: Unknown
</pre></div>
</div>
</div>
<p>This would narrow down the search to one of the following <a href="https://en.wikipedia.org/wiki/DDR3_SDRAM">models listed on
Wikipedia</a> (only relevant extract of
the table shown):</p>
<table class="data-table">
<caption>JEDEC standard modules (extract)</caption>
<thead>
<tr>
<th>Standard name</th>
<th>I/O bus clock</th>
<th>Module name</th>
<th>Timings</th>
</tr>
</thead>
<tr>
<td>
DDR3-1333F<br />
DDR3-1333G<br />
DDR3-1333H<br />
DDR3-1333J
</td>
<td>666.67 MHz</td>
<td>PC3-10600</td>
<td>
7-7-7<br />
8-8-8<br />
9-9-9<br />
10-10-10
</td>
</tr>
</table>
<p>At this point, I had no idea of whether any of the listed 4 types was ok or
I should find out which exact one I needed. I know next to nothing about
hardware, and I wasn’t going to risk it. I needed more information.</p>
<p>I opened up the netbook and looked inside. The builtin memory module is easy to
find: four black rectangles lined up in front of the empty upgrade slot:</p>
<figure style="max-width: 400px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/memory-module.jpg" alt="The memory module, built into the motherboard" />
</div>
<figcaption>
<p class="description">The memory module, built into the motherboard</p>
</figcaption>
</figure>
<p>Each one of the black rectangles has an inscription with details about the
manufacturer, as well as the part model and number. It’s tiny and difficult to
read. A picture can help to make out the text. In my case, they look like this:</p>
<figure style="max-width: 400px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/memory-chip.jpg" alt="A single memory chip, inscribed with model parts and numbers" />
</div>
<figcaption>
<p class="description">A single memory chip, inscribed with model parts and numbers</p>
</figcaption>
</figure>
<p>That’s a mouthful, but it makes for a starting point. I decided to look up
“H5TQ1G83BFR” first. I got a few links as a result, but the one that was most
interesting was a <a href="https://www.skhynix.com/product/filedata/fileDownload.do?seq=2274">PDF with technical
details</a> of
the products offered by the manufacturer. The first page lists three product
lines, one of which is “H5TQ1G83BFR-xxC”, where the “xxC” part would match up
the inscription “H9C” on my chips. Promising!</p>
<p>As mentioned, I know nothing about this topic, so the document could be in
Klingon for all I can understand. Thankfully, I just needed part numbers. Since
the document seems to cover the “xxC” lines, and I own an “H9C”, I looked up
“H9”, and found exactly what I wanted (again, an extract for the relevant
table):</p>
<table class="data-table">
<caption>Operating Frequency (extract)</caption>
<thead>
<tr>
<th>Speed Grade (Marking)</th>
<th>Remark (CL-tRCD-tRP)</th>
</tr>
</thead>
<tr>
<td>-G7</td>
<td>DDR3-1066 7-7-7</td>
</tr>
<tr>
<td>-H9</td>
<td>DDR3-1333 9-9-9</td>
</tr>
</table>
<p>According to this, my netbook uses DDR3-1333 9-9-9 which in turn, according to
Wikipedia, means the exact spec is DDR3-1333H. With that information (and
remembering I was limited to buying 1GB of memory), I could finally go online
shopping with enough confidence that I was getting the right product.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Info on using <code>dmidecode</code> at <a href="https://www.howtoforge.com/dmidecode-finding-out-hardware-details-without-opening-the-computer-case">HowtoForge</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Ruby: difference between `__method__` and `__callee__`https://blog.pablobm.com/2016/01/20/ruby-difference-between-method-and-callee/2016-01-20T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Just yesterday I learned a little something in Ruby that I found interesting.
It’s the difference between <code>__method__</code> and <code>__callee__</code>.</p>
<p>Both are supposed to return the name of the current method, but there’s a subtle
difference:</p>
<ul>
<li><code>__method__</code>: returns the name of the method <strong>as defined</strong></li>
<li><code>__callee__</code>: returns the name of the method <strong>as called</strong></li>
</ul>
<p>I put together an example that illustrates the difference:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="keyword">class</span> <span class="class">Bar</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> <span class="keyword">def</span> <span class="function">foo</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> puts <span class="string"><span class="delimiter">"</span><span class="content">__callee__ is </span><span class="inline"><span class="inline-delimiter">#{</span>__callee__<span class="inline-delimiter">}</span></span><span class="delimiter">"</span></span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> puts <span class="string"><span class="delimiter">"</span><span class="content">__method__ is </span><span class="inline"><span class="inline-delimiter">#{</span>__method__<span class="inline-delimiter">}</span></span><span class="delimiter">"</span></span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="keyword">end</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> <span class="keyword">alias</span> <span class="symbol">:baz</span> <span class="symbol">:foo</span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span><span class="keyword">end</span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>bar = <span class="constant">Bar</span>.new
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>bar.foo <span class="comment"># => Prints "foo" and "foo"</span>
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>bar.baz <span class="comment"># => Prints "baz" and "foo"</span>
</pre></div>
</div>
</div>
<p>This illustrates very well the difference between “message” and “method”. One is
the name we use to invoke a method. The other is the name of the method
ultimately called.</p>
36https://blog.pablobm.com/2016/01/07/36/2016-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>62</li>
<li>1 + 2 + 3 + 4 + 5 + 6 + 7 + 8</li>
<li>12 * 22 * 32</li>
<li>13 + 23 + 33</li>
<li>1 + 2 + 3 + … + 36 = 666</li>
<li>Moi</li>
</ul>
Proprietary drivers on a custom Debian installerhttps://blog.pablobm.com/2015/07/31/proprietary-drivers-on-a-custom-debian-installer/2015-07-31T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Recently I had the need to create a custom GNU/Linux distribution. It would
install the system in a computer, along with a few handpicked packages, all with
minimum fuss. Luckily enough, there’s software for this: <a href="https://live-systems.org">Debian
Live</a>.</p>
<figure style="max-width: 120px;" class="post-illustration post-illustration--side">
<div class="post-illustration__images">
<img src="/images/debian-logo.png" alt="Debian" />
</div>
<figcaption>
<p class="attribution">Image by <a href="https://www.debian.org">Debian</a></p>
<p class="license"><a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/">CC-SA License</a></p>
</figcaption>
</figure>
<p>The project’s website is not initially the most inviting, but it doesn’t take
much digging to find the <a href="http://live-systems.org/manual/stable/index.en.html">documentation of the
project</a>. Just following
the section titled “For the impatient” gets you up and running in no time,
creating a customised installer of Debian GNU/Linux. The rest of the
documentation covers a number of advanced cases.</p>
<h2 id="proprietary-drivers-during-installation">Proprietary drivers during installation</h2>
<p>However, I did come across a stumbling block. Some of the laptops where I have
to install this require a proprietary driver for their WiFi card. The problem
here is not getting these drivers installed: instead, the problem is getting the
card to work during the installation process itself, so that updated packages
can be downloaded if desired.</p>
<h2 id="initial-attempt">Initial attempt</h2>
<p>Reading the documentation, it’s straightforward to see how to get the drivers
working after the installation. Simply add the name of your package to one of
the “chroot” package lists:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># config/package-lists/firmware.list.chroot
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>firmware-b43-installer
</pre></div>
</div>
</div>
<p>And make sure you are fetching packages from the contrib/non-free repositories
if required:</p>
<div class="language-sh code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># auto/config
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>set -e
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>lb config noauto \
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> --archive-areas 'main contrib non-free' \
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> "${@}"
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
</pre></div>
</div>
</div>
<p>So that’s simple. Now, how to make this work for the installation process too?
On further reading, I am led to think I only need a “binary” package list
referencing the same package but, much as I try, that leads nowhere. The
required firmware doesn’t end up in the expected place</p>
<h2 id="the-solution">The solution</h2>
<p>Finally I figured out a way. During the build process, and after the firmware is
unpackaged on the “chroot” tree, I can have a hook (see <a href="http://live-systems.org/manual/stable/html/live-manual.en.html#490">section 9.2 of the
manual</a>)
copy the files across to the “binary” tree. For this I just needed this file (as
well as the two listings above):</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># config/hooks/firmware.binary
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>#!/bin/sh
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>cp -pr chroot/lib/firmware/b43 binary/firmware/
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>
</pre></div>
</div>
</div>
<p>And that is this problem solved! I’ll note that the installation process still
asks whether you want to supply firmware from the wayward device. In my case, it
asks three times, each time mentioning different required files. Just go “Yes”
until it’s satisfied. Of course, if it always asks about the same files, it’s
probably because something went wrong and it can’t actually find them.</p>
<p>A way to avoid this question asked at all is using a preseeded configuration:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span># config/includes.binary/install/preseed.cfg
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>...
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>d-i hw-detect/load_firmware boolean true
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>...
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>
</pre></div>
</div>
</div>
<p>That should make it clear that yes, we want that firmware loaded and there’s no
need to ask so many questions.</p>
<h2 id="a-working-example">A working example</h2>
<p>A nice feature of Debian Live is that it’s all configured using just a few text
files. Very convenient to track your progress with version control. I published
the config files for my distro on Github if you want to check it out. At the
time of writing these lines, it is a working example of the technique described
above:
<a href="https://github.com/pablobm/w2c3-livecd">https://github.com/pablobm/w2c3-livecd</a></p>
35https://blog.pablobm.com/2015/01/07/35/2015-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>mm film</li>
<li>Bromine</li>
<li>Hmm… 5 times 7?</li>
<li>Pretty uninteresting number, to be honest</li>
<li>But now, it’s my uninteresting number</li>
<li>So I’ll have to make it interesting myself</li>
</ul>
Fishy-looking ATMhttps://blog.pablobm.com/2014/11/16/fishy-looking-atm/2014-11-16T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Last month I spotted the following ATM card slot, outside a branch of HSBC:</p>
<figure style="max-width: 400px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/atm-slot-1.jpg" />
</div>
<figcaption>
<p class="description">The ATM slot as I found it</p>
</figcaption>
</figure>
<p>I don’t know about you, but to me that looks fishy as hell:</p>
<ol>
<li>The card reader is a separate, protruding, differently coloured piece of
plastic</li>
<li>The surrounding surface shows clear lacerations from tool work</li>
</ol>
<p>Additionally, it looked very much like the one described in a blog post that I
had seen recently, by Paul Battley. Please read this account of the story, it’s
not long: <a href="http://po-ru.com/diary/skimmer/">http://po-ru.com/diary/skimmer/</a></p>
<p>I went into the office and reported my suspicions, however these didn’t get the
treatment I expected. The person who I talked to simply smiled, nodded, and
showed no sign of caring in the slightest. The most I got was an (I felt)
unconvincing assurance that the matter would be looked into at their earliest
convenience. The day after nothing had changed. I then phoned HSBC, but got the
very same treatment.</p>
<p>Two weeks after this incident, the card slot remained the same, so I could only
assume there was no skimmer device. Then another week later, the slot finally
changed, but didn’t improve the situation:</p>
<figure style="max-width: 400px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/atm-slot-2.jpg" />
</div>
<figcaption>
<p class="post-illustration--caption">Same ATM slot, some weeks later</p>
</figcaption>
</figure>
<p>Interestingly, Paul Battley’s situation had been handled similarly. In his case
it was with RBS as the bank, and without the final slot change:
<a href="http://po-ru.com/diary/skimmer-deuxieme-partie/">http://po-ru.com/diary/skimmer-deuxieme-partie/</a></p>
<p>Paul wonders whether we are screwed because ATM technicians can’t spot skimmer
devices, or doomed because safe ATMs can’t be told from tampered ones. After my
experience, I have to lean towards the second option.</p>
<p>This also makes me wonder to what extent banks are aware of this problem. Is
this a case of the machine looking so fishy that people walk into the branch
everyday to report suspicion? If that’s it, I can understand why staff don’t
seem to care.</p>
<p>I also have reservations about how this problem is treated at a higher level:
when I called HSBC, the person on the other side showed no interest either. I
was actually asked for the address where this had happened, but it didn’t sound
at all like they were typing it down anywhere. I know because my strong accent
forces people on the phone to ask again for names and directions, and I always
end up spelling them out. None of this occurred in this call, despite my
ineptitude at pronouncing the word
“<a href="https://en.wikipedia.org/wiki/Clerkenwell">Clerkenwell</a>“.</p>
<p>I can see the bank’s side of the argument though. The one obvious way to get
this right would be for the ATM’s front to be made of a single piece, with
minimal protuberances or holes. I understand this can get expensive if they have
to replace all machines across the country. I can’t tell how expensive though,
and how this affects the business’s bottom line. Of course as a plain citizen
who mistrusts banks and financial institutions by default, I’d tend to think
they should bloody well get their asses in gear and solve this problem as this
should be just peanuts from their executives’ bonuses, made up from money they
swindle from us in a daily basis. But I don’t have the hard numbers, so I won’t
go into that argument just yet.</p>
Closing a FirefoxOS apphttps://blog.pablobm.com/2014/03/30/closing-firefoxos-app/2014-03-30T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>The documentation for
<a href="https://developer.mozilla.org/en-US/Firefox_OS">FirefoxOS</a> doesn’t make it
obvious how to completely close an application. Makes some sense, since their
view seems to be that apps are simply web applications, and should be always on.
However, I don’t think this always applies.</p>
<p>Fortunately, there’s a way to do what I want:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>window.close()
</pre></div>
</div>
</div>
<p>My use case is the following. I am building an app for FirefoxOS that makes
extensive use of fine-grained geolocation. GPS uses up a lot of battery, and
it’s not trivial for the app to tell when to use it and when not. Therefore, I
need a way for users to close the app, safe in their knowledge that it won’t be
draining resources in the background.</p>
<p>I have built <a href="https://github.com/pablobm/fxos-experiments/tree/master/quit">a little proof of
concept</a> that I
have published on Github. It works on my FxOS 1.1 phone (ZTE Open).</p>
34https://blog.pablobm.com/2014/01/07/34/2014-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>F9</li>
<li>Spain</li>
<li>Selenium</li>
<li>“If it exists…”</li>
<li>Me</li>
</ul>
Dual flush: interface design gone down the drainhttps://blog.pablobm.com/2013/06/11/dual-flush-interface-design/2013-06-11T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>For some time now, I have been thinking about <a href="http://en.wikipedia.org/wiki/Dual_flush_toilet">dual flush
toilets</a>. I find them annoying,
or rather: I find their interface annoying.</p>
<p>Their intended purpose is good: not all visits to the toilet require the same
amount of water to flush, so why empty a whole cistern every time? Dual flush
allows us to save water, which is good for both the environment and our pockets.</p>
<p>However, every time I see one, I am at a loss as to how to correctly operate it.
Is is the large portion of the button that I have to press or is it the small
one?</p>
<figure style="max-width: 200px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/dual-flush-crescent-2.jpg" />
</div>
</figure>
<p>Regional disclaimer: In Europe, every dual-flush I have seen has a button-based
interface. I hear there are lever-based ones in other parts of the world, but I
cannot comment on those.</p>
<p>Initially, it would seem reasonable that the large portion is intended to
release more water. That sounds simple enough, right? Well, no, because the
small portion is often so small or narrow that it is difficult to press it
without accidentally pressing the other one in the process.</p>
<p>Therefore, thinking that the opposite is true would also make sense. After all,
if you are going to press one of the buttons accidentally, it better be the one
that uses less water. You’d then make this be the larger button. Then the
smaller button would be pressed by users who are certain they require to spend
the whole cistern.</p>
<p>I have been so bothered with this lately that I have started to study them.
Unfortunately, this forces me to spend more time in the toilet when I am in a
new place, but it’s all for science! :-)</p>
<p>Whenever I see a dual flush, I bring out the chronometer on my mobile phone. I
press one of the flush buttons, time how long it takes to operate, wait for the
cistern to fill back up, and try again with the other button. Normally several
times, just to make sure.</p>
<p>Funny thing is: I have barely done any progress because most of the time both
options seem to take as long to operate! So not only the buttons are badly
designed, but also the feedback (the appearance of the resulting stream of
water) is all wrong and hardly gives any information as to what actually is
going on.</p>
<p>Not all designs I have seen are as bad, but in general the landscape in this
field is pretty depressing. The best designs I have seen are not really good,
but just less bad. For example, recently I saw this one at a restaurant:</p>
<figure style="max-width: 280px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/dual-flush-I-II.jpg" />
</div>
</figure>
<p>In this instance, pressing the lower portion resulted in a short flush, whereas
pressing the upper portion resulted in a long flush.</p>
<p>I am not sure if you can appreciate it well in the picture, but the lower
portion is bulging out, whereas the top one has a hollow shape, as if sinking
away from the user. This makes the lower portion easier to press. Additionally,
there is one clear mark on the lower one and two on the upper one, but easy to
see and touch evident. Also, the upper portion is slightly larger, as the lower
portion has the typical crescent shape, but less pronounced than in other
examples. This should follow the idea that the larger surface creates a longer
flush.</p>
<p>However I think this is not good enough. The size of each portion is still
small, making it accident-prone. Less so that other common designs though, but
when you press a button and it sinks in, it’s not trivial to avoid pressing the
other one.</p>
<p>Also, I wonder how evident the meaning of the markings is. In my mind it makes
sense that two markings mean “more” than one, and therefore should release more
water, but what do other people think? After seeing how bad these interfaces
normally are, I wonder if their creators ever bothered doing any user testing at
all.</p>
<p>But anyway, I’ll continue doing my research. To some it may seem silly toilet
humour but, for something that plays such an important role in our lives, I
think it’s important to get it right.</p>
My first impressions of Netflixhttps://blog.pablobm.com/2013/05/02/my-first-impressions-of-netflix/2013-05-02T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<figure style="max-width: 200px;" class="post-illustration post-illustration--side">
<div class="post-illustration__images">
<a href="http://netflix.com">
<img src="/images/netflix-logo.png" />
</a>
</div>
</figure>
<p>Yesterday I signed up for Netflix. Before now, I had used it at other people’s
homes, but I had not signed up myself because I barely watch series/films at
home. But I finally caved in last night, and now I am officially impressed.</p>
<p>Always using my Android tablet, I created an account, then downloaded their app
where I browsed their selection and watched my first show. The process went
flawlessly, and I have commend them for the experience that they have managed to
deliver, at least to a first timer like me. After this start, there are only
three minor things, and a relatively major one, that I want to comment on.</p>
<p>First, I found funny that they ask users straight away what kind of content they
want to watch: drama, comedy, horror, etc. I skipped that part because I just
don’t know! I want to watch good shows regardless of their genre. I understand
they want me to start feeding their suggestions engine, but I don’t want to
restrict my choices just yet.</p>
<p>The second thing was that, at the beginning, I feared that a Facebook account
was required to register, as it used to be the case with Spotify. Fortunately,
after reading a bit more I saw this was not the case. I am definitely <strong>not</strong>
going to link this up with my Facebook account. I often wonder why I have a
Facebook account in the first place, as I rarely use it, but that’s another
story for another day.</p>
<p>Third: I was also asked if I was going to use Netflix on (if I remember
correctly) my PC, my Mac or my games console. Funnily enough this list doesn’t
include tablets, even though I was signing up from one.</p>
<p>But anyway, I don’t think the above are too important. I’m going to move on to
the “relatively major” thing, that I must say almost put me off from Netflix
altogether: the permissions required by the Android app. Why on Earth does the
Netflix app require permission to:</p>
<ul>
<li>Read phone status and identity</li>
<li>Read sensitive log data</li>
</ul>
<p>I am very concerned about my privacy, and I rarely let these things slip in. I
let this one in because I really wanted to try the service.</p>
<p>The phone identity I assume has to do with limiting the number of devices that
access the same account, so I don’t give my login details to my friends to use
Netflix for free. I do think this is complete bollocks, but I guess this has to
do with their contracts with the studios.</p>
<p>The “sensitive log data” permission… I have no clue why that is needed. I will
come back to this and see what the repercussions are and what I can do about it,
but I am honestly uneasy about this. I hear there ways to get better control of
apps’ permissions, so I’ll have to take a look at that at a later moment.</p>
<p>And one bonus comment now. While writing this, I logged onto the Netflix website
with my laptop, and clicked on a show. I was mortified to see that the player
requires Microsoft Silverlight. I have been avoiding to install it since it
first came up, and I’d like to stay clear of it for the foreseeable future.</p>
<p>I understand their contracts with the studios require them to use DRM, and this
cannot be achieved with HTML5. I also realise that it is a bit funny that I
would not have complained if it had required Flash instead. I like Flash (not)
as much as I like Silverlight (not), but the first one crept up into our
computers long time ago, so I am not so fussed about it now (and I block it on
my browsers anyway).</p>
<p>But enough of negativity for now. The Silverlight problem I’ll have to accept,
and the Android permissions problem… I’ll see what I can do. Apart from these
two points (and the other minor ones above), I feel I must congratulate the
Netflix team for delivering such an awesome experience. This is the future. I am
glad that finally somebody was able to pull off something like this. This is the
best way to combat piracy: to offer a better alternative.</p>
33https://blog.pablobm.com/2013/01/07/33/2013-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>1, 3, 11, 33</li>
<li>Arsenic</li>
<li>Vertebrae</li>
<li>Crucifixion</li>
<li>Me</li>
</ul>
I don't own a smartphonehttps://blog.pablobm.com/2012/11/13/i-dont-own-a-smartphone/2012-11-13T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com<div class="update-notice">
<p><strong>Update:</strong> I finally gave in and got a smartphone in <time>October 2019</time>.</p>
</div>
<figure style="max-width: 150px;" class="post-illustration post-illustration--side">
<div class="post-illustration__images">
<a href="http://en.wikipedia.org/wiki/Nokia_1100">
<img src="/images/nokia-1100.jpg" />
</a>
</div>
</figure>
<p>I don’t own a smartphone. Not even a sort-of-smart phone. Instead I use a
dumbphone, a Nokia 1100 to be precise, that I have been using for over a year
now.</p>
<p>It’s 2012, I am a tech-savvy person who builds web applications for a living:
why would I want to use a dumbphone? I have the following reasons not to want a
smartphone:</p>
<ul>
<li>They have very poor battery life. Batteries need to be charged every day,
and people can’t go on a weekend holiday without also having to take their
chargers with them (and expect electricity to be available).</li>
<li>I know I’m going to lose it. I am the kind of person that forgets to take
stuff with them. I’ll leave it at the pub, the public transport, the swimming
pool… somewhere. It would be too precious and expensive to run that risk.</li>
<li>Similar to the point above, I know I’m going to drop it. Smartphones are
also fragile. I wouldn’t be at ease carrying one around every day.</li>
<li>Not only the terminals are expensive, but also there are high usage costs.
Often involving contracts that bind you to an operator for over a year.</li>
<li>Smartphone technology moves quickly, and terminals are continuously going
out of date. There’s a new iPhone every year, and approximately a new “blessed”
Android phone every year too. Eventually (even more true with Androids) software
updates stop happening and you end up with an outdated phone anyway.</li>
<li>Modern phones are ridiculously large. Honestly, I still think that the
iPhone is too large. I don’t want to carry a brick around. And don’t even get me
started on the Samsung Galaxy Note…</li>
<li>With great power comes great responsibility, in this case in the form of a
security risk. There have been reports of apps and phones (some popular) doing
dodgy things behind the user’s back. Also, imagine the damage if I lost my phone
and somebody accessed sensitive information.</li>
</ul>
<p>Of course this is a tradeoff. Smartphones are a great technology, and mobile
apps can be really useful. But honestly, I really don’t see a need for them in
my life. Incidentally, I find this makes me more organised: I check maps,
timetables, and any other information in advance. I arrive in places knowing
what to do, because I have done my homework. I am not so dependent on technology
as users of smartphones seem to be, so often checking their maps, social network
statuses (this one really irks me), calendars and God knows what.</p>
<p>I don’t need any of that. I want a simple phone. Mine gives me this:</p>
<ul>
<li>Voice calls and text messages. (Incredible technology!).</li>
<li>The terminal was so cheap I bought two just in case. The total cost of
purchase was £35.99, with one charger.</li>
<li>My current plan gives me more minutes and texts than I can use, for about
£5 a month. And I can leave anytime</li>
<li>A fully charged battery can go on for five days</li>
<li>It’s durable and sturdy. I can drop it and nothing will happen.</li>
<li>It’s compact and fits my pockets easily</li>
<li>Nobody is going to be tempted to steal it</li>
</ul>
<p>Of course, this doesn’t guarantee that I won’t end up owning a smartphone one
day, although I am not in a rush. For now, I would like to wait until I see a
reasonably-priced, small terminal with battery life of at least 3 days.</p>
Browser privacyhttps://blog.pablobm.com/2012/03/05/browser-privacy/2012-03-05T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>For some time now, I have been using the <a href="http://en.wikipedia.org/wiki/Privacy_mode">private browsing
mode</a> (or “incognito mode” or “porn
mode”) of my browsers for casual usage. I feel wary of the incredible amount of
information that I can let out about myself if I do otherwise.</p>
<h2 id="real-world-cases">Real-world cases</h2>
<p>I had been aware of this for some time, and we have had some real world
examples. For example, in 2006 <a href="http://en.wikipedia.org/wiki/AOL_search_data_scandal">AOL released search data for research
purposes</a>. All published
records were anonymised, by means of showing user IDs instead of real names. But
it was still easy to recognise many people by the things they were looking for,
as well as group up their searches and know more about them.</p>
<p>Something similar happened to Netflix in 2008. As part of a data mining
competition, they published their logs of film viewing. Then somebody thought of
<a href="http://www.cs.utexas.edu/~shmat/netflix-faq.html">cross referencing them with rating data posted by users on
IMDB</a>. They were able to find
matches for many of those.</p>
<h2 id="revelation">Revelation</h2>
<p>But anyway, thing is, this really struck me this day when I was using Google
Chrome and I entered a URL on the address bar (the Omnibox as they call it). It
was then that I realised that Chrome was trying to autocomplete what I entered.</p>
<p>Autocomplete a URL. Think about it. It was not even an attempt to search, but a
URL. Chrome was asking Google for candidates to autocomplete what I was
entering. As part of the process, Chrome was telling Google exactly what I was
entering on the address bar, real time. This meant that I didn’t even have to
explicitly perform a search on Google for them to know what was the next website
I was going to visit.</p>
<p>By the way, just in case it was autocompleting from the browser’s own history or
something like that, I checked. The following screenshot shows the requests that
my browser fired when I was entering a URL on the address bar. (By the way, I
used <a href="http://www.charlesproxy.com/">Charles Web Debugging Proxy</a> for this.
Pretty nifty tool).</p>
<figure style="max-width: 412px;" class="post-illustration post-illustration--center">
<div class="post-illustration__images">
<img src="/images/nosy-http-requests.png" alt="Chrome telling Google exactly where I intend to be. Potentially cheeky!" />
</div>
<figcaption>
<p class="description">Chrome telling Google exactly where I intend to be. Potentially cheeky!</p>
</figcaption>
</figure>
<p>Chrome telling Google exactly where I intend to be. Potentially cheeky!</p>
<p>Of course I could turn this specific feature off, but that’s not really the
point and you know it. People don’t change their defaults, and I mean the
average user of the web, not you and not me<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. Oh,
and of course this doesn’t save me from the Netflix scenario. There’s not really
much helping against that.</p>
<h2 id="indirect-consequences-the-filter-bubble">Indirect consequences: The Filter Bubble</h2>
<p>Some will still say “Well, I don’t care about my personal data, blah, blah,
doesn’t affect me, blah, blah”. Well, it affects you in more ways than the
evident. In fact: it is already affecting you in a very subtle, yet dramatic
way.</p>
<p>Some time in 2011, I saw this short TED talk were Eli Pariser introduced (at
least to me) the term “<a href="http://www.thefilterbubble.com/ted-talk"><strong>The Filter
Bubble</strong></a>”. Go have a look, it’ll take
only 9 minutes of your time and is very enlightening:</p>
<p>If you didn’t have 9 minutes, don’t worry, this is the skinny of it:</p>
<ul>
<li>Google, Facebook, Yahoo, etc… They keep a history of what contents you
prefer.</li>
<li>Since they know what you normally go for, they’ll feed you more of it.</li>
<li>Eventually, things that you didn’t show interest on will be removed from
your search results, news feeds, etc.</li>
</ul>
<p>Initially, that’s all good and well but: what about things that are important,
but you just don’t check that often? Or what about challenging opinions? Will I
just be told what I want to hear instead of what is actually true, or at least
unbiased or just from a different point of view?</p>
<p>In one of the examples in the talk, conservatives were being given only
conservative links, and similarly with progressives. All this based in data such
as search history, links clicked on Facebook or what have you. At the end of the
day, people would believe that this information bubble they live is a reflection
of the world as it is. However, it’ll be only a reflection of the world as they
would like it to be. Wishful thinking.</p>
<h2 id="solutions">Solutions</h2>
<p>I don’t think there’s much of a solution, no. Logging of habits that takes place
while you are not logged into a site, those can be solved to an extent by
browsing anonymously, in incognito mode. However, for browsing that happens
while you are logged into Facebook, GMail… there’s no easy way out.</p>
<p>Only thing you can do is keep all this in mind when you use the Internet. And by
the way: is really Facebook the best way to use your time? Just sayin’.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Well, not even me to be honest. For instance: I use MacOS X Lion at work
and Snow Leopard at home. On neither of them I have changed the default mouse
scrolling. I do have this inclination for using standard configurations for
everything I use (with some exceptions). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
32https://blog.pablobm.com/2012/01/07/32/2012-01-07T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<ul>
<li>25</li>
<li>Germanium</li>
<li>Space</li>
<li>Beethoven Piano sonatas</li>
<li>Kabbalistic Paths of Wisdom</li>
<li>Magic Johnson</li>
<li>Me</li>
</ul>
ActiveRecord: getting a backtrace of your SQL querieshttps://blog.pablobm.com/2011/12/06/activerecord-backtrace-sql-queries/2011-12-06T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>The other day I had some strange database queries in my Rails log. I didn't
know what was originating them, so I set out to track down their origin.</p>
<p>Thus I found out about <code>ActiveSupport::Notifications</code>. It is used by Rails to
notify about events ocurring inside the framework, and is used to generate the
standard logs that we see in our applications. We can turn them to our
advantage to solve the problem at hand.</p>
<p>We can tap into these notifications to find out what is generating those DB
queries. To do this, we print out the backtrace of the program when we get
notified of an SQL query happening. Like this:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span><span class="comment"># lib/query_tracer.rb</span>
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span><span class="keyword">module</span> <span class="class">QueryTracer</span>
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> <span class="comment"># Borrowed some ANSI color sequences from elsewere</span>
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> <span class="constant">CLEAR</span> = <span class="constant">ActiveSupport</span>::<span class="constant">LogSubscriber</span>::<span class="constant">CLEAR</span>
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> <span class="constant">BOLD</span> = <span class="constant">ActiveSupport</span>::<span class="constant">LogSubscriber</span>::<span class="constant">BOLD</span>
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> <span class="keyword">def</span> <span class="predefined-constant">self</span>.<span class="function">start!</span>
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> <span class="comment"># Tap into notifications framework. Subscribe to sql.active_record messages</span>
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> <span class="constant">ActiveSupport</span>::<span class="constant">Notifications</span>.subscribe(<span class="string"><span class="delimiter">'</span><span class="content">sql.active_record</span><span class="delimiter">'</span></span>) <span class="keyword">do</span> |*args|
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> <span class="constant">QueryTracer</span>.publish(*args)
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> <span class="comment"># Notice the 5 arguments that we can expect</span>
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> <span class="keyword">def</span> <span class="predefined-constant">self</span>.<span class="function">publish</span>(name, started, ended, id, payload)
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> name = payload[<span class="symbol">:name</span>]
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> sql = payload[<span class="symbol">:sql</span>]
<span class="line-numbers"><a href="#n18" name="n18">18</a></span>
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> <span class="comment"># Print out to logs</span>
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> <span class="constant">ActiveRecord</span>::<span class="constant">Base</span>.logger.debug <span class="string"><span class="delimiter">"</span><span class="inline"><span class="inline-delimiter">#{</span><span class="constant">BOLD</span><span class="inline-delimiter">}</span></span><span class="content"> TRACE: </span><span class="inline"><span class="inline-delimiter">#{</span>sql<span class="inline-delimiter">}</span></span><span class="inline"><span class="inline-delimiter">#{</span><span class="constant">CLEAR</span><span class="inline-delimiter">}</span></span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> clean_trace.each <span class="keyword">do</span> |line|
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> <span class="constant">ActiveRecord</span>::<span class="constant">Base</span>.logger.debug <span class="string"><span class="delimiter">"</span><span class="content"> </span><span class="inline"><span class="inline-delimiter">#{</span>line<span class="inline-delimiter">}</span></span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n24" name="n24">24</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n25" name="n25">25</a></span>
<span class="line-numbers"><a href="#n26" name="n26">26</a></span> <span class="keyword">def</span> <span class="predefined-constant">self</span>.<span class="function">clean_trace</span>
<span class="line-numbers"><a href="#n27" name="n27">27</a></span> <span class="constant">Rails</span>.backtrace_cleaner.clean(caller[<span class="integer">2</span>..<span class="integer">-1</span>])
<span class="line-numbers"><a href="#n28" name="n28">28</a></span> <span class="keyword">end</span>
<span class="line-numbers"><a href="#n29" name="n29">29</a></span><span class="keyword">end</span>
<span class="line-numbers"><strong><a href="#n30" name="n30">30</a></strong></span>
<span class="line-numbers"><a href="#n31" name="n31">31</a></span><span class="constant">QueryTracer</span>.start!
</pre></div>
</div>
</div>
<p>To make this work, we only need to require the module from anywhere
appropriate. An initializer would be a good place, for example:</p>
<div class="language-ruby code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="comment"># config/initializers/query_tracer.rb</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>require <span class="string"><span class="delimiter">'</span><span class="content">query_tracer</span><span class="delimiter">'</span></span>
</pre></div>
</div>
</div>
<p>Now, for each SQL query generated within ActiveRecord, we will get the
following output in the logs:</p>
<div class="language-plaintext code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span> **TRACE: INSERT INTO \`user\_words\` (\`created\_at\`, \`updated\_at\`, \`user\_id\`, \`word_id\`) VALUES ('2011-11-27 18:33:53', '2011-11-27 18:33:53', 2, 2)**
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> app/models/user.rb:18:in `block in add_word'
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> app/models/user.rb:12:in `tap'
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> app/models/user.rb:12:in `add_word'
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> app/controllers/words_controller.rb:6:in `create'
</pre></div>
</div>
</div>
<p>And that's it!</p>
Amazon's Kindle customer servicehttps://blog.pablobm.com/2011/11/26/amazons-kindle-customer-service/2011-11-26T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<figure style="max-width: 150px;" class="post-illustration post-illustration--side">
<div class="post-illustration__images">
<a href="http://en.wikipedia.org/wiki/Amazon_Kindle"><img src="/images/amazon-kindle-3.jpeg" /></a>
</div>
<figcaption>
<p class="post-illustration__attribution">Image by <a href="http://en.wikipedia.org/wiki/File:Amazon_Kindle_3.JPG">NotFromUtrecht</a></p>
<p class="post-illustration__license"><a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/deed.en">CC-BY-SA License</a></p>
</figcaption>
</figure>
<p>Two weeks ago, I lost my Kindle. I had it at my brother's place, but then I
didn't have it on the tube. No clue of what could have happened in the middle.</p>
<p>I was already thinking of buying a new one, when I got a call on my phone. It
was an Amazon customer services rep. They had my Kindle. Somebody had returned
it to them.</p>
<p>Oh, and they were shipping it back to me for free. It's back with me now.</p>
<p>Another related story. This is actually my second Kindle. The first one had a
problem, half the screen was dead. I called Amazon at 6pm, and I got a
replacement by 10am the next morning. For free again. They also arranged for a
courier to collect the broken one any time that suited me.</p>
<p>Honestly, it makes me feel a bit bad that I have only bought one Kindle book from them since I first purchased the device. The rest have been books I downloaded from Project Gutenberg and such.</p>
<p>It's just that I hate DRM. <strong>Please publishers, sell more DRM-free books</strong> :-(</p>
<p>Everyone else: buy the Amazon Kindle. It's an awesome device.</p>
Eclipse doesn't let me to findViewByIdhttps://blog.pablobm.com/2011/11/22/eclipse-android-findviewbyid/2011-11-22T00:00:00+00:002024-02-16T11:30:28+00:00Pablo Brasero Morenohttps://www.pablobm.com
<p>Yesterday I did my first bit of Android development ever. I created a very
simple program that simply updated some text on screen.</p>
<p>I had a problem though. On my Activity, I had the following piece of code:</p>
<div class="language-js code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="reserved">public</span> <span class="keyword">void</span> updateText(View view) {
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> TextView t = (TextView)findViewById(R.id.button);
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> t.setText(<span class="string"><span class="delimiter">"</span><span class="content">Updated!</span><span class="delimiter">"</span></span>);
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>}
</pre></div>
</div>
</div>
<p>And the following bit ox XML in my layout (inside a LinearLayout):</p>
<div class="language-xml code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><Button</span>
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> <span class="attribute-name">android:id</span>=<span class="string"><span class="delimiter">"</span><span class="content">@+id/button</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span> <span class="attribute-name">android:layout_width</span>=<span class="string"><span class="delimiter">"</span><span class="content">wrap_content</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> <span class="attribute-name">android:layout_height</span>=<span class="string"><span class="delimiter">"</span><span class="content">wrap_content</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> <span class="attribute-name">android:text</span>=<span class="string"><span class="delimiter">"</span><span class="content">@string/button</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span> <span class="attribute-name">android:onClick</span>=<span class="string"><span class="delimiter">"</span><span class="content">updateText</span><span class="delimiter">"</span></span>
<span class="line-numbers"><a href="#n7" name="n7">7</a></span><span class="tag">/></span>
</pre></div>
</div>
</div>
<p>Eclipse insisted that <code>R.id</code> was illegal. Specifically, the error was <code>id
cannot be resolved or is not a field</code>, even though the button did exist in the
view.</p>
<p>The problem was that I had not yet defined the string "button" in my
<code>strings.xml</code> file:</p>
<div class="language-xml code-block code-block--coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><string</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">"</span><span class="content">button</span><span class="delimiter">"</span></span><span class="tag">></span>Click me!<span class="tag"></string></span>
</pre></div>
</div>
</div>
<p>Not sure of the exact internals, but I guess that Eclipse was trying to compile
the layout, and it failed because the string was missing. In turn, this showed
an error in the activity, because it was using a broken layout.</p>