Not Invented Herehttps://boddy.im/Thu, 16 Nov 2017 00:00:00 +0000How I use Vimhttps://boddy.im/vim-dev-env.html<div class="highlight"><pre>This is my .vimrc file. There are many like it, but this one is mine. </pre></div> <h2>Outline</h2> <ul> <li><a href="#introduction">Introduction</a></li> <li><a href="#install">Installation</a></li> <li><a href="#plugins">Plugins</a><ul> <li><a href="#filesystem">Navigating the local filesystem</a></li> <li><a href="#git">Version control with Git</a></li> <li><a href="#status">Status Bar</a></li> <li><a href="#linting">Linting</a></li> <li><a href="#python">Python</a></li> <li><a href="#tagbar">Tagbar</a></li> </ul> </li> <li><a href="#persistent_undo">persistent_undo</a></li> <li><a href="#misc">vim miscellany</a></li> <li><a href="#motivation">Motivation</a></li> </ul> <p><a name="introduction" ></a></p> <h1>Introduction</h1> <p>I've used <a href="http://www.vim.org/">vim</a> as my primary editor/IDE for the past 10 years. Back then I was writing data-analysis code over an ssh session to hosts in the <a href="https://duckduckgo.com/?q=SLAC+stanford&amp;t=canonical&amp;ia=web">SLAC</a> computing center in California while sitting several thousand miles away in front of a terminal in the UK and a friendly post-doc showed me this nifty looking editor that he was using... </p> <p>Since then I've been tempted away by a few shiny IDEs but always give up and come back to the venerable editor (one exception to this is <a href="https://www.jetbrains.com/idea/">intellij</a> as it's a wonderful refactoring-engine for Java. By comparison Pycharm is rubbish IMHO).</p> <p>And so I thought it might be useful to post about my <code>.vimrc</code> file explaining what the various plugins and configuration options do. Since I've been writing a lot of python over the past year there's a bias here.</p> <p>This scope of this post creeped up but it is mainly:</p> <ul> <li>a walk-through of my <code>.vimrc</code> file (anything not explained is hopefully obvious from the comments) </li> <li>a litany of the useful vim-plugins therein and how/why to use them</li> <li>some general vim tips</li> </ul> <p><a name="install" ></a></p> <h2>Installation</h2> <p>If you want to skip to the goods you can clone my <a href="https://github.com/cboddy/dot-files">dot-files repo</a> and run <a href="https://github.com/cboddy/dot-files/blob/master/vim.sh">vim.sh</a> to download and install all of the plugins below. </p> <p>Updating to a more recent version of vim is usually trivial though, for Debian/Ubuntu you can do: </p> <div class="highlight"><pre>sudo add-apt-repository ppa:jonathonf/vim sudo apt update sudo apt install vim </pre></div> <p>If you don't have vim-8 installed all the plugins except ALE will work anyway.</p> <p><a name="plugins" ></a></p> <h2>Plugins</h2> <h3>Vundle</h3> <p>Let's start with <a href="https://github.com/VundleVim/Vundle.vim">Vundle</a>: a plugin manager for vim that also fetches plugins straight from Github. We start by setting the runtime-path (where Vundle will download the plugins files).</p> <div class="highlight"><pre>set rtp+=~/.vim/bundle/Vundle.vim call vundle#begin() Plugin &#39;gmarik/Vundle.vim&#39; </pre></div> <p><a name="filesystem" ></a></p> <h2>Navigating the local filesystem</h2> <h3>NERDTree</h3> <p><a href="https://github.com/scrooloose/nerdtree">NERDTree</a> is for exploring the file-system and is sometimes useful for jumping between files in your current project. Personally I don't like this being open on-startup.</p> <div class="highlight"><pre>Plugin &#39;scrooloose/nerdtree&#39; &quot; Open/close NERDTree Tabs with let NERDTreeDirArrows=0 &quot; To have NERDTree not open on startup let g:nerdtree_tabs_open_on_console_startup = 0 </pre></div> <p>A few highlights:</p> <ul> <li>to open a new NERDTree tab use <code>:NERDTree</code></li> <li>to open a file under the cursor in a v-split pane hit <code>s</code>.</li> <li>closes like a normal vim tab</li> </ul> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/nerd_tree.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/nerd_tree.gif" alt="NERD-tree"/> </a></p> <h3>ctrlp</h3> <p><a href="https://github.com/ctrlpvim/ctrlp.vim">ctrlp</a> is the most useful way to navigate between files while working on a project IMHO.</p> <div class="highlight"><pre>Plugin &#39;ctrlp/ctrlp.vim&#39; &quot; search ctrlp with regex by default let g:ctrlp_regexp_search = 1 let g:ctrlp_clear_cache_on_exit = 0 </pre></div> <p>Quite simply, press <code>Ctrl+p</code> while in normal mode to open a buffer that will match to filenames in the directory you're in (and recursively in it's sub-directories). The options I've set will:</p> <ul> <li>allow you to use regex to match filenamese</li> <li>cache the directory index (<code>F5</code> to refresh). This is useful if like me you ever have to work with a shared NFS mounts on which the indexing operation can be sluggish</li> </ul> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/ctrlp.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/ctrlp.gif" alt="ctrlp"/> </a></p> <p><a name="git" ></a></p> <h2>Version control with Git</h2> <h3>vim-fugitive</h3> <p>The <a href="[https://github.com/tpope/vim-fugitive">vim-fugitive repo</a> says it better than I could:</p> <div class="highlight"><pre>I&#39;m not going to lie to you; fugitive.vim may very well be the best Git wrapper of all time </pre></div> <p>As someone that uses vim-fugitive to interface with git every day I don't consider this an understatement. This is a large topic but my top highlights are:</p> <ul> <li><code>:GCommit, :Gpush, :Gpull</code> wrap <code>git {commit|push|pull}</code></li> <li><code>:Gblame</code>: opens a vertical-split window with git-blame info, and you can then open a commit and see the diff in another bugger</li> <li><code>:Gdiff</code>: see diff output for the current file.</li> <li>You can pass any command directly to git by starting a command with <code>:Git</code>, for example add the current file to the change-set with <code>:Git add path/to/current/file</code> or just <code>:Git add %</code></li> <li>we're using fugitive to display the git branch in the status line with the lightline plugin (see below).</li> </ul> <div class="highlight"><pre>Plugin &#39;tpope/vim-fugitive&#39; </pre></div> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/fugitive.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/fugitive.gif" alt="fugitive"/> </a></p> <h3>vim-gitgutter</h3> <p><a href="https://github.com/airblade/vim-gitgutter">vim-gitgutter</a> complements fugitive by displaying the git sign for lines that have added/removed/updated in the current file (using the output <code>git diff</code>)</p> <div class="highlight"><pre>Plugin &#39;airblade/vim-gitgutter&#39; </pre></div> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/gitgutter.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/gitgutter.gif" alt="gitgutter"/> </a></p> <h3>Bonus: configure git to use vimdiff</h3> <p><code>vimdiff</code> is a diff-tool that opens a vertical-split window in vim. I find it much easier to see what's changed than when using git's in-built diff tool. Here we jump to my <code>.gitconfig</code>:</p> <div class="highlight"><pre><span class="k">[diff]</span> <span class="na">tool</span> <span class="o">=</span> <span class="s">vimdiff</span> <span class="k">[difftool]</span> <span class="na">prompt</span> <span class="o">=</span> <span class="s">false</span> <span class="k">[alias]</span> <span class="na">d</span> <span class="o">=</span> <span class="s">difftool</span> </pre></div> <p>We can now look at the output of <code>git diff</code> in <code>vimdiff</code>. For example, to diff the curreent index with two commits ago you can do </p> <div class="highlight"><pre>git d HEAD~2 </pre></div> <p>Highlights of this method is that when navigating the file in either window:</p> <ul> <li>the files can be edited as if they were normally opened files in vim</li> <li>"dp" will put the change under the current cursor to the other file</li> <li>"do" with obtain the change from the other file and apply it to the cursor position in the current file</li> </ul> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/vimdiff.png"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/vimdiff.png" alt="vimdiff" /> </a></p> <p><a name="status" ></a></p> <h2>Status Bar</h2> <h3>lightline</h3> <p><a href="https://github.com/itchyny/lightline.vim">lightline</a> allows you to configure a snazzy status bar (at the bottom of the window). The snippet below will show:</p> <ul> <li>a color-scheme named <code>wombat</code> for the status-bar itself</li> <li>the vim-mode (<code>INSERT</code>/<code>NORMAL</code>/<code>VISUAL</code>)</li> <li>the git-branch if the file in the current buffer is part of a git repo (more on this later)</li> <li>the python-virtualenv in use (more on this later).</li> </ul> <div class="highlight"><pre>Plugin &#39;itchyny/lightline.vim&#39; &quot; set lightline to include git-branch let g:lightline = { \ &#39;colorscheme&#39;: &#39;wombat&#39;, \ &#39;active&#39;: { \ &#39;left&#39;: [ [ &#39;mode&#39;, &#39;paste&#39; ], \ [ &#39;gitbranch&#39;, &#39;readonly&#39;, &#39;filename&#39;, &#39;modified&#39; ] , \ [ &#39;venv&#39;, &#39;readonly&#39;] ] \ }, \ &#39;component_function&#39;: { \ &#39;gitbranch&#39;: &#39;fugitive#head&#39;, \ &#39;venv&#39;: &#39;virtualenv#statusline&#39; \ }, \ } &quot; Always show statusbar set laststatus=2 </pre></div> <p>This produces the following status bar:</p> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/lightline.png"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/lightline.png" alt="lightline status bar"/> </a></p> <p><a name="linting" ></a></p> <h2>Linting and syntax highlighting</h2> <p>Many people use syntastic for syntax highlighting in vim. It has one drawback which is that it runs synchonously in the main UI thread. <a href="https://github.com/w0rp/ale">ALE</a> (Asynchronous Lint Engine) is arguably an asynchronous successor.</p> <h3>ale</h3> <p>ALE let's you specify specific linters for different languages. Below I've specified <code>flake8</code> for linting and <code>autopep8</code> for fixing syntax "errors". </p> <div class="highlight"><pre>Plugin &#39;w0rp/ale&#39; &quot; pip install flake8 let g:ale_linters = {&#39;python&#39;: [&#39;flake8&#39;]} let g:ale_fixers = {&#39;python&#39;: [&#39;remove_trailing_lines&#39;, &#39;trim_whitespace&#39;, &#39;autopep8&#39;]} </pre></div> <p>Note:</p> <ul> <li>this assumes flake8 and autopep8 are already installed </li> <li>any config for either in <code>~/.config/pep8</code> or <code>~/.config/flake8</code> will be respected.</li> <li>the linting is done asynchronously </li> <li>to apply the fixer (in this case <code>autopep8</code>) use the command <code>:ALEFix</code></li> </ul> <p>This produces:</p> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/ale.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/ale.gif" alt="ale"/> </a></p> <p><a name="python" ></a></p> <h2>Python</h2> <p>Here are a few plugins that are great for python specifically.</p> <h3>jedi-vim</h3> <p><a href="https://github.com/davidhalter/jedi-vim">jedi</a> is probably the best auto-completion library for Python that I've ever used (including Pycharm). jedi-vim requires that your vim binary was compiled with the <code>python</code>/<code>python3</code> option but that's pretty much guaranteed unless you're compiling vim yourself.</p> <div class="highlight"><pre><span class="s2">&quot; ------ jedi-python ------&quot;</span> <span class="n">Plugin</span> <span class="s1">&#39;davidhalter/jedi-vim&#39;</span> <span class="n">let</span> <span class="n">g</span><span class="p">:</span><span class="n">jedi</span><span class="c1">#popup_on_dot = 0</span> <span class="nb">map</span> <span class="o">&lt;</span><span class="n">Leader</span><span class="o">&gt;</span><span class="n">b</span> <span class="n">Oimport</span> <span class="n">pdb</span><span class="p">;</span> <span class="n">pdb</span><span class="o">.</span><span class="n">set_trace</span><span class="p">()</span> <span class="c1"># BREAKPOINT&lt;C-c&gt;</span> </pre></div> <p>Anyway, this is another large topic but here are a few highlights:</p> <ul> <li>autocomplete (<code>C-x-o</code>)</li> <li>jump-to variable/method/function/class definition (<code>leader+g</code>) <ul> <li>this works with defined on <code>PYTHONPATH</code> (including in your virtual-env) but not egg files weirdly)</li> </ul> </li> <li>find-usages (<code>leader+f</code>)</li> <li>jump-to method/class/function/variable declaration (<code>leader+d</code>)</li> <li>rename (<code>leader+r</code>)</li> <li>show docstring with <code>K</code></li> <li>As a bonus: <code>leader+b</code> is remapped to add a pdb-breakpoint. </li> </ul> <p>Below shows a few of these in action:</p> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/jedi.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/jedi.gif" alt="jedi"/> </a></p> <h3>vim-virtualenv</h3> <p><a href="https://github.com/jmcantrell/vim-virtualenv">vim-virtualenv</a> allows you to set the python-virtualenv used in vim (and by extension <code>jedi-vim</code>). You can set/unset it with <code>:VirtualEnvActivate virtual_env_name</code> or <code>:VirtualEnvDeactivate</code>.</p> <p><a name="tagbar" ></a></p> <h3>tagbar</h3> <p>Personally I find tags of limited use, but they're handy for exploring a new or otherwise un-familiar codebase. Here we use <a href="https://github.com/majutsushi/tagbar">tagbar</a>. </p> <div class="highlight"><pre>&quot; --- C-tags integration --- &quot; Plugin &#39;majutsushi/tagbar&#39; </pre></div> <p>Note, this requires <code>ctags</code> or similar to be installed, so for Debian/Ubuntu:</p> <div class="highlight"><pre>sudo apt-get install ctags </pre></div> <p>Then we can toggle the tagbar window with <code>:Tagbar</code> and navigate code at a higher level of abstraction, as shown below.</p> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/tagbar.gif"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/tagbar.gif" alt="tagbar"/> </a></p> <h2>Other useful plugins</h2> <p>Honourary mention to <code>vim-go</code> and <code>rust.vim</code> which provide great integration for those languages too. </p> <p>Also, for keeping a work-log or wiki I can't recommend <a href="http://vimwiki.github.io">vimwiki</a> enough: it is like an even-better org-mode ;-)</p> <p><a name="persistent_undo" ></a></p> <h2>persistent-undo</h2> <p>This isn't a plugin but something I find useful for essentially caching the list of updates to a file such that if you close and re-open a file you can undo the last few edits (note, most vim distros in the wild will be compiled with the <code>persistent_undo</code>") </p> <div class="highlight"><pre><span class="x">&quot; Keep undo history across sessions by storing it in a file</span> <span class="x">if has(&#39;persistent_undo&#39;)</span> <span class="x"> let undo_dir = expand(&#39;</span><span class="p">$</span><span class="nv">HOME</span><span class="x">/.vim/undo_dir&#39;)</span> <span class="x"> if !isdirectory(undo_dir)</span> <span class="x"> call mkdir(undo_dir, &quot;&quot;, 0700)</span> <span class="x"> endif</span> <span class="x"> set undodir=</span><span class="p">$</span><span class="nv">HOME</span><span class="x">/.vim/undo_dir</span> <span class="x"> set undofile</span> <span class="x">endif</span> </pre></div> <h2>Colours</h2> <p>It's possible to set the colour-scheme directly in vim but these days most terminals (<a href="https://gnometerminator.blogspot.de/p/introduction.html">terminator</a> for linux and <a href="https://www.iterm2.com/">iTerm2</a> for mac) have <a href="http://ethanschoonover.com/solarized">solarized</a> colour schemes baked in. Once set, vim will use these and the job is done.</p> <h2>tmux</h2> <p>On a different note, <a href="https://github.com/tmux/tmux/wiki">tmux</a> is a wonderful way to multiplex teminal sessions and I find it complements vim very nicely. </p> <p>I could probably fill an equally lengthy post about tmux. </p> <p><a name="misc" ></a></p> <h2>vim miscellany</h2> <p>Again, this is a much larger topic but here are a few tricks I find useful:</p> <ul> <li><code>:sp</code> and <code>:vs</code> will split the current window horizontally/vertically</li> <li><code>gg=G</code> will auto-indent the entire file</li> <li><code>e#</code> navigates to the previously opened file in the current buffer</li> <li><a href="http://vim.wikia.com/wiki/Macros">vim macros</a> are pretty rad</li> <li>you can use <code>%</code> as a shortcut for the current file-path in any command in command mode e.g. get it's md5-hash <code>:! md5sum %</code></li> <li>you can read from stdin via a UNIX pipe with <code>vim -</code>, e.g. </li> </ul> <div class="highlight"><pre>&gt;&gt; curl example.com | vim - </pre></div> <p><a name="motivation" ></a></p> <h2>Motivation</h2> <p>Warning: this section is opinionated. </p> <p>You might ask: why bother with all this? Surely [insert GUI-based IDE + whatever else] is good enough? </p> <p>Well, it may be good enough. But often when working on some data-analysis or building some service what matters far more than the length of time spent working is <strong>how long I can spend in a state of flow</strong>. This usually means being focussing on one thing; being un-distracted. Using this editor is a great way of staying focussed and minimising the operational overheads of software engineering. </p> <p>Things I find good for staying focussed:</p> <ul> <li>Keeping my eyes fixed on one application</li> <li>Having things like auto-complete and linters</li> <li>Using as few keystrokes as possible for common tasks (eg. VCS commands, running tests)</li> </ul> <p>Conversely, these have the opposite effect:</p> <ul> <li>Navigating menus</li> <li>Using a mouse at all</li> <li>Switching to another terminal window or (much) worse a web-GUI</li> </ul> <p>So in a way these plugins and hotkeys are a way of making it easier to stay in the flow.</p> <h2>vim-8 is awesome</h2> <p>Most package managers will distribute binary versions of vim 7.4.XXXX which is fine. There are many reasons to update to vim version 8.X is but perhaps the biggest is that it allows tasks to be performed outside the main UI thread. All of the plugins we'll discuss will work fine with either version <strong>except for the ALE plugin that requires version 8+</strong>. Also, certain commonly-enabled compilation flags (<code>python</code> and <code>persistent_undo</code> are assumed).</p> <h2>Wrapping up</h2> <p>Putting it altogether, what I focus on while working usually looks something like this. </p> <p><a href="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/fullshot.png"> <img style="max-width:100%" src="https://raw.githubusercontent.com/cboddy/_vim_gifs/master/fullshot.png" alt="full working environment"/> </a></p> <p>If you've made it this far I hope found it useful.</p>Chris BoddyThu, 16 Nov 2017 00:00:00 +0000tag:boddy.im,2017-11-16:vim-dev-env.htmlvimeditorpythonMuting the newshttps://boddy.im/android-mute-scheduler.html<h1>TL; DR</h1> <p><a href="https://play.google.com/store/apps/details?id=im.boddy.MuteTime3">Mute the news</a> when listening to the radio on a thing running Android.</p> <h1>Introduction</h1> <p>As someone who listens to a lot of <a href="http://www.bbc.co.uk/6music">BBC Radio 6</a>, I highly recommend pretty much all of the weekend shows, they're usually full of really great music. </p> <p>As part of the radio service, the <a href="https://en.wikipedia.org/wiki/BBC_News">BBC News</a> butts-in with a 2-3 minute broadcast at hourly intervals, full of current events to help keep the public informed. Now, aside from the known <a href="https://www.theguardian.com/media/2013/apr/12/news-is-bad-rolf-dobelli">adverse effects of consuming too much news</a>, the tone of the broadcast is usually completely at odds with what I'm doing at the time. Why should I be made aware of what someone said in London or the latest tweet of a large-tangerine with small hands? Why must the minds of millions be made aware of minor crimes in major cities? At times during the run up to the Brexit referendum it felt like listening to the Nigel Farage hate-trumpet.</p> <p>In consuming media this way a person acquiesces control to the broadcaster, allowing them to shape your reality with great subtlety, often making you morbidly aware of seemingly endless minutiae for the political or commercial gain.</p> <p>But I digress.</p> <p>For some time I've just reached over and muted the radio when the news jingle starts. But last weekend I had a few hours to kill in a pub before meeting Sarah to play <a href="https://boardgamegeek.com/boardgame/2655/hive">Hive</a>. So obviously I wrote an Android App (below) to mute the streaming-music volume at set times of the day for a configurable period of time. Note this doesn't affect the phone-ringer or alarm volumes, only streaming music like playing the radio through a web-browser and can be (de) activated with a toggle.</p> <h3>mute.time</h3> <p>The source-code is on <a href="https://github.com/cboddy/mute.time">github</a>, and the (free, open-source, ad-free) app is published on the <a href="https://play.google.com/store/apps/details?id=im.boddy.MuteTime3">Android play-store</a>.</p> <p><a href="/images/mute_time_screenshot.png"> <img style="max-width:100%" src="/images/mute_time_screenshot.png" alt="Screenshot of the mute-time utility."/> </a></p>Chris BoddySat, 04 Feb 2017 00:00:00 +0000tag:boddy.im,2017-02-04:android-mute-scheduler.htmlandroidmutejava2016 in Books (part 2)https://boddy.im/books-2016-part-2.html<h1>Introduction</h1> <p>Anyone looking for a book?</p> <p>Below is a brief review of some books I've read in the second half of 2016.</p> <p>This makes just over 40 books read in 2016, next year I'm tempted to include some graphs and maybe a few journal articles.</p> <h1>Non Fiction</h1> <h3>Dreams of a Final Theory: Steven Weinberg</h3> <p>In the late 80s a particle-physics experiment called the Superconducting Super Collider (SSC) had it's funding cut, despite the fact that construction was well under way. This was most likely because Texas (where it was being built) also had a lot of money being spent there as part of the International Space Station project and the ensuing political difficulties that followed the concentration of resources into just one state. Over many years a gradual but consistent decline in the Department of Energy budget put them at odds and the SSC lost. </p> <p>The main thrust of this book is Weinberg (who won part of the Nobel Prize for the theory of Electroweak Symmetry Breaking) arguing for the SSC over the scientifically bereft ISS, along with a now standard popular-science-level Particle Physics introduction and a lot of less-standard ranting about religion and philosophers of science. </p> <p>There are many parallels between then and now and you can almost replace Tevatron/LEP with LHC and the arguments for building the next-generation of particle physics experiments after the LHC and the existential crisis facing Particle Physics experiments and theory in the absence of significant new physics from the LHC (except for a SM-like Higgs of course) are very relevant.</p> <h3>The Elements of Computing Systems: Building a Modern Computer from First Principles: Noam Nisson &amp; Simon Schocken</h3> <p>This book/course (that also goes by the moniker 'From Nand to Tetris') took a month but what a rewarding endeavour. The first chapter engages the reader to derive logic operators (AND, NOT, OR) purely in terms of NAND (like a transistor) and gives a scaffolding to build:</p> <ul> <li>an ALU that runs on (a provided) emulator;</li> <li>a 16-bit assembler for a somewhat simplified instruction set that runs on said emulator;</li> <li>a stack based VM that compiles to said 16-bit instruction-set;</li> <li>a high level Java/.Net style compiler/language spec to run on the stack VM;</li> <li>something between a light-weight OS and a library for the high level language for interacting with memory mapped input devices, allocating memory etc.</li> </ul> <p>I can't recommend this book enough for understanding each level of abstraction in a running computer.</p> <h3>Boss Fight Books: Spelunky: Derek Yu</h3> <p>Spelunky is a game with which I became somewhat obsessed years ago, logging many, many hours of play. Having studied (and read quite a bit about) its procedural level generation in the past there was still plenty of things to learn in this book by its designer. Absolutely fascinating.</p> <h3>Surreal Numbers: Donald Knuth (Yes, that Donald Knuth.)</h3> <p>This is a rather strange book full of proofs about a branch of mathematics called Conway's numbers. It's also the first work of fiction to feature a new result before being published in any journal. Frankly, the love story is somewhat laughable for its lack of credibility but the ambition here is very noble: in wrapping up Number Theory derivations in a fictional narrative Knuth really, really wants to reach people that a math journal cannot. A+</p> <h3>Eyes on the Sky: Francis Graham-Smith</h3> <p>A whirlwind tour across the telescopes of the electromagnetic spectrum from Gamma-Ray bursts to Radio wavelengths and from the classic lenses of antiquity to the cutting edge of astronomy. This is probably a bit specialist but as a nerd who did an undergraduate courses on modern telescopes I loved it. A+</p> <h3>Weapons of Math Destruction: Cathy O'Neil</h3> <p>A somewhat alarmist review of the ways in which tasks formerly performed by humans are now being done by (usually quite rubbish) automation. The author is quite opinionated and makes quite a few un-substantiated claims but the basic thesis that automation tends to affect those who are poor and un-represented far more than the wealthy elite thus exacerbating wealth inequality in society is well researched and argued for. For example, when getting insurance or a bank loan if you go to an expensive, boutique service you'll probably still be seen by a human, rather than an algorithm that can make mistakes or codify the (hopefully subconscious) biases and prejudices of those who made it.</p> <h3>Effective Python</h3> <p>A good complement to 'the Hackers guide to Python' this book really helped me think more like a python-ista before starting a new role in which I get to be one.</p> <h3>The 59 Icosahedra: H.S.M. Coxeter, P. Du Val, H.T. Flather, J.F. Petrie</h3> <p>This charity shop gem is really an extended journal article exploring the maths of the Icosahedra (20-sided 3d-shapes) group members and operation, complete with a full enumeration of each many diagrams and in some cases pictures of paper models made by the great Coxeter. It's on my to-do list to hijack a 3-d printer and get it to print some of these.</p> <h3>Myth of a Strong Leader: Archie Brown</h3> <p>This was on Bill Gates winter reading-list and so far I'm loving it. It's sort of a history of politics in the 20th Century analysing the party leaders. Strangely haven't found any references to former psuedo-reality-tv-show-hosts.</p> <h1>Fiction</h1> <h3>Johnny Mnemonic: William Gibson</h3> <p>Reading this made me appreciate the 80s Keanu Reeves film, given that it was only a short story.</p> <h3>The Dispossessed : Ursula LeGuin</h3> <p>A thinly veiled and slightly exaggerated metaphor about the differences between Capitalist and Communist societies. </p> <h3>Left Hand of Darkness : Ursula LeGuin</h3> <p>Another sci-fi classic that's set on a distant world in which the human inhabitants periodically change sex and spend most of the time as an a-sexual hemaphrodite. Not only that, but they make first contact with a more technologically advanced portion of humanity.</p> <h3>Zones of Thought: Vernon Vince</h3> <ul> <li>A Fire Upon the Deep</li> <li>A Deepness in the Sky</li> <li>Children of the Sky</li> </ul> <p>This trilogy sort of snuck up on me. The first two are very reminiscent of the Culture novels by Iain M Banks (which I also loved) and thoroughly explore some very interesting ideas such as a feudal society of dog-like creatures that can only think coherently (and collectively) in small packs, and get screwed up when two packs get too physically close together. The second book I actually preferred, focusing on a race of meter-tall spiders in a just-post-industrial-revolution-society undergoing a world-scale conflict. Not only that, but they collectively hibernate several hundred years at a time and are set for first contact with a more technologically advanced humanity. I feel a strange affinity to spiders after reading this book. The third ties up the story nicely but is otherwise miss-able in my humble opinion.</p> <h3>Howls Moving Castle: Diana Wynne Jones</h3> <p>A cute kids book I read on Sarah's recommendation. Certainly richer than the (excellent) film.</p> <h3>Call of Cuthulu: H P Lovecraft</h3> <h3>The Colour of Space: H P Lovecraft</h3> <h3>Solaris: Stanislav Lem</h3> <p>A superb sci-fi great that discusses metaphysics. A lot.</p> <h3>The Dark Tower series: Stephen King</h3> <ul> <li>The Gunslinger</li> <li>The Drawing of Three</li> <li>The Wastelands</li> <li>Wizard and Glass</li> </ul> <p>What to say about this series? Clearly I can't stop reading them, but in the same way that it's difficult to stop eating junk food full of Mono-sodium-Glutemate. The writing is not superb and you have to be quite forgiving about the plot in places but I can see this making a pretty good HBO show later this year.</p>Chris BoddyMon, 09 Jan 2017 00:00:00 +0000tag:boddy.im,2017-01-09:books-2016-part-2.htmlbooksreview2016 in Books (part 1)https://boddy.im/books-2016-part-1.html<h1>Introduction</h1> <p>Who doesn't like books? </p> <p>Below is a brief review of some books I've read in the first half of 2016. </p> <p>This would probably have come out sooner but apparently I like to write book reviews during the 15-30 second crunch-time gaps between Counter-Strike (Source) rounds.</p> <h1>Non Fiction</h1> <h3>No Logo: Naomi Klein</h3> <p>Is it a manifesto and primer for the culture jammer movement? Is it a catalogue of heart-breaking corporate mis-deeds? Yes! But what's more it's a thoroughly researched (both primary research and review) of corporate behaviour across the 80s and the 90s that leaves you feeling uncomfortable and ready to boycott something. Fascinating and well written.</p> <h3>Structured Interpretation of Computer Programs: Abelman, Sussman and Sussman</h3> <p>Ever wanted to learn computer science from scratch exclusively with a purely functional language (Lisp)? This book is for you! Requiring a lot from the reader in places, very interesting.</p> <h3>Designing BSD Rootkits: Joseph Kong</h3> <p>Ever wanted to know how vulnerable c-code is to buffer-overflow attacks?</p> <h3>Building Microservices: Sam Newman</h3> <p>I think this book should be titled "Building Services" since almost all the advice applies equally to SOA but buzzwords aside, it's a pretty decent guide to the practicalities and traps of building services and SOA instead of a monolithic, n-tier application.</p> <h3>Algorithm Design Manua: Steven Skiena</h3> <p>This was actually a re-re-reading of my favourite CS book.</p> <h3>Sapiens: A Brief History of Humankind: Yuval Harari</h3> <p>Focusing on three revolutions in human history:</p> <ul> <li>cognitive ~ 70,000 years ago;</li> <li>agricultural ~ 10,000 years ago;</li> <li>scientific ~ 500 years ago.</li> </ul> <p>A compelling read with many erudite observations and a satisfyingly high density of sources. This book is worth it alone for the insight that humanity's ability to invent abstract concepts (language, religion, money, government etc.) not only allowed it to co-exist in groups of size larger than the largest chimpanzee band, but is to a large extent responsible for civilisation. To me, this is the most interesting book on this list.</p> <h3>The Hackers Guide to Python: Julien Danjou</h3> <p>After writing python for years I decided to shore up my knowledge for a particular project. This book is a cracking guide to some corners of the language that are easily missed (package distribution, scalability and ORMs to mention but a few) from a prominent OpenStack dev including interviews with others from that project.</p> <h1>Fiction</h1> <h3>Stranger in a Strange Land: Robert Heineman</h3> <p>Where to begin? A 500 page behemoth full of the sort of dialogue one would expect in a video game side quest. Not to mention the casual sexism through-out (is good-natured sexism a thing?) or the thinly veiled fantasies of the author. Still though, this book has a lot to say about sex, politics and religion (albeit via the proxy of fringe cults) and gave us the concept of "grokking". </p> <p>Long tracts of this book are sort-of dreadful but to quote a certain Martian: "I grok it is a goodness."</p> <h3>The Man in the High Castle: Philip K Dick</h3> <p>I'll admit I came to this one after exposure to advertising for the TV Show with the same title. I found this to be a somewhat voyeuristic novel in which the author explores a San Francisco that may have been, set in a parallel universe in which the USA has been conquered and colonised by Axis powers that grow increasingly antagonistic towards each other. Most of the execution of this story was pretty meh but then out of nowhere the finale was utterly sublime.</p> <h3>Purity: Jonathan Franzen</h3> <p>With all of the same qualities of his previous work "The Corrections" including an uncanny similarity to the style of Milan Kundera, Purity tells the intimate story of a few believable characters with serious issues around (among other things) trust. This one requires a more willing suspension of disbelief in places than his previous works (that I greatly enjoyed).</p> <h3>Equal Rites, Wyrd Sisters and Small Gods: Terry Pratchett</h3> <p>Terry Pratchett was brilliant. As night-time reads, Discworld novels are perfect: there are always more of them. Not only that, they have a built-in negative feedback mechanism in which after finishing one you're always ready for another, but experience tells you reading more than one in a row will be disappointing and they will all sort of blur into each other.</p> <h3>Count Zero: William Gibson</h3> <p>More focused and compelling than it's predecessor Neuromancer and set in the same universe, this book continues the dark, noir-ey sci fi of William Gibson.</p> <h3>Mona Lisa Overdrive: William Gibson</h3> <p>Concluding the tale of Neuromancer and Count Zero with more of the same pulp-ey sci-fi goodness. Being stuck in an airport lounge for most of a day gave me the great opportunity to read this almost in one sitting. </p> <h3>Wool Omnibus pt. 1 &amp; 2: Hugh Howey</h3> <p>The short story on which these are based is quite beautiful, but the next few parts of this post-apocalyptic-everyone-now-lives-in-a-silo-underground series are decidedly meh.</p> <h3>The Lady Astronauts of Mars: Mary Kowal</h3> <p>A charming, poignant little short story set in a parallel-world in which Mars was colonised shortly after the moon landings.</p>Chris BoddyTue, 14 Jun 2016 00:00:00 +0100tag:boddy.im,2016-06-14:books-2016-part-1.htmlbooksreviewA sunrise-alarm-clock-web-service using a Raspberry-Pihttps://boddy.im/sunrise-alarm-clock-web-service.html<h1>Introduction</h1> <p><a href="/images/alarm.small.jpg"> <img style="max-width:100%" src="/images/alarm.small.jpg" alt="Jar-alarm-clock"/> </a> Waking up can be tricky. I'd heard good things about alarm clocks that light up gradually over about 30 minutes mimicking the sunrise from dull red to bright yellow light. </p> <p>Many retail models are available with huge variation in price and review strength but for one with a respectable lumen count at the time of writing the price was ~ £100. What's worse, they all had garish front displays that you couldn't turn off and a very limited selection of wake-up music (or radio). </p> <p>£100 seems an lot for something that's not quite right, surely? </p> <h1>Requirements</h1> <p>So, what are we building? </p> <ul> <li>A powerful lamp with a full range of colours and variable output</li> <li>A web-interface for setting the time, light profile, etc.</li> <li>A web-server for interacting with the light</li> </ul> <h1>Hardware</h1> <h3>Host</h3> <p>There are various options for small boards running full linux distributions. I happened to have a slightly spurned but otherwise serviceable <a href="https://www.raspberrypi.org/products/model-b/">Raspberry Pi Model B</a> lying around (it was replaced as a media server some time ago by a beefier <a href="https://www.raspberrypi.org/products/raspberry-pi-2-model-b/">B+ model</a>) so I used that, but a cheaper option would be a <a href="https://www.raspberrypi.org/products/pi-zero/">Pi-zero</a> that somehow will eventually cost $5! </p> <h3>Lights</h3> <p>At first I tried using some LED arrays that were powered directly from the Pi's GPIO pins, but that limited the choice to devices like the (otherwise super cool) <a href="https://www.piborg.org/ledborg">LedBorg</a> which wasn't nearly bright enough to wake me up.</p> <p>This had two consequences:</p> <ol> <li>A more powerful led-strip was going to ~ double the budget. </li> <li>It would also require power source external to the Pi which would probably mean soldering. </li> </ol> <p>Eventually I found the <a href="http://www.amazon.co.uk/gp/product/B00EFYW028">LPD8806 led-strip</a> for £29 which like the Pi take a 5V supply, got a super cheap <a href="http://www.amazon.co.uk/gp/product/B00ZWLTI64">usb-wifi dongle</a> for £2.47 and decided to try and power both the Pi and the led-strip from the same power supply (see diagram) using a <a href="http://www.amazon.co.uk/gp/product/B005JO0F52">5V, 2A DC supply</a> power supply for £5.99 using this <a href="http://www.amazon.co.uk/gp/product/B0182WA908">converter</a> for £2.26. </p> <h3>Soldering</h3> <p>Of course, the led-strip didn't have any connector, just four bare wires. This necessitated getting my hands dirty (and mildly singed) to create a very simple circuit connecting the rasberry-pi and the led-strip in parallel with the power supply, as shown below.</p> <p><a href="/images/sunrise1.jpg"> <img style="max-width:100%" src="/images/sunrise1.jpg" alt="Soldering goodness"/> </a></p> <p><strong>Be super careful when soldering not to create any short circuits or burn yourself. If you don't know what you're doing here learn about that first!</strong></p> <h1>Software</h1> <p>I've made the project to control the led array, host the webserver, web-app and runtime management <a href="https://github.com/cboddy/sunrise-light-alarm">available here</a>. If you don't care about software you can skip over to the <a href="#install">install</a> section.</p> <h3>Web-server</h3> <p>The <a href="https://github.com/cboddy/sunrise-light-alarm/blob/master/sunrise.py">web-server</a> is a super simple <a href="https://pypi.python.org/pypi/Flask">Flask</a> service which stores the application state (ie. what time and days of the week to go off) as a json document. It also hosts the web-app and directly controls the led-array. </p> <h3>Web-app</h3> <p>The <a href="https://github.com/cboddy/sunrise-light-alarm/blob/master/public/js/app.js">web-app</a> uses <a href="https://facebook.github.io/react/">React</a> and <a href="http://getbootstrap.com/">Bootstrap</a> to show the, uh, minimal (and responsive) web-ui shown below. Clearly web design is very important to me. </p> <p><a href="/images/sunrise-webui.png"> <img style="max-width:100%" src="/images/sunrise-webui.png" alt="Web UI"/> </a></p> <h3>Runtime-management</h3> <p>Some minimal runtime-management is needed to make sure the web-server start when the raspberry-pi is powered up and restarts if it's crashed. The <a href="http://www.raspbian.org/">Raspbian</a> distribution still uses sys-v-init (since it's based on Debian-Jessie) so we use a <a href="https://github.com/cboddy/sunrise-light-alarm/blob/master/sunrise">sys-v-init script</a> to do exactly that. </p> <p>Generally sys-v-init is much fiddlier than more modern boot systems like the now deprecated Ubuntu <a href="http://upstart.ubuntu.com/">upstart</a> (which was great) or the much maligned <a href="https://en.wikipedia.org/wiki/Systemd">systemd</a> that will replace it in later debian distros but whatever, this works. </p> <p><a name="install"> </a></p> <h1>Installation</h1> <h3>Enable SPI</h3> <p>By default the <a href="https://en.wikipedia.org/wiki/Serial_Peripheral_Interface_Bus">SPI</a> bus on a raspberry-pi is disabled. Since we'll use to control the led-strip, first <a href="https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md">enable the SPI bus</a> in the raspi-config which can be opened by:</p> <pre> sudo raspi-config </pre> <h3>Project</h3> <p>To install all the project software (including dependencies) run the <a href="https://github.com/cboddy/sunrise-light-alarm/blob/master/install.sh">install script</a> passing in the user and remote host at which your raspberry-pi is available on the network. For example, I have a user "pi" and the device has the hostname "raspberrypi" so to install server etc. I would do</p> <pre> > source install.sh pi@raspberrypi </pre> <p>If you now browse to <a href="http://raspberrypi">http://raspberrypi</a> on that same network you should now see the web-ui show above.</p> <p><em>Disclaimer: in a more formal setting I'd be using <a href="http://www.ansible.com/">Ansible</a> for this sort of thing but somehow it didn't feel right given how janky the rest of the project is ;-)</em></p> <h1>Result</h1> <p>In case you think I'm making all this up here it is.</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/9P10_nMiUQY" frameborder="0" allowfullscreen></iframe> <h1>Conclusion</h1> <p>I really enjoyed this Christmas project and would highly recommend it to anyone who feels like doing some hardware for a change. The total hardware cost was under £65, but using a pi-zero would reduce that by ~ £20 and now I have a sweet alarm-clock-in-a-jar!</p> <p>I also believe it would serve as a good starting point for building something bigger with more lights with cooler animations since it hooks up a web-ui to the leds directly. </p>Chris BoddySat, 09 Jan 2016 00:00:00 +0000tag:boddy.im,2016-01-09:sunrise-alarm-clock-web-service.htmlelectronicsraspberrypireactflaskA JVM client for IPFShttps://boddy.im/ipfs-scala.html<h1>Introduction</h1> <p>The <a href="https://ipfs.io/">inter-planetary file-system (IPFS)</a> is a super exciting open-source project that, among other things, fixes several problems with current data storage protocols combining ideas used in version control systems, and decentralised peer-to-peer protocols like bit-torrent. The primary IPFS implementation efforts are (so far) in Go and JavaScript and the rapid development pace aside, the IPFS HTTP client API is stable enough to be used by projects like <a href="https://peergos.org/">Peergos</a>.</p> <p>However, using IPFS from the JVM would be immensely easier with a JVM client that wraps the IPFS REST API...</p> <h1>scala-ipfs-api</h1> <p><a href="https://github.com/ipfs/scala-ipfs-api">And lo, there is now an IPFS client for the JVM</a>. </p> <p>Although the client's written in Scala, it's written with Java projects in mind requiring a <a href="https://github.com/ipfs/scala-ipfs-api/blob/master/ipfs.api.jar">single jar</a> in your Java project class-path (made possible with the <a href="https://github.com/sbt/sbt-assembly">sbt-assembly</a> plugin).</p> <p>Writing this client has certainly improved my understanding of the project, and using IPFS as a storage engine for Peergos is now a giant step closer. </p>Chris BoddySun, 27 Sep 2015 00:00:00 +0100tag:boddy.im,2015-09-27:ipfs-scala.htmlipfsscalajvmRemote file-system browser web-apphttps://boddy.im/file-browser-web-app.html<h1>Introduction</h1> <p>Work with Ian on <a href="https://github.com/ianopolous/Peergos">Peergos</a> is progressing well, to the point where our distributed-secure-file-system finally needs a remote file-system browser (like Explorer, Finder etc.) and despite my standing revulsion for all things Javascript, I plunged in.</p> <h1>Requirements</h1> <p>Specifically, we required a web-app file-system browser which could support file-upload &amp; download, deletion, sharing and so on. I immediately wanted to use <a href="http://getbootstrap.com/">Bootstrap</a> to make the design responsive (look different on different sized screens) but there was still one problem...</p> <p>Let's face it: web UIs are tricky. I attribute most of this to the weak-typing and general difficulty to create invariants in Javascript. For example, what do you think this expresson:</p> <blockquote> <p>["10", "10", "10"].map(parseInt)</p> </blockquote> <p>evaluates to? Props if you were expecting this:</p> <blockquote> <p>[10, NaN, 2]</p> </blockquote> <p>But I digress. For this reason, I decided to try something new: Facebook's <a href="https://facebook.github.io/react/">React</a> Javascript UI library. At first I was skeptical but after four hours I had a working prototype and after two working days (integrated over a week) the task was feature complete. Essentially within React's model you tend to write more modular, composable JS code... and that's something the world desperately needs.</p> <h1>A web-app file-system browser</h1> <p><a href="https://github.com/cboddy/react-filesystem-browser">The project is open-source and available on Github</a>. To simulate the backend filesystem I created a simple <a href="http://flask.pocoo.org/">Flask</a> server that supports filesystem operations in a chroot environment with cookie-based session management and password-based authentication (specified as a server command-line parameter) but bear in mind this is little more than a mock for the web-ui, albeit a functional one supporting:</p> <ul> <li>Grid and list views; </li> <li>File upload and download; </li> <li>Context menu (right / long click) with rename and delete options; </li> <li>Navigation buttons (back, up etc.);</li> <li>Sorting on file attributes (size, name, last-modified time); </li> <li>Directory creation;</li> <li>Cookie-based session management and simple password authentication;</li> </ul> <p>A few screen-grabs on a large / small screen are shown below.</p> <h3>Desktop grid-based view</h3> <p><a href="/images/grid_view1.png"> <img style="max-width:100%" src="/images/grid_view1.png" alt="Desktop grid-view of my local / dir."/> </a></p> <h3>Desktop list-based view</h3> <p><a href="/images/list_view1.png"> <img style="max-width:100%" src="/images/list_view1.png" alt="Desktop list-view of my local / dir."/> </a></p> <h3>Mobile grid-based view</h3> <p><a href="/images/grid_view2.png"> <img style="max-width:100%" src="/images/grid_view2.png" alt="Mobile grid-view ."/> </a></p> <h3>Mobile list-based view</h3> <p><a href="/images/list_view2.png"> <img style="max-width:100%" src="/images/list_view2.png" alt="Mobile list-view."/> </a></p> <h1>Conclusion</h1> <p>This web-app is now feature complete and destined to be the Peergos front-end but please feel free to fork it or submit a pull request. Also a disclaimer, although there is password authentication it is still pretty far from being "secure" without proxying it through a local HTTPS server and auditing the Flask server security. </p>Chris BoddySat, 23 May 2015 00:00:00 +0100tag:boddy.im,2015-05-23:file-browser-web-app.htmlpeergosfile-browserreactbootstrapRSS Repo: Yet another RSS aggregatorhttps://boddy.im/rss-repo-live.html<h1>Introduction</h1> <p>I have a confession to make: I still use <a href="http://en.wikipedia.org/wiki/RSS">RSS</a> to navigate the web... even now! </p> <p>Years after the fabulous RSS aggregator service <a href="http://www.google.com/reader/about/">Google Reader</a> went the way of the HAL 9000 there are still those that feel its absence, myself included. Of course, many alternative services and browser plugins tried to fill the gap but frankly, I've never found one that was quite right. Some modest requirements:</p> <ul> <li>No advertisements of any kind; </li> <li>A simple way to import / export my data (feed-list) in standard <a href="http://en.wikipedia.org/wiki/OPML">OPML</a> format;</li> <li>No third party cookies; </li> <li>No default, ever-present feeds; </li> <li>A secure connection;</li> <li>A single-button to fetch updates from all of my feeds;</li> <li>Render HTML article content in a cross-platform (including mobile) client;</li> <li>And of course, it should be open-source :-)</li> </ul> <p>That sounds pretty easy, right? Well with that in mind I started building <a href="http://rss.boddy.im">RSS Repo</a> (<a href="https://rss.boddy.im">HTTPS</a>). After a few weekends it is now fairly usable, and so I thought I'd write a post about it and see if anyone else is interested... so feel free to sign-up!</p> <h1>Architecture</h1> <p><a href="http://rss.boddy.im">RSS Repo</a> is a single-page application written in regular HTML and Javascript, using the <a href="http://getbootstrap.com/">Bootstrap</a> CSS-library to render in a mobile-friendly way. On the other side is a pretty simple <a href="http://flask.pocoo.org/">Flask</a> server backed by SQL. </p> <p>In this model, the server is fetching / storing the RSS updates and the browser is rendering the article-HTML directly in the single-page application.</p> <p>Originally I thought this would be a web-view sitting inside a mobile app built with something like <a href="http://cordova.apache.org/">Cordova</a> so you could read cached articles without a web-connection but I think actually browser local storage or <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">IndexedDB</a> would be easier of the mobile browser coverage were good enough. </p> <p>For now the HTTPS certificate is self-signed since I didn't want to shell out for one from a certificate authority. Of course, once the EFF initiative <a href="https://www.eff.org/deeplinks/2014/11/certificate-authority-encrypt-entire-web">Let's encrypt</a> is available I'll switch to that.</p> <h1>What it looks like</h1> <p>The simple UI is shown in the screen-grabs below; in one table sit the added / imported RSS feeds with a count of the unread articles, and in the other a list of articles (those in bold are currently unread). </p> <p><a href="/images/rss_repo_with_menu.png"> <img style="max-width:100%" src="/images/rss_repo_with_menu.png" alt="Desktop shot"/> </a></p> <p><a href="/images/rss_repo_wide.png"> <img style="max-width:100%" src="/images/rss_repo_wide.png" alt="Another Desktop shot"/> </a></p> <p>And here is what a mobile browser view looks like (although this is actually a small desktop Firefox window).</p> <p><a href="/images/rss_repo_small.png"> <p align="center"><img style="max-width:100%" src="/images/rss_repo_small.png" alt="Mobile shot"/></p> </a></p> <h1>Conclusion</h1> <p>Well, that's it: the site is live and open to <a href="http://rss.boddy.im/">signing-up</a> so please let me know if you have any suggestions. The project isn't yet on Github but the plan is to do so and use parts of the site as tutorials of stuff like user sessions, automated emails for sign up and best practices for password storing... watch this space.</p>Chris BoddySun, 12 Apr 2015 00:00:00 +0100tag:boddy.im,2015-04-12:rss-repo-live.htmlrss-aggregationgoogle-readerrss-repoPorting the NaCl cryptography libraryhttps://boddy.im/jvm-tweet-nacl.html<h1>Introduction</h1> <p><a href="http://nacl.cr.yp.to/">NaCl</a> is one of the most widely used cryptography libraries currently available. <a href="http://tweetnacl.cr.yp.to/">Tweet-NaCl</a> distills it into an entirely understandable, auditable and portable 100-tweets worth of C code (no really: it is even on <a href="https://twitter.com/tweetnacl">Twitter</a>). </p> <p>As part of <a href="https://github.com/ianopolous/Peergos">Peergos</a>, Ian and I really needed a set of crypto primitives that could be used in both a Javascript environment (ie. a browser) and in the JVM. Tweet-NaCl was already available in <a href="https://github.com/dchest/tweetnacl-js">Javascript</a> but we could not find a stable, working version written in pure Java...</p> <h1>Pure Java Tweet-NaCl</h1> <p><a href="https://github.com/ianopolous/tweetnacl-java">Java Tweet-NaCl</a> is a pure Java port of the Tweet-NaCl cryptography library. The actual port was pretty easy except when it briefly turned into a regex-olympics competition to turn as much of the C to Java automatically. </p> <p>To verify the correctness of our pure-Java port, we wrapped the pure-C Tweet-NaCl library in the Java Native Interface (JNI). This allowed us to directly test the various Tweet-NaCl API calls in the C-source and our Java port. </p> <p>To verify that our port is compatible with the Javascript port we're using in Peergos, we also tested the entire API running the Javascript version in <a href="http://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/">Nashorn</a>.</p> <p>So there you go: three implementations of the Tweet-NaCl library running in the same JVM: </p> <ul> <li>The original C running in the JNI;</li> <li>A Javascript port of the original C running in Nashorn;</li> <li>Our new pure-Java port running directly.</li> </ul> <p>And best of all, all tests are now passing!</p>Chris BoddyMon, 06 Apr 2015 00:00:00 +0100tag:boddy.im,2015-04-06:jvm-tweet-nacl.htmljavajvmjnitweet-naclcryptographyProfiling the Raspberry PI network I/O rate on the JVMhttps://boddy.im/rpi-network-test.html<h1>Introduction</h1> <p>It is a fact universally ackowledged that the <a href="http://www.raspberrypi.org/">Raspberry PI</a> is a thing of beauty. I recently stumbled across some performance test data I collected earlier this year concerning the network performance of the little things, in particular when using the (Oracle) Java Virtual Machine (JVM) versus native.</p> <h1>Context</h1> <p>At the time we were building a distributed NoSQL database built on the JVM and the <a href="http://www.raspberrypi.org/archives/4920">Rasbian folks had recently added the Oracle JVM to the Raspian repositories</a>. To demonstrate how a cluster of machines running the database would fare, we needed a <strong>cluster</strong>. We used AWS to build a cluster in their Ireland availability zone but I thought it would be much more fun to demonstrate a physical cluster and designed a 64-node Raspberry PI cluster demo that we could show off with and would be reasonably portable. I was convincing enough to secure a few Raspberry PIs to test but the company didn't survive long enough to publish anything about either the database or build the cluster. </p> <p>In theory, the network card on a Raspberry PI model B should be able to deliver a bandwidth of about about 100 Mbits per second. This would certainly handle many database transactions per second (or a bunch of web traffic or make for a decent media server), but sometimes there is a semantic gap between what is advertised and the reality. I'd also heard various horror stories regarding former JVM network-stack implementations. The question I wanted to know was how good the Oracle JVM would be at performing network I/O operations on a Raspberry PI. </p> <h1>Setup</h1> <p>Specifically I wanted to benchmark the network I/O performance between two Raspberry PI units and also between one Raspberry PI and a conventional server over TCP. </p> <p>Our inventory for the test is as follows:</p> <ul> <li>Two Raspberry PI model B units;</li> <li>A beefy server with a 1 Gbps network card;</li> <li>A switch capable of delivering 1 Gbps uncontended between two endpoints.</li> </ul> <p align="center" ><img src="/images/rpi-config.png" alt="setup"/></p> <p>To measure the native network bandwidth (as seen by an application), I used a free-and-open-source benchmarking tool called <a href="http://iperf.fr/">iPerf</a>:</p> <p>For the JVM I used a simple Java I/O test-harness (available on my <a href="https://github.com/cboddy/iotester">Github</a>) that I had previously built to profile the network and disk performance of various cloud providers (Amazon, Rackspace etc.).</p> <h1>Results</h1> <p>The transfer rate over a few minutes for each of the four scenarios is shown in the plot. </p> <p align="center" ><img src="/images/network-rate.jpg" alt="setup" width="640"/></p> <p>The key features I see reading this plot are:</p> <ul> <li> <p>From RPI to the server the transfer rate is stable using both the iPerf and my JVM code and pretty close to the advertised maximum of 100 Mbps.</p> </li> <li> <p>There is a small but non-neglibible overhead using the JVM, compared to native. This was expected but it's also of the order 1% which is pretty good, props to Oracle. I didn't try too hard to optimise the JVM tests but if you see something that can be improved let me know.</p> </li> <li> <p>The transfer rate is less stable when the RPI is <strong>receiving</strong> the traffic, particularly on the JVM.</p> </li> <li> <p>The RPI can received a good deal less than the Network card is specced for. I tried fiddling various buffer sizes to improve the iPerf results but they seemed fairly stable at 64KB and similarly with my JVM code. </p> </li> </ul> <h1>Conclusion</h1> <p>None of these results are too surprising but I was satisfied that they would do the job.</p>Chris BoddySat, 06 Sep 2014 00:00:00 +0100tag:boddy.im,2014-09-06:rpi-network-test.htmlraspberry-pijavajvmnetwork profiling