Binary World Excitements Philosophized Private Uncategorized


I would have commented on YouTube but, well. No account, no comment. Hence, I decided to share the video here. One of the many black holes of the Internet.

Really, really inspiring. Will help decide young people what to do. Or what not to do. Either way, helps keeping the thoughts going crazy :) Thanks!

Binary World Work

MovieOS Security: 5 lessons IT can learn from ‘Rogue One: A Star Wars Story’‑lessons-it-can-learn-from-rogue-one-a-star-wars-story-1702.html

Binary World Work


It’s Christmas in the office.

Binary World Code Work

First Steps with Boost::Spirit

Imagine there were some expres­sion pseudo language you need to convert into JavaS­cript or some other, pro­pri­et­ary language. That was the task at hand some months ago for which I wrote a Regular Expres­sions-based parser. At first it sounded like a good idea: no nested function calls and all quite simple boolean expres­sions. But then there came nested function calls the need for more elaborate trans­form­a­tions than VARIABLE=5 to VARIABLE===5.

This is when you realise that a parser cannot really be avoided any more. But what if Lexx and Yacc are not your favourite toolset? And ANTLR does not have the best possible support coverage for C++ (but still is a very convenien parser generator)? Ah right, we’re talking C++, so that’s the baseline here.

After a bit of digging and Duck­DuckGo’ing, I found Boost Spirit. Yes, there’s a parser generator in Boost, and even a header-only imple­ment­a­tion! After well more than 10 years using Boost, I was quite surprised. Boost Spirit makes it easy to create attribute grammars and even gen­er­at­ors. So perfect for the task at hand!

What’s the task again? Well, it’s the following: create a trans­form­a­tion from an expres­sion pseudo language to JavaS­cript. The language looks a bit like the following:

FIELD5 IN SomeSetOfValues AND NUMFIELD5 IN ( 1; 2 )
NUMFIELD6 IN ( 2 ) AND NUMFIELD7 NONEIN ( 88, 89, 90, 91, 92, 93 ) AND NUMFIELD8 = 1
someFunction( calculateSomethingElse( FIELD9 ), FIELD10 ) > someOtherFunction( FIELD11 )

Quite weird, right? Also note the different list sep­ar­at­ors and some strange checks for the empty set. But that’s not what the language parser needs to take of. Well, even though the language is quite simple and easy to read, it’s at least as difficult as a math­em­at­ic­al expres­sion parser. But that’s basically it.

So let’s begin. I’ll outline the main learnings and what is needed for a Boost Spirit-based grammar. In a second article, I might also try to introduce the generator part. But let’s for now focus on the gen­er­a­tion of an abstract syntax tree (AST) which can be post­pro­cessed if required.

I’ve decided to split the grammar into the following main parts:

  • The Expres­sion rule starts with an addition term.
  • Term rules are math­em­at­ic­al terms made up of the four main oper­a­tions (add, sub, mul, div), e.g. NUMFIELD10 * NUMFIELD10.
  • Equation rules combine math­em­at­ic­al equations (lt, le, gt, ge, eq), set oper­a­tions, and negation, e.g. NUMFIELD7 NONEIN ( 88, 89, 90, 91, 92, 93 ).
  • Rule rules are basically boolean expres­sions (and, or), e.g. ( FIELD1 = EMPTY OR FIELD1 <> 1 ).

Then we also have the basic atoms like variable names, function names, numbers, lists, strings, dates, and other atoms. I cannot post the complete imple­ment­a­tion here but I try to walk you through the basics to get started. Boost Spirit exploits C++ operator over­load­ing to a great extend and so creates it’s own domain-specific language (DSL) which emulates the EBNF form. Once you got used to how it behaves, it’s really straight­for­ward to write even complex (attribute) grammars.

So now, it’s get started. What’s required? We’ll include a bunch of headers which will add the func­tion­al­ity we need:

#include <boost/config/warning_disable.hpp>

#include <boost/spirit/include/qi.hpp>

#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>

#include <boost/fusion/include/adapt_struct.hpp>

#include <boost/variant/recursive_variant.hpp>

#include <array>
#include <vector>

The boost::phoenix headers are required for attribute binding. Boost Fusion makes C/C++ struc­tures first class citizens in Boost Spirit.

The first couple of defines cover the atoms in our language.

using qi::int_;
using qi::uint_parser;
using qi::lit;
using qi::lexeme;

using namespace qi::labels;

using ascii::space;
using ascii::char_;
using ascii::digit;

using phoenix::at_c;

qi::rule< Iterator, std::string(), ascii::space_type > SIGN;
qi::rule< Iterator, variable(), ascii::space_type > VARIABLE;
qi::rule< Iterator, std::string(), ascii::space_type > NAME;
qi::rule< Iterator, std::string(), ascii::space_type > NUMBER;
qi::rule< Iterator, std::string(), ascii::space_type > LIST_ELEMENT;
qi::rule< Iterator, std::string(), ascii::space_type > STRING;

SIGN %= -( lit('+') | lit('-') ); // Number sign
VARIABLE = lexeme[ char_("a-zA-Z_") [at_c<0>(_val) = _1] >> *( char_("a-zA-Z_0-9") [at_c<0>(_val) += _1] ) ]; // Typical variable names [a-zA-Z_][a-zA-Z_0-9]*
NAME %= lexeme[ char_("a-zA-Z_") >> *( char_("a-zA-Z_0-9") ) ];
NUMBER %= -SIGN >> lexeme[ +digit >> -( char_(',')[_val += '.'] >> +digit ) ]; // Numbers with German decimal point (comma)
LIST_ELEMENT %= lexeme[ +(char_ - '(' - ')' - ';' - ',')];
STRING = lit('\'') >> lexeme[ +(char_ - '\'') ] >> lit('\'');

The lexeme parts make sure that there is no separator involved. The strange [at_c] attrib­utes are where the captures are assigned to the AST data struc­tures. In order to mark the variable, for example, as a special entity, I’ve decided to provide a dedicated structure for it:

nsmespace myns {
  struct variable { std::string name; };

Simple but efficient. In the post­pro­cessing and gen­er­a­tion steps this will be handled dif­fer­ently than other names such as function names. But we cannot just use that structure in boost::spirit just like that. Currently, there’s real no notion of “first element” or “n‑th element” of the structure, This is, however used by the odd at_c<n>(...) func­tion­al­ity: it binds the output attribute of the respect­ive capture to the given field. The field may either be a variable (syntax then reads like ref(v) where v is the variable), a member variable of a structure, or even a member function (the at_c<n>(...) syntax). One could also bind functions or lambdas. We really do have some kind of flex­ib­il­ity here!

The char_ represent a character, digit a digit. The under­score was intro­duced purely because of name clashes with the basic types.

But what is required for the structure variable to be usable in boost::spirit? Not more than the following piece of code defined in the *global* namespace:

  (std::string, name)

It’s basically the same for all the other struc­tures you’ll need. If you have a vector of strings (i.e. std::vector< std::string >) named values you’ll just need to define (std::vector< myns::expression_value >, values). If you want to define a recursive structure using boost::recursive_wrapper or a variant using boost::variant, you just do the same:

nsmespace myns
  struct rule;
  boost::variant< boost::recursive_wrapper< rule >, equation_value > rule_value;
  struct rule {
    rule() : operation(-1) {}
    int operation;
    std::vector< rule_value > equations;

  (int, operation)
  (std::vector< myns::rule_value >, equations)

And how do the remaining rules look like? Basically just like the atom defin­i­tions above. I’ve decided to split up the ADD/SUB and MUL/DIV rules as my AST forms an non-binary expres­sion tree (i.e. for one operation, more than two elements are listed like, e.g. +(a b c)) and wrapping ADD and SUB grammars in one rule would not work here. We need to generate a new node the the AST for each operation.

my_expression = my_term_add_sub [push_back(at_c<1>(_val), _1)];

my_term_add_sub %= my_term_add | my_term_sub | my_term_mul_div;
my_term_add = // a+b, ...
my_term_mul_div [push_back(at_c<1>(_val), _1)]
  >> +(
    lit('+') [at_c<0>(_val) = 0]
    >> (
      my_term_sub [push_back(at_c<1>(_val), _1)] |
      my_term_mul_div [push_back(at_c<1>(_val), _1)]
my_term_sub = // The same as for add but reverse (don't call my_term_sub but my_term_add)

my_term_mul_div %= my_term_mul | my_term_div | my_term;
my_term_mul = // a*b, ...
my_term [push_back(at_c<1>(_val), _1)]
  >> +(
    lit('*') [at_c<0>(_val) = 2]
    >> (
      my_term_div [push_back(at_c<1>(_val), _1)] |
      my_term [push_back(at_c<1>(_val), _1)]
my_term_div = // The same as for add but reverse (don't call my_term_div but my_term_mul)

my_term_set =
  lit("LEER") |
  lit('(') >> lit("LEER") >> lit(')') |
    >> LIST_ELEMENT [push_back(_val, _1)] % char_(",;")
    >> lit(')')
my_term_sub_expression =
  >> my_expression [_val = _1]
  >> lit(')');
my_term_function =
  NAME [at_c<0>(_val) = _1]
  >> lit('(')
  >> my_equation [push_back(at_c<1>(_val), _1)] % char_(",;")
  >> lit(')');
my_term %=
  my_term_function |
  my_term_sub_expression |
  my_term_set |

The AST expres­sion tree is modelled as follows:

namespace myns
  struct equation;
  typedef boost::recursive_wrapper< equation > equation_value;
  struct func {
    std::string name;
    std::vector< equation_value > arguments;

  struct expression;
  typedef boost::variant< boost::recursive_wrapper< expression >,
    std::vector< std::string >,
    std::string > expression_value;

  struct expression {
    expression() : operation(-1) {}
    int operation;
    std::vector< expression_value > values;

  struct equation {
    equation() : negate(false), operation(-1) {}
    bool negate;
    expression_value left;
    int operation;
    expression_value right;

  (std::string, name)
  (std::vector< qslang::equation_value >, arguments)
  (int, operation)
  (std::vector< qslang::expression_value >, values)
  (bool, negate)
  (qslang::expression_value, left)
  (int, operation)
  (qslang::expression_value, right)

I’ll leave this uncom­men­ted. It’s basically EBNF with attrib­utes and the data struc­tures should be fairly self-explan­at­ory (haha; I’m just lazy, agreed). Sadly, I cannot post the complete source and maybe it’s not even the best way to model the grammar. But it should help inter­ested coders getting the idea.

Binary World

Let’s Encrypt continued

The first cer­ti­fic­ate I’ve ordered during the closed beta phase has been replaced now. HPKP is enabled again. With the standard Let’s Encrypt (LE) procedure, a new private key is generated every time a new cer­ti­fic­ate is issues.

In order be able to enable HPKP again, you either need to use the standard LE client with a pre­vi­ously created CSR or use some other client.

With the LE client, that would mean: letsencrypt-auto certonly -a manual --csr {csr} as outlined in an article by Thomas Leister (in German).

One very nice altern­at­ive client is acme-tiny. I’ve decided to follow that route. It’s quite nicely outlined in a post by Scott Helme.

I still haven’t estab­lished a proper key turnover procedure for standard TLS cer­ti­fic­ates and for DNSSEC (where the key regis­tra­tion with the TLD is the most chal­len­ging and time-intensive task, depending on your provider). That’s some project for rainy, boring

Binary World

Let’s Encrypt

The first pro­duc­tion Let’s Encrypt cer­ti­fic­ate is issued and installed on HTTP/2 is also enabled, thanks to nghttpx. Works like a charm :)

I’ll try to post some config settings for nghttpx and also Let’s Encrypt, but that’s a really straight forward task (if no servers are listening to ports 80 and 443).

Binary World dmarc Private


There are quite a few services available (even for free to some degree) that analyse the DMARC [1, 2] domain and failure reports for a domain: DMARC analyzer and dmarcian are the most prominent examples, afaik.

I’ve also played with these services but… I just wanted to have a high-level view on what’s happening out there. Any maybe also do the analysis myself. This is why I checked out rddmarc and played with it a bit. The result (which is a very basic web interface) is now publicly available here: dmarc-stats.

A demo is also available.

Feel free to fork and improve :)

Binary World Private Work


The single-eyed that con­stantly reminds me of the obvious at work. In general, I mean :)

No comment
No comment
Binary World Excitements Private

Carpe Notcem

It’s been quite a while since I really wrote blog posts other than “oh, look, a thingy!”. But maybe I somehow manage to reanimate my blogging habits. I’ll try to start with a small one.

Some years ago, back in 2005, the Carpe Noctem Robotic Soccer team was estab­lished at the Uni­ver­sity of Kassel’s Dis­trib­uted Systems Group. My close friend Roland and I, both quite long-term Rob­oCup­pers who joined the global ini­ti­at­ive in 2001, founded the team to, on the one hand have a starting point for our research. And true, on the other hand it’s just great fun to build and program robots and to attend real com­pet­i­tions :) Not only for us but also for (pro­spect­ive) students.

Today, most of our former students got PhD students con­duct­ing their own research. Some took over the team lead and continued what we started back then: offering a platform for under­grads and graduate students where they can play, research, and sharpen their profile. To be honest I’m quite a bit proud of what we’ve achieved :)

For such a project, press coverage always is something that may attract external sup­port­ers or sponsors. In a sense, it also allows the public to take part in the devel­op­ment and progress. This is why I want to share two press releases of the current progress of the team, even though I’m not involved at all in the devel­op­ment any more. The first one is a newspaper article, the second one a YouTube video:

Great done guys and good luck in the Neth­er­lands! :)

Binary World Excitements Private

Firefox Colour Management

Yesterday, I finally uploaded some pictures to my gallery again. After quite some time (the jf Smugmug plugin is not that fast when talking about uploading; at least compared to the Smugmug native upload via the web browser as it seems to be single-threaded), I browsed through the gallery in order to check the quality. I really was shocked with what I saw: bleeding colours and far too extreme colour contrasts, the sat­ur­a­tion of some pictures seemingly was far too high.

As I have a (more or less) quite typical monitor con­fig­ur­a­tion (i.e. a laptop and a bigger screen for working with pictures and other colour-sensitive stuff), I dragged the browser window to the laptop’s screen to check the colours on another screen. And guess what: the colours looked great again. Ok, admit­tedly “great” should read “better” here as the laptop’s screen’s quality is far from great (Lenovo X200s). When looking back, I remember some strange redraw behaviour when dis­play­ing the pictures in Firefox, exactly for the areas in the image where colours went crazy. That was a hint I silently ignored.

What could be the reason? Linux’ colour man­age­ment? No, all right in that respect. Windows’s (running in a VM) colour managent? No, also no problem. I also could have wiped the ICC inform­a­tion from my pictures but that would not have been a solution but a tweak. To make a long story short: this is when I firs dis­covered Firefox’ colour man­age­ment cap­ab­il­it­ies [moz­illazine, gballard]. After disabling all that stuff, my pictures’ colours returned to normal again. Also not the best solution but the only one that works. Other browsers (such as Chromium) do not have colour man­age­ment built-in anyway.

To disable the colour man­age­ment in Firefox, simply go to about:config and set:


Maybe it doesn’t behave sane because of my monitor con­fig­ur­a­tion? Don’t really known. Maybe I’ll invest some more time to dig into that phe­nomen­on. Regard­less of that, I really should colour calibrate my monitors :)