Binary World Work

MovieOS Security: 5 lessons IT can learn from ‘Rogue One: A Star Wars Story’‑lessons-it-can-learn-from-rogue-one-a-star-wars-story-1702.html

Binary World Work


It’s Christmas in the office.

Binary World Code Work

First Steps with Boost::Spirit

Imagine there were some expres­sion pseudo language you need to convert into JavaS­cript or some other, pro­pri­et­ary language. That was the task at hand some months ago for which I wrote a Regular Expres­sions-based parser. At first it sounded like a good idea: no nested function calls and all quite simple boolean expres­sions. But then there came nested function calls the need for more elaborate trans­form­a­tions than VARIABLE=5 to VARIABLE===5.

This is when you realise that a parser cannot really be avoided any more. But what if Lexx and Yacc are not your favourite toolset? And ANTLR does not have the best possible support coverage for C++ (but still is a very convenien parser generator)? Ah right, we’re talking C++, so that’s the baseline here.

After a bit of digging and Duck­DuckGo’ing, I found Boost Spirit. Yes, there’s a parser generator in Boost, and even a header-only imple­ment­a­tion! After well more than 10 years using Boost, I was quite surprised. Boost Spirit makes it easy to create attribute grammars and even gen­er­at­ors. So perfect for the task at hand!

What’s the task again? Well, it’s the following: create a trans­form­a­tion from an expres­sion pseudo language to JavaS­cript. The language looks a bit like the following:

FIELD5 IN SomeSetOfValues AND NUMFIELD5 IN ( 1; 2 )
NUMFIELD6 IN ( 2 ) AND NUMFIELD7 NONEIN ( 88, 89, 90, 91, 92, 93 ) AND NUMFIELD8 = 1
someFunction( calculateSomethingElse( FIELD9 ), FIELD10 ) > someOtherFunction( FIELD11 )

Quite weird, right? Also note the different list sep­ar­at­ors and some strange checks for the empty set. But that’s not what the language parser needs to take of. Well, even though the language is quite simple and easy to read, it’s at least as difficult as a math­em­at­ic­al expres­sion parser. But that’s basically it.

So let’s begin. I’ll outline the main learnings and what is needed for a Boost Spirit-based grammar. In a second article, I might also try to introduce the generator part. But let’s for now focus on the gen­er­a­tion of an abstract syntax tree (AST) which can be post­pro­cessed if required.

I’ve decided to split the grammar into the following main parts:

  • The Expres­sion rule starts with an addition term.
  • Term rules are math­em­at­ic­al terms made up of the four main oper­a­tions (add, sub, mul, div), e.g. NUMFIELD10 * NUMFIELD10.
  • Equation rules combine math­em­at­ic­al equations (lt, le, gt, ge, eq), set oper­a­tions, and negation, e.g. NUMFIELD7 NONEIN ( 88, 89, 90, 91, 92, 93 ).
  • Rule rules are basically boolean expres­sions (and, or), e.g. ( FIELD1 = EMPTY OR FIELD1 <> 1 ).

Then we also have the basic atoms like variable names, function names, numbers, lists, strings, dates, and other atoms. I cannot post the complete imple­ment­a­tion here but I try to walk you through the basics to get started. Boost Spirit exploits C++ operator over­load­ing to a great extend and so creates it’s own domain-specific language (DSL) which emulates the EBNF form. Once you got used to how it behaves, it’s really straight­for­ward to write even complex (attribute) grammars.

So now, it’s get started. What’s required? We’ll include a bunch of headers which will add the func­tion­al­ity we need:

#include <boost/config/warning_disable.hpp>

#include <boost/spirit/include/qi.hpp>

#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>

#include <boost/fusion/include/adapt_struct.hpp>

#include <boost/variant/recursive_variant.hpp>

#include <array>
#include <vector>

The boost::phoenix headers are required for attribute binding. Boost Fusion makes C/C++ struc­tures first class citizens in Boost Spirit.

The first couple of defines cover the atoms in our language.

using qi::int_;
using qi::uint_parser;
using qi::lit;
using qi::lexeme;

using namespace qi::labels;

using ascii::space;
using ascii::char_;
using ascii::digit;

using phoenix::at_c;

qi::rule< Iterator, std::string(), ascii::space_type > SIGN;
qi::rule< Iterator, variable(), ascii::space_type > VARIABLE;
qi::rule< Iterator, std::string(), ascii::space_type > NAME;
qi::rule< Iterator, std::string(), ascii::space_type > NUMBER;
qi::rule< Iterator, std::string(), ascii::space_type > LIST_ELEMENT;
qi::rule< Iterator, std::string(), ascii::space_type > STRING;

SIGN %= -( lit('+') | lit('-') ); // Number sign
VARIABLE = lexeme[ char_("a-zA-Z_") [at_c<0>(_val) = _1] >> *( char_("a-zA-Z_0-9") [at_c<0>(_val) += _1] ) ]; // Typical variable names [a-zA-Z_][a-zA-Z_0-9]*
NAME %= lexeme[ char_("a-zA-Z_") >> *( char_("a-zA-Z_0-9") ) ];
NUMBER %= -SIGN >> lexeme[ +digit >> -( char_(',')[_val += '.'] >> +digit ) ]; // Numbers with German decimal point (comma)
LIST_ELEMENT %= lexeme[ +(char_ - '(' - ')' - ';' - ',')];
STRING = lit('\'') >> lexeme[ +(char_ - '\'') ] >> lit('\'');

The lexeme parts make sure that there is no separator involved. The strange [at_c] attrib­utes are where the captures are assigned to the AST data struc­tures. In order to mark the variable, for example, as a special entity, I’ve decided to provide a dedicated structure for it:

nsmespace myns {
  struct variable { std::string name; };

Simple but efficient. In the post­pro­cessing and gen­er­a­tion steps this will be handled dif­fer­ently than other names such as function names. But we cannot just use that structure in boost::spirit just like that. Currently, there’s real no notion of “first element” or “n‑th element” of the structure, This is, however used by the odd at_c<n>(...) func­tion­al­ity: it binds the output attribute of the respect­ive capture to the given field. The field may either be a variable (syntax then reads like ref(v) where v is the variable), a member variable of a structure, or even a member function (the at_c<n>(...) syntax). One could also bind functions or lambdas. We really do have some kind of flex­ib­il­ity here!

The char_ represent a character, digit a digit. The under­score was intro­duced purely because of name clashes with the basic types.

But what is required for the structure variable to be usable in boost::spirit? Not more than the following piece of code defined in the *global* namespace:

  (std::string, name)

It’s basically the same for all the other struc­tures you’ll need. If you have a vector of strings (i.e. std::vector< std::string >) named values you’ll just need to define (std::vector< myns::expression_value >, values). If you want to define a recursive structure using boost::recursive_wrapper or a variant using boost::variant, you just do the same:

nsmespace myns
  struct rule;
  boost::variant< boost::recursive_wrapper< rule >, equation_value > rule_value;
  struct rule {
    rule() : operation(-1) {}
    int operation;
    std::vector< rule_value > equations;

  (int, operation)
  (std::vector< myns::rule_value >, equations)

And how do the remaining rules look like? Basically just like the atom defin­i­tions above. I’ve decided to split up the ADD/SUB and MUL/DIV rules as my AST forms an non-binary expres­sion tree (i.e. for one operation, more than two elements are listed like, e.g. +(a b c)) and wrapping ADD and SUB grammars in one rule would not work here. We need to generate a new node the the AST for each operation.

my_expression = my_term_add_sub [push_back(at_c<1>(_val), _1)];

my_term_add_sub %= my_term_add | my_term_sub | my_term_mul_div;
my_term_add = // a+b, ...
my_term_mul_div [push_back(at_c<1>(_val), _1)]
  >> +(
    lit('+') [at_c<0>(_val) = 0]
    >> (
      my_term_sub [push_back(at_c<1>(_val), _1)] |
      my_term_mul_div [push_back(at_c<1>(_val), _1)]
my_term_sub = // The same as for add but reverse (don't call my_term_sub but my_term_add)

my_term_mul_div %= my_term_mul | my_term_div | my_term;
my_term_mul = // a*b, ...
my_term [push_back(at_c<1>(_val), _1)]
  >> +(
    lit('*') [at_c<0>(_val) = 2]
    >> (
      my_term_div [push_back(at_c<1>(_val), _1)] |
      my_term [push_back(at_c<1>(_val), _1)]
my_term_div = // The same as for add but reverse (don't call my_term_div but my_term_mul)

my_term_set =
  lit("LEER") |
  lit('(') >> lit("LEER") >> lit(')') |
    >> LIST_ELEMENT [push_back(_val, _1)] % char_(",;")
    >> lit(')')
my_term_sub_expression =
  >> my_expression [_val = _1]
  >> lit(')');
my_term_function =
  NAME [at_c<0>(_val) = _1]
  >> lit('(')
  >> my_equation [push_back(at_c<1>(_val), _1)] % char_(",;")
  >> lit(')');
my_term %=
  my_term_function |
  my_term_sub_expression |
  my_term_set |

The AST expres­sion tree is modelled as follows:

namespace myns
  struct equation;
  typedef boost::recursive_wrapper< equation > equation_value;
  struct func {
    std::string name;
    std::vector< equation_value > arguments;

  struct expression;
  typedef boost::variant< boost::recursive_wrapper< expression >,
    std::vector< std::string >,
    std::string > expression_value;

  struct expression {
    expression() : operation(-1) {}
    int operation;
    std::vector< expression_value > values;

  struct equation {
    equation() : negate(false), operation(-1) {}
    bool negate;
    expression_value left;
    int operation;
    expression_value right;

  (std::string, name)
  (std::vector< qslang::equation_value >, arguments)
  (int, operation)
  (std::vector< qslang::expression_value >, values)
  (bool, negate)
  (qslang::expression_value, left)
  (int, operation)
  (qslang::expression_value, right)

I’ll leave this uncom­men­ted. It’s basically EBNF with attrib­utes and the data struc­tures should be fairly self-explan­at­ory (haha; I’m just lazy, agreed). Sadly, I cannot post the complete source and maybe it’s not even the best way to model the grammar. But it should help inter­ested coders getting the idea.

Binary World Private Work


The single-eyed that con­stantly reminds me of the obvious at work. In general, I mean :)

No comment
No comment
Excitements Travel Work

A Trip to İYTE, Part IV

Again a new day and the last one in İYTE. After finishing all the relevant duties, a car brought us to the city centre of Izmir. Being that close to the airport hopefully makes getting to the airport it easier for us.

The tem­per­at­ure’s still well above the average I’m familiar with. However, it seems that I slowly adapt to the quite different envir­on­ment­al con­di­tions including not being under­stood well enough. Con­sid­er­ing the kind of summer we currently have in Germany, getting used to the heat is quite insane. Improving com­mu­nic­a­tion abilites where possible, at least to get some tea, in turn, should be a well-thought idea and seems to work out somehow. Except for this Wednesday evening: we decided to walk along the water front, looked out for some nice res­taur­ant, and even­tu­ally passed by one of these nice-looking seafood places. Ok, less nice-looking than clean and organised but the menu read quite well. And as I thought of having some appet­isers which the menu promoted in a really tasty way, we dicided to risk it. In the end it really was risky, though.

Looking at the dishes promised something really worth the prices — price-wise, they’re all well above what we found elsewhere in Izmir. We started with some appet­isers for 6 TL each, as we found out later (2 TL relates approx­im­ately 1 EUR). Quite expensive for what we finally got. Water was obviously not included as they charged another 8 TL (refill, I suppose). The cover charge was 6 TL per person. Then we wanted to eat some fish, which actually was the reason why we hung around that place. I called for the waiter and asked him how to order fish; the main dishes seemed to have nothing to do with sea or food or seafood. He explained to me that we would be charged 100 TL per kilo of (fresh?) fish and that the fish weights quite exactly 1.25 kg. Being charged by kilo is perfectly ok but not that price for that fish. So I told him that we first have to discuss and that I’ll come back on him or his colleague later. He looked like being a bit annoyed and threw the fish back into the box.

We finally decided to only have some salad for 9 TL and 19 TL respect­ively. Not that cheap but still not that expensive either. And it really was enough. Just in the middle of our green meal, a waitress approaches with two pices (halves) of fish, grilled and plain without any side dishes. Quite aston­ished I told her that we didn’t order fish but salad. She, looking quite confused, took the fish and called for the other waiter. He approached, again with the fish. I again told him that we did not order fish but a salad. For­tu­nately, he vanished imme­di­ately, though with a quite fierce look. From that moment on, he always looked very annoyed when passing by.

We finished our salads and ordered the bill. Something around 70 TL (including an already added tip of 10%) for a salad with some appet­isers. Well done, tourists!

A photo and the name of this place with the not-so-friendly waiters will follow. By the way, they even only offered paper napkins, so really no high-class place but something with exclus­ively high-class prices (also food-wise). I’d guess that there are many res­taur­ants in the centre of İzmir that offer good service and quality. So try to look out for these or maybe give me some hints? Don’t actually know when I’ll be back ™ but who knows? :)

Except the fact that I was almost knocked over by a taxi (taksi) because I just didn’t notice that something very fast approaches me just as fast, nothing else worth to be mentioned happened that evening.

Tomor­row’s the last regular day in Izmir. We’ll start preparing ourselves for Germany and try to buy some nice sweets that might help bridging the gap between 16°C and 36°C.

Excitements Travel Work

A Trip to İYTE, Part III

This moring I realised that leaving my shoes outside is really not a so good idea. It seems that something (probably a dog) took one of my shoes and … well, it’s gone. Thank­fully, I still have one pair. When going back to the City of Izmir, I’ll have to have a look at the bazaar in search for shoes. The ones left are defin­it­ively not suited very well for the weather in Germany — about 26°C less than here in Turkey.

Tuesday again was a warm day, even though it would have get a bit “colder”. Even though we only had around 36°C, the afternoon was reserved for my book. In the morning, the air con­di­tioned computer engin­eer­ing depart­ment helped for­get­ting the dry outside and even­tuelly, I also didn’t see the sun that often. A dis­cus­sion about some ideas for joint projects and a possible erasmus cooper­a­tion almost concluded that second day at İYTE.

One thing still open on my todo list for that: pay the guest house. Seemed to me like a quite straight forward idea but again, the language problem showed up. Almost nobody I met on and around the campus is able to speak or under­stand the English language. With a little bit of gesturing, however, things get man­age­able again. The very nice guy from the security in the dean’s office tried to guide me into the right direction and offered me a bagfull of Turkish words I really wasn’t able to translate into something more useful to me. Ok, I admit that one should probably be better prepared when trav­el­ling to a foreign country. As it was only one week, I decided to go with only English, though. Finally, after talking or gesturing to two more persons (to be honest, I just showed them a post-it with the depart­ment and the name of the respons­ible person written to it), both of them very, very nice and very helpful, I reached my des­tin­a­tion and was able to pay. Ten minutes later I let the building, headed towards the computer science depart­ment and again stayed there for some minutes to enjoy the well-tem­per­at­ured air and use a free internet access.

Con­sid­er­ing the WLAN thingy, there’s something to say here. In one of the previous posts I mentioned that there is free WLAN available on the campus. Well, yes, there are open access points but the one I used in the beginning was shut down on Monday. The second one I spotted worked quite well except that the signal was much worse. Yesterday evening, after having placed a phone call where I had to stand quite still in the middle of the street in order to have a well-balanced signal to no signal ratio, a guy from the security approached slowly. I decided to end the con­ver­sa­tion and leave.

During my first day on the campus, I spotted several other open networks, each of which would have required me to walk more than thirty minutes one way. Yesterday, I spotted another two, one on the campus and one in a bar right next to the campus. But as I’ve decided to go to Izmir on Wednesday afternoon and Thursday it’s now obsolete anyway. But more on that tomorrow, once everything’s finished and packed again.

Excitements Travel Work

A Trip to İYTE, Part II

The second day is over and still, it was again a bit warmer than it is in Germany right now. The lab tour in the DIVE labor­at­ory at İYTE had to be postponed to tomorrow, so no news here. I had nice a little chat with a colleague from Hungary (who was invited last week and leaves İYTE today) and staff from İYTE. That was basically it. Over noon, doing something outside was defin­it­ively no option, so after leaving the institute, I decided in favour of reading a book. For the evening, Bora has invited us for a BBQ which was really, really good (espe­cially the appet­isers).

İYTE and the area around is really a nice place to stay, even in Summer if you can manage to avoid working around noon. Many insects can be seen every­where around houses, such as crickets or grasshop­pers. A lot of dogs are around on the whole campus. Just some ten years ago, as Bora told me, (wild) dogs were spread all over the cities as well. To fight them, the gov­ern­ment dis­trib­uted toxins. Just about ten years ago, they started using a less deadly strategy. As of now, dogs are mainly seen outside the cities and towns where cats have taken their places. But still, when you first come the die İYTE campus, you’ll have to get used to being followed by at least one dog. And I’m not talking about bag dogs that are more like a rat than a dog! They’re real dogs and very lovable animals. We also spotted some tortoises (turtles) right next to the entrance of the flat.

Excitements Travel Work

A Trip to İYTE, Part I

It was around January when I was invited to give an Erasmus lecture at the İzmir Institute of Tech­no­logy (IYTE) in İzmir, Turkey. The campus is located near Urla at the Aegean Sea. Now in July, one of the hottest months of the year in that medi­ter­ranean area with tem­per­at­ures around 35°C in average, it would be the best to just stay in or near the sea, lying under and be shaded by palm trees for the whole day. But as this is no holiday but a working trip, I’ll probably be in air­con­di­tioned rooms more often than at the seaside anyway. Be assured, I’ll try to optimise this time schedule towards the latter option :) For a guy from Germany, the tem­per­at­ures are defin­it­ively much more con­veni­ent in Spring or Autumn. This week, the weather forecast announced around 40°C.

On Saturday evening, my plane arrived at İzmir airport (ADB). A car from İYTE waited and took me right onto the campus near Urla and the small town Güzel­bahçe. Having some guest room on the campus is much more con­veni­ent, I thought, so I booked it for the week. Having a cab to take me to the institute every day would defin­it­ively be annoying. The lectures were scheduled for three days, Monday to Wednesday but only for half a day each. The topic of the whole course is “Beha­vi­our­al Robotics” whereas my part covers cooper­a­tion in that domain. The slides will be made available soon.

The DIVE Lab headed by Bora İ. Kumova is a well equipped place to study robotics. A bunch of NXTs are available for labs and students to play around with — given they prefer doing stuff like that instead of hanging out with friends in some bar near the campus. I’ll cover more on that topic (robotics, not the bar) later today, as Bora will hopefully show me around his lab and the institute.

Yesterday was my first day at İYTE. We drove around the campus and had a look at the buildings and insti­tutes, the small town of Güzel­bahçe, and the beach. It’s only about 2 km away from the guest house and well reachable by foot; you only have to take care not to step over the dead fox lying somewhere on the trail. The location gives a quite impress­ive scenery, lying on a kind of small peninsula between mountains and the Aegean sea. There are res­taur­ants and super­mar­kets available in Güzel­bahçe, offering fresh fruit grown in the region, so you don’t need to be afraid of starving. A drawback: the sea is not that refresh­ing as the water is just as the air around it, warm.

For­tu­nately, several open WLANs (and therefore free) are available on the campus, so it is possible to check mails or even place a call. The mobile phone costs would otherwise ruine the callee for sure.

Today, I’ll be in the computer engin­eer­ing depart­ment at İYTE and maybe can provide more inform­a­tion later today. Some pictures will also be added once better suited Internet access is available.

Excitements Work

Linking weak references

Höllenjob (bildlich und wörtlich) und nüchterne Analyse (“Manchmal ist das Hil­freich­ste, was man bei einem Unglück tun kann, nicht im Weg zu stehen.”; Robin Murphy/c’t).

Sorry, musste sein.

Binary World Excitements Work

Type Casting

Even though I started pro­gram­ming more than 16 years ago and got to used to several different pro­gram­ming languages in the meantime, this is the first time I came across the behaviour shown below.

Assume the following piece of code:

int main(int argc, char *argv[]) {
    int x = -2;
    unsigned int y = 10;
    std::cout << (x * y) << std::endl;
    std::cout << (x - y) << std::endl;

I’d have expected something like


but instead it returns


Obviously, this is a type casting issue. Well, more a kind of pecu­li­ar­ity caused by the C/C++ type hierarchy: The signed integer is promoted to an unsigned int, causing the whole cal­cu­la­tion to be handled this way. Nothing new but quite inter­est­ing for me and some col­leagues, though :)

For more inform­a­tion in C/C++ type casting issues, have a look at the following web page: