Powerpoint monkey

I doubt this announcement will be of much use to anyone, but I've uploaded most of my latex presentations to my github repo (those for which I found the source code, anyway).

At least it's a great way to see crappy latex code which probably hasn't compiled in newer versions of latex for at least a couple of years.

C++ exceptions under the hood 4: catching what you throw

In this series about exception handling, we have discovered quite a bit about exception throwing by looking at compiler and linker errors but we have so far not learned anything yet about exception catching. Let's sum up the few things we learned about exception throwing:

• A throw statement will be translated by the compiler into two calls, __cxa_allocate_exception and __cxa_throw.
• __cxa_allocate_exception and __cxa_throw "live" on libstdc++
• __cxa_allocate_exception will allocate memory for the new exception.
• __cxa_throw will prepare a bunch of stuff and forward this exception to _Unwind_, a set of functions that live in libstdc and perform the real stack unwinding (the ABI defines the interface for these functions).

Quite simple so far, but exception catching is a bit more complicated, specially because it requires certain degree of reflexion (that is, the ability of a program to analyze its own source code). Let's keep on trying our same old method, let's add some catch statements throughout our code, compile it and see what happens:

#include "throw.h"
#include <stdio.h>

// Notice we're adding a second exception type
struct Fake_Exception {};

void raise() {
    throw Exception();
}

// We will analyze what happens if a try block doesn't catch an exception
void try_but_dont_catch() {
    try {
        raise();
    } catch(Fake_Exception&) {
        printf("Running try_but_dont_catch::catch(Fake_Exception)\n");
    }

    printf("try_but_dont_catch handled an exception and resumed execution");
}

// And also what happens when it does
void catchit() {
    try {
        try_but_dont_catch();
    } catch(Exception&) {
        printf("Running try_but_dont_catch::catch(Exception)\n");
    } catch(Fake_Exception&) {
        printf("Running try_but_dont_catch::catch(Fake_Exception)\n");
    }

    printf("catchit handled an exception and resumed execution");
}

extern "C" {
    void seppuku() {
        catchit();
    }
}

Note: You can download the full sourcecode for this project in my github repo.

Just like before, we have our seppuku function linking the C world with the C++ world, only this time we have added some more function calls to make our stack more interesting, plus we have added a bunch of try/catch blocks so we can analyze how does libstdc++ handles them.

And just like before, we get some linker errors about missing ABI functions:

> g++ -c -o throw.o -O0 -ggdb throw.cpp
> gcc main.o throw.o mycppabi.o -O0 -ggdb -o app
throw.o: In function `try_but_dont_catch()':
throw.cpp:12: undefined reference to `__cxa_begin_catch'
throw.cpp:12: undefined reference to `__cxa_end_catch'

throw.o: In function `catchit()':
throw.cpp:20: undefined reference to `__cxa_begin_catch'
throw.cpp:20: undefined reference to `__cxa_end_catch'

throw.o:(.eh_frame+0x47): undefined reference to `__gxx_personality_v0'

collect2: ld returned 1 exit status

Again we see a lot of interesting stuff going on here. The calls to __cxa_begin_catch and __cxa_end_catch are probably something we could have expected: we don't know what they are yet, but we can presume they are the equivalent of the throw/__cxa_allocate/throw conversions (you do remember that our throw keyword got translated to a pair of __cxa_allocate_exception and __cxa_throw functions, right?). The __gxx_personality_v0 thing is new, though, and the central piece of the next few articles.

What does the personality function do? We already said something about it on the introduction to this series but we will be looking into it with some more detail next time, together with our new two friends, __cxa_begin_catch and __cxa_end_catch.

Bebugging / Fault injection

Releasing any software with a degree of confidence on its quality can be a difficult task. A way to improve this confidence is adding test. Easy enough, but then how do you know if you are actually testing what needs to be tested? Metrics like code coverage are very helpful but they also provide a false sense of security that can be even worse than having no unit testing at all.

One way of determining how reliable your testing suit is, is to test your testing suit. No, not by writing more tests but by writing more bugs!

The idea is simple: give your code to someone who is not a programmer on the project, someone who can know nothing about the implicit assumptions and preconditions you and your team mates already have about the code, the same assumptions and preconditions necessary to make stuff don't crash. Ask him to break things. Nothing too fancy, only subtle stuff; a memory leak over there, a sign vs unsigned comparison over here, an equal changed by a non-equal in an if (and please do it in a branch!).

Once you get a faulty branch of your project, try to see how many of the bugs you can detect using your testing suite (yes, valgrind should probably be part of your testing suit, albeit not a unit-test). No diffs, please.

Seeing how many bugs go unnoticed on a fault injection session can give you an idea of how comprehensive your unit tests are. It can be a very humbling experience, too.

You can find more information about bebugging in Wikipedia.

C++ exceptions under the hood 3: an ABI to appease the linker

On our journey to understand exceptions we discovered that the heavy-lifting is done in libstdc++ as specified by the C++ ABI. Reading some linker errors we deduced last time that for handling exceptions we need help from the C++ ABI; we created a throwing C++ program, linked it together with a plain C program and found that the compiler somehow translated our throw instruction into something that is now calling a few libstd++ functions to actually throw an exception. Lost already? You can check the sourcode for this project so far in my github repo.

Anyway, we want to understand exactly how an exception is thrown, so we will try to implement our own mini-ABI, capable of throwing an exception. To do this, a lot of RTFM is needed, but a full ABI interface can be found here, for LLVM. Let's start by remembering what those missing functions are:

> gcc main.o throw.o -o app
throw.o: In function `foo()':
throw.cpp:4: undefined reference to `__cxa_allocate_exception'
throw.cpp:4: undefined reference to `__cxa_throw'
throw.o:(.rodata._ZTI9Exception[typeinfo for Exception]+0x0): undefined reference to `vtable for __cxxabiv1::__class_type_info'
collect2: ld returned 1 exit status

__cxa_allocate_exception

The name is quite self explanatory, I guess. __cxa_allocate_exception receives a size_t and allocates enough memory to hold the exception being thrown. There is more to this that what you would expect: when an exception is being thrown some magic will be happening with the stack, so allocating stuff here is not a good idea. Allocating memory on the heap might also not be a good idea, though, because we might have to throw if we're out of memory. A static allocation is also not a good idea, since we need this to be thread safe (otherwise two throwing threads at the same time would equal disaster). Given these constraints, most implementations seem to allocate memory on a local thread storage (heap) but resort to an emergency storage (presumably static) if out of memory. We, of course, don't want to worry about the ugly details so we can just have a static buffer if we want to.

__cxa_throw

The function doing all the throw-magic! According to the ABI reference, once the exception has been created __cxa_throw will be called. This function will be responsible of starting the stack unwinding. An important effect of this: __cxa_throw is never supposed to return. It either delegates execution to the correct catch block to handle the exception or calls (by default) std::terminate, but it never ever returns.

vtable for __cxxabiv1::__class_type_info

A weird one... __class_type_info is clearly some sort of RTTI, but what exactly? It's not easy to answer this one now and it's not terribly important for our mini ABI; we'll leave it to an appendix for after we are done analyzing the process of throwing exceptions, for now let's just say this is the entry point the ABI defines to know (in runtime) whether two types are the same or not. This is the function that gets called to determine whether a catch(Parent) can handle a throw Child. For now we'll focus on the basics: we need to give it an address for the linker (ie defining it won't be enough, we need to instantiate it) and it has to have a vtable (that is, it must have a virtual method).

Lot's of stuff happen on these functions, but let's try to implement the simplest exception thrower possible: one that will call exit when an exception is thrown. Our application was almost OK but missing some ABI-stuff, so let's create a mycppabi.cpp. Reading our ABI specification we can figure out the signatures for __cxa_allocate_exception and __cxa_throw:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

namespace __cxxabiv1 {
    struct __class_type_info {
        virtual void foo() {}
    } ti;
}

#define EXCEPTION_BUFF_SIZE 255
char exception_buff[EXCEPTION_BUFF_SIZE];

extern "C" {

void* __cxa_allocate_exception(size_t thrown_size)
{
    printf("alloc ex %i\n", thrown_size);
    if (thrown_size > EXCEPTION_BUFF_SIZE) printf("Exception too big");
    return &exception_buff;
}

void __cxa_free_exception(void *thrown_exception);

#include <unwind.h>
void __cxa_throw(
          void* thrown_exception,
          struct type_info *tinfo,
          void (*dest)(void*))
{
    printf("throw\n");
    // __cxa_throw never returns
    exit(0);
}

} // extern "C"

Note: You can download the full sourcecode for this project in my github repo.

If we now compile mycppabi.cpp and link it with the other two .o files, we'll get a working binary which should print "alloc ex 1\nthrow" and then exit. Pretty simple, but an amazing feat nonetheless: we've managed to throw an exception without calling libc++. We've written a (very small) part of a C++ ABI!

Another important bit of wisdom we gained by creating our own mini ABI: the throw keyword is compiled into two function calls to libstdc++. No voodoo there, it's actually a pretty simple transformation. We can even disassemble our throwing function to verify it. Let's run this command "g++ -S throw.cpp".

seppuku:
.LFB3:
    [...]
	call	__cxa_allocate_exception
	movl	$0, 8(%esp)
	movl	$_ZTI9Exception, 4(%esp)
	movl	%eax, (%esp)
	call	__cxa_throw
    [...]

Even more magic happening: when the throw keyword gets translated into these two calls, the compiler doesn't even know how the exception is going to be handled. Since libstdc++ is the one defining __cxa_throw and friends, and libstdc++ is dynamically linked on runtime, the exception handling method could be chosen when we first run our executable.

We are now seeing some progress but we still have a long way to go. Our ABI can only throw exceptions right now. Can we extend it to handle a catch as well? We'll see how next time.

Monitor file changes on a CLI

The other day I had a problem with a config file being overwritten. Some process, I did not know which one, was overwriting a configuration file I manually changed. Annoyed by this, I started looking for the culprit. lsof was no good, because that would list the open files; this process would most likely just open the file, write to it and then close it again. My chances of ever catching this process in the act were nil. Luckily I found auditd. Install it like this:

sudo apt-get install auditd

Then to monitor a file you can use the following command:

sudo auditctl -w $FILE -p war

Wait until $FILE has changed, then execute this command to get the results:

ausearch -f $FILE

Voila, now you have your culprit. Kill -9 at will.

C++ exceptions under the hood II: a tiny ABI

If we are going to try and understand why exceptions are complex and how do they work, we can either read a lot of manuals or we can try to write something to handle the exceptions ourselves. Actually, I was surprised by the lack of good information on this topic: pretty much everything I found is either incredibly detailed or very basic, with one exception or two. Of course there are some specifications to implement (most notably the ABI for c++ but we also have CFI, DWARF and libstdc) but reading the specification alone is not enough to really learn what's going on under the hood.

Let's start with the obvious then: wheel reinvention! We know for a fact that plain C doesn't handle exceptions, so let's try to link a throwing C++ program with a plain C linker and see what happens. I came up with something simple like this:

#include "throw.h"
extern "C" {
    void seppuku() {
        throw Exception();
    }
}

Don't forget the extern stuff, otherwise g++ will helpfully mangle our little function's name and we won't be able to link it with our plain C program. Of course, we need a header file to "link" (no pun intended) the C++ world with the C world:

struct Exception {};

#ifdef __cplusplus
extern "C" {
#endif

    void seppuku();

#ifdef __cplusplus
}
#endif

And a very simple main:

#include "throw.h"

int main()
{
    seppuku();
    return 0;
}

What happens now if we try to compile and link together this frankencode?

> g++ -c -o throw.o -O0 -ggdb throw.cpp
> gcc -c -o main.o -O0 -ggdb main.c

Note: You can download the full sourcecode for this project in my github repo.

So far so good. Both g++ and gcc are happy in their little world. Chaos will ensue once we try to link them, though:

> gcc main.o throw.o -o app
throw.o: In function `foo()':
throw.cpp:4: undefined reference to `__cxa_allocate_exception'
throw.cpp:4: undefined reference to `__cxa_throw'
throw.o:(.rodata._ZTI9Exception[typeinfo for Exception]+0x0): undefined reference to `vtable for __cxxabiv1::__class_type_info'
collect2: ld returned 1 exit status

And sure enough, gcc complains about missing C++ symbols. Those are very special C++ symbols, though. Check the last error line: a vtable for cxxabiv1 is missing. cxxabi, defined in libstdc++, refers to the application binary interface for C++. So now we have learned that the exception handling is done with some help of the standard C++ library with an interface defined by C++'s ABI.

The C++ ABI defines a standard binary format so we can link objects together in a single program; if we compile a .o file with two different compilers, and those compilers use a different ABI, we won't be able to link the .o objects into an application. The ABI will also define some other formats, like for example the interface to perform stack unwinding or the throwing of an exception. In this case, the ABI defines an interface (not necessarily a binary format, just an interface) between C++ and some other library in our program which will handle the stack unwinding, ie the ABI defines C++ specific stuff so it can talk to non-C++ libraries: this is what would enable exceptions thrown from other languages to be caught in C++, amongst other things.

In any case, the linker errors are pointing us to the first layer into exception handling under the hood: an interface we'll have to implement ourselves, the cxxabi. For the next article we'll be starting our own mini ABI, as defined in the C++ ABI.

Quick C/C++ reference on the console

What do you usually do when you don't remember whether fp is the first argument for fprintf or for fwrite? (Hint: it's the first argument for one, the last for the other. Handy, huh?). Well, there's no need to go and check the answer on Google, just sudo apt-get install manpages-dev and then do "man fprintf" or "man fwrite".

Note: you might also want posix-dev to check stuff like htons.

C++ exceptions under the hood

Everyone knows that good exception handling is hard. Reasons for this abound, in every single layer of an exception "lifetime": it's hard to write exception safe code, an exception might be thrown from unexpected places (pun intended!), it's can be complicated to understand badly designed exception hierarchies, it's slow because a lot of voodoo is happening under the hood, it's dangerous because improperly throwing an exception might call the unforgiving std::terminate. And although anyone who might have had to battle an "exceptional" program might know this, the reasons for this mess are not widespread knowledge.

The first question we need to ask ourselves is then, how does it all work. This is the first article on a long series, in which I'll be writing about how exceptions are implemented under the hood in c++ (actually, c++ compiled with gcc on x86 platforms but this might apply to other platforms too). On these articles the process of throwing and catching an exception will be explained with quite a lot of detail, but for the impatient people here is a small brief of all the articles that will follow: how is an exception thrown in gcc/x86:

1. When we write a throw statement, the compiler will translate it into a pair of calls into libstdc++ functions that allocate the exception and then start the stack unwinding process by calling libstdc.
2. For each catch statement, the compiler will write some special information after the method's body, a table of exceptions this method can catch and a cleanup table (more on the cleanup table later).
3. As the unwinder goes through the stack it will call a special function provided by libstdc++ (called personality routine) that checks for each function in the stack which exceptions can be caught.
4. If no matching catch is found for the exception, std::terminate is called.
5. If a matching catch is found, the unwinder now starts again on the top of the stack.
6. As the unwinder goes through the stack a second time it will ask the personality routine to perform a cleanup for this method.
7. The personality routine will check the cleanup table on the current method. If there are any cleanup actions to be run, it will "jump" into the current stack frame and run the cleanup code. This will run the destructor for each object allocated at the current scope.
8. Once the unwinder reaches the frame in the stack that can handle the exception it will jump into the proper catch statement.
9. Upon finishing the execution of the catch statement, a cleanup function will be called to release the memory held for the exception.

This already looks quite complicated and we haven't even started; that was but a short and inaccurate description of all the complexities needed to handle an exception.

To learn about all the details that happen under the hood on the next article we will start to implement our own mini libstdlibc++. Not all of it though, only the part that handles exceptions. Actually not even all of that, only the bare minimum we need to make a simple throw/catch statement work. Some assembly will be needed, but nothing too fancy. A lot of patience will be required, I'm afraid.

If you are too curious and want to start reading about exception handling implementation then you can start here, for a full specification of what we are going to implement on the next few articles. I'll try to make these articles a bit more didactic and easier to follow though, so see you next time to start our ABI!

** Disclaimer note: I'm in no way versed on the magic going on when an exception is thrown. These series will be about trying to demystify the stuff going on under the hood and learning something in the process, and while I hope some of it will be correct I have no doubts there will be a lot of subtleties not quite right. Let me know if you think I should correct something **

Cool C++0X features XVII: Lambdas syntax

After creating a device to sum an initializer list as an example of this new feature last time, we created a generic function to receive a function object, which worked just very similarly to Smalltalk's inject. After creating the usual function object, overloading the () operator, we saw a much cooler way of doing that using lambdas, like this:

int main() {
	auto f = [] (int a, int b){ return a+b; } ;
	do_something(f, 0, {1, 2, 3, 4});
	return 0;
}

Pretty cool. Pretty weird too. Let's analyze how to declare a lambda before we continue discussing about the usage of this new feature.

auto f

Auto f. Lambdas have a type, and you can explicitly declare it. The type might be quite complicated though, and it doesn't really add any information that makes reading the code any easier, so we are better off using C++0x's new feature, auto, which will infer the type for us. Saves a lot of typing, trust me.

auto f = [] (int a, int b)

Brackets, and then the parameters specification. Looks weird but the brackets are there just to tell the compiler "hey, an anonymous method comes here" (actually the brackets could be omitted in this case but we'll need them later on). After that, it's just a normal method declaration. Useless trivia: before C++0x you could have anonymous objects but not anonymous methods. Can you think where would you have an anonymous object? I think I even wrote an article about it on this blog, but I'm too lazy to search for it.

auto f = [] (int a, int b) { return a+b; }

After the lambda's signature, which is the same as the signature of a common method, you have the body of the method. It can be as short or as long as you want, it's just a method's body (though for the sanity of the maintainer you'd better keep it short).

Watch out though, that's not the end of the declaration, we're missing a crucial piece on this lambda:

auto f = [] (int a, int b) { return a+b; }<strong>;</strong>

I'd use the blink tag, but I think it has been deprecated. Notice that last semicolon; when you declare the lambda's body you finish with a semicolon inside, just as you would inside a normal method, but that's not the end of the expression, for the lambda declared outside the body of that method still needs another semicolon to appraise the compiler god.

And now, we know almost everything we need to know about lambda's syntax. Notice how the return type of the method is automagically deduced. That's useful, but there's something the compiler can't deduce by itself about the return type. If you are trying to return a reference to something, you need to make this explicit, the compiler has no way of detecting if you need a copy return or a reference return (until the next draft of C++, which I heard incorporates mind reading capabilities into the compiler. You may have to wait a couple of years though). This is easy to specify, though:

auto f = [&] (int a, int b) { return a+b; };

Just add an ampersand between the brackets and the lambda will return a reference instead of a copy of whatever object you're trying to return. How about receiving a reference instead of returning one? That's easy, remember the signature is the same as any other method:

auto f = [] (int& a, int& b) { return a+b; };

That's it, now we know how to use basic lambdas. You should keep in mind, though, this mythical beast has a lot more to it than just its syntax. We'll discuss about some of it's darkest secrets, how to use it, when to use it, how does it work. That discussion is for next time, though.

Cool C++0X features XVI: Lambdas

Last time we created a device to sum an initializer list of ints, something like this:

void Add(initializer_list<int> lst) {
	int sum = 0;
	for (auto i = lst.begin(); i != lst.end(); ++i)
		sum += *i;

	cout << sum << "n";
}

And then we said this can be improved using some new C++0x wizardry to support actions other than adding. How would we do that? Easy, we need to decouple the iteration of the list from the operation logic. We can do something like this:

// Note how we don't care about the type of OP, just that
// it can be called (i.e. has an operator ())
template <class OP>
void do_something(OP op, int init, initializer_list<int> lst) {
	int sum = init;
	for (auto i = lst.begin(); i != lst.end(); ++i)
	{
		int x = *i;
		sum = op(sum, x);
	}

	cout << sum << "n";
}

struct Sum {
	int operator() (int a, int b)
	{
		return a + b;
	}
};

int main() {
	do_something(Sum(), 0, {1, 2, 3, 4});
	return 0;
}

We had to do some changes other than passing the operation-object into the do_something method; since the start value (zero) was hardcoded we had to remove it to really decouple the action from the iteration.

Other than creating a function object (which is the correct name for the object wrapping our operation) we don't see any strange changes, there's no C++0x there, but C++0x gives us a little tool which gives you the power of creating much simpler and nicer code, or to make the next maintainers' life a living hell. That's a discussion for other time though, now let's take a sneak preview a lambdas, the evolution of function objects:

int main() {
	auto f = [] (int a, int b){ return a+b; } ;
	do_something(f, 0, {1, 2, 3, 4});
	return 0;
}

Note that we didn't change anything on the method iterating the list, we just changed main! There's a lot to talk about lambdas, so this is only an intro to the subject. Next time we'll discuss the subtleties of the new syntax.

"The VM session was closed before any attempt to power it on"

Some times Virtualbox gets lazy and decides not to work, no explanation given (no, that cryptic message certainly doesn't count as an explanation). I have no idea what causes this message, at the moment my best theory is an improper planet alignment, but in my experience detaching all media from a VM, running it and then attaching all media again seems to fix the problem.

Printing broken in Ubuntu? Try lpr

Lately I've seen quite a few Ubuntu machines with broken printing (more broken than usual, that is). Sometimes printers are detected, sometimes not. When it doesn't work you usually get a warning saying "Getting printer information failed" even though the printer is connected and you can even ping it. Searching around a little bit I read a couple of threads and bug reports that make me think that it's actually a Gnome 3 migration problem, but I'm not sure. It doesn't matter much anyway: screw UIs, just use lpr < foo.pdf from a CLI and keep on killing forests!

On a cheerful closing note: how funny is it that when the regular printing applet in Gnome is broken we have to use a utility by "Apple Inc"?

Setting up a Linux gateway/router

Yes, there are a lot of guides for this, but since I need to document how I did it for my office a lot of time ago, I might as well write a post about it here.

As expected, if we are going to replace a device, say, a router, we need to replace it with something that can provide the same functionality. In this case, we have chosen a Linux server, so we need to figure out which services are provided by the router and then emulate them someway:

• DHCP to manage leases
• DNS to translate domains to IPs
• NAT, to multiplex a single connection
• Service forwarding, to expose internal services to an external network

Luckily Linux supports all of these:

ISC for DHCP

bind9 for DNS

iptables for NAT

iptables again, for service forwarding

Let's set up each of these services.

Preliminary work

Before you setup any services, you are going to need two things: first two network cards, one for the outgoing connection and another one for the (switched) LAN, and a way of telling your server that you want all traffic from network 1 forwarded to network 2. You may want to install more than two cards, in case you need to route several LANs. We'll see that later.

You will also need an OS. I have chosen Ubuntu because it's very simple to install, and has all the software we need available in the repositories, but you can use any other distribution if it suits your needs.

Also, throughout this guide I will assume a setup like this:

• WAN access through eth0, DHCP address
• LAN routing in eth1, network 192.168.10.1/24

Setting up NATting and forwarding

Services like DNS and DHCP are nice-to-have, but having real connectivity is way more important. Let's set up the NAT and connection forwarding features of the new router, then we can test if our setup is working properly by pinging an IP of one LAN from the other.

We'll do this by setting up NAT with iptables. We'll also have to configure the OS to forward connections from one network card to the other:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
# Add a line like this for each eth* LAN
iptables --append FORWARD --in-interface eth1 -j ACCEPT

We will also need to setup the IP for eth0, since there won't be a DHCP server (we ARE the server!). Open /etc/network/interfaces and add something like this:

# Configure the WAN port to get IP via DHCP
auto eth0
iface eth0 inet dhcp
# Configure the LAN port
auto eth1
iface eth1 inet static
	address 192.168.10.1	# (Or whatever IP you want)
	netmask	255.255.255.0	# Netmasks explanations not included

Once that's checked, restart networking services like this:

sudo /./etc/init.d/networking restart

Everything ready, now just plug your PC to the new router and test it. Remember to manually set your IP in the same network range as your router, since there's no DHCP at the moment. This may be useful to debug a problem.

In your client PC, set your IP address:

ifconfig eth0 192.168.10.10

Test if you IP is set:

ping 192.168.10.10

If you get a reply, your new IP is OK, if not there's a problem with your client. Second step, see if you can reach the router:

ping 192.168.10.1

Note that you may need to renew everything (i.e. restart networking and manually assign your IP) after you connect the cable.

Again, if you get a reply then you have connectivity with the router. So far we haven't tested the iptables rules nor the forwarding, so any issue at this point should be of IP configuration. If everything went well, it's time to test the NAT rules and the forwarding.

ping 192.168.1.1

That should give you an error. Of course, since there's no DHCP there's no route set. Let's manually set a route in the client:

sudo route add default gateway 192.168.10.1

Then again:

ping 192.168.0.1

Magic! It works! If it doesn't, you have a problem either in the NAT configuration or the IP Forwarding of the router. You can check this with wireshark, if the pings reach the server but they never get a reply back then it's the NAT, i.e. it can forward the IP packages on eth1 to eth0 but the router has no NAT, and it doesn't know how to route the answer back. If the pings never even reach eth0, then you have an ip forwarding problem.

We can try something more complex now, like pinging a domain instead of an IP. Something like this:

ping google.com

You should get a message saying the host is unknown. Can you guess why? Right, there's no DNS. Let's install one, but first we have to make these changes permanent.

Persisting the forwarding rules

In order to have the forwarding rules persisting after a reboot, we need first to change /etc/sysctl.conf to allow IP forwarding. It's just a mater of uncommenting this line:

net.ipv4.ip_forward = 1

We will also have a lot of iptables rules we need to setup during boot time. I have created a script at /home/router/set_forwarding.sh, which I also linked into /etc/init.d/rc.local so it's run whenever the system boots.

Setting up DNS

DNS will be necessary to resolve domains to IPs. bind9 is the default option for Debian based servers (are there others? no idea).

sudo apt-get install bind9

This will get your DNS server up and running, but you will still need to add this server manually to your client (again, because there's no DHCP running):

sudo echo "nameserver 192.168.10.1" > /etc/resolv.conf

And now:

ping google.com

Magic again, it (may) work. If it doesn't, you may need to open /etc/bind/named.conf and setup your router (192.168.0.1) as a forwarder, then restart the bind server.

Setting up a custom TLD in your DNS

Now we have a DNS running, so we can set up local zones for the LAN. For example, if you want your router to have a nice user friendly name, instead of just an IP.

Let's start by adding a local zone to /etc/bind/named.conf.local, for a domain we'll call "boc":

zone "boc" {
        type master;
        file "/home/router/named/boc.db";
};

Now we need to add a reverse zone:

zone "10.168.192.in-addr.arpa" {
	type master;
        file "/home/router/named/rev.10.168.192.in-addr.arpa";
};

We still need to create both files (boc.db and rev.10.168.192.in-addr.arpa), but will do that later. Lastly, a place to log all the DNS queries (if you want):

logging {
    channel query.log {
        file "/home/router/named/dns.log";
        severity debug 3;
    };

    category queries { query.log; };
};

For the log entry I have chosen /home/router/named as the log directory, just because for this project I'm keeping everything together (config and logs) so it's easy for people not used to administer a Linux box, but of course this means that apparmor must be configured to allow reads and writes for bind in this directory. We'll get to that in a second, first let's create the needed zone files for our new TLD.

Remember our two zone files? I put them on /home/router/named, but usually they are on /etc/bind. Again, I did this so I can have all the config files together. These are my two files:

For boc.db:

boc.      IN      SOA     ns1.boc. admin.boc. (
                                                        2006081401
                                                        28800
                                                        3600
                                                        604800
                                                        38400
 )

boc.      IN      NS              ns1.boc.

wiki             IN      A       192.168.0.66
ns1              IN      A       192.168.0.1
router           IN      A       192.168.0.1

For rev.10.168.192.in-addr.arpa

@ IN SOA ns1.boc. admin.example.com. (
                        2006081401;
                        28800; 
                        604800;
                        604800;
                        86400
)

                     IN    NS     ns1.boc.
1                    IN    PTR    boc

Most of these lines are black magic, and since an explanation of both DNS and Bind is out of scope let's just say you can add new DNS entries by adding lines like this:

NICE_NAME           IN      A       REAL_IP

This will make bind translate NICE_NAME.boc to REAL_IP. Of course, this will depend on the TLD you defined. Now restart bind to get a fuckton of errors. It will complain about not being able to load a master file in /home/router/named. Remember that apparmor thing I mentioned?

Setting up apparmor to allow new directories

Apparmor is a service that runs in the background, checking what other binaries can and can't do. For example, it will allow bind9 to open a listening socket on port 53 (DNS), but it will deny an attempt to open a listening socket on port 64. This is a security meassure to limit the damage a compromised bind9 binary running as root might do. And since we are going to use a non standard configuration, we need to tell apparmor that it's OK.

After installing bind9 we should get a new file in /etc/apparmor.d/usr.sbin.named. Add the following lines at the bottom:

  /home/router/named/** rw,
  /home/router/named/ rw,

And restart apparmor service:

/./etc/init.d/apparmor restart

Since we were modifying apparmor to allow a non-standard bind installation, now restart bind too. This time it will start without any errors, and you should be able to tail -f /home/router/named/dns.log to see the DNS queries on real time. If it doesn't, check that /home/router/named is writable to the bind user (I did a chgrp -R bind named).

Setting up DHCP

We have DNS and NAT so far, but the client configuration so far has been absolutely manual. We can't have many clients with this sort of setup, so let's automate the client config with a DHCP server. Begin by installing isc-dhcp-server.

Edit /etc/dhcp/dhcpd.conf, set the domain-name and the domain-name-servers, like this:

option domain-name "boc";
option domain-name-servers ns1.boc, ns2.boc;

default-lease-time 86400;
max-lease-time 172800;

authoritative;

I'm not sure if the first line is needed. The other two will set the DNS servers for your clients. In my case, ns1 and ns2 resolve to 192.168.10.1 and 192.168.0.1 respectively. Also, increasing your lease time is recommended, I used one day for default leases. I set this DHCP server as the authoritative server. If this is your router, that's probably what you want.

Now we need to define the network topology:

# This is the WAN network, and we won't provide a service here
subnet 192.168.0.0 netmask 255.255.255.0 {
}

# Define the service we provide for the LAN
subnet 192.168.10.1 netmask 255.255.255.0 {
	range 192.168.10.100 192.168.10.200;
	option routers 192.168.10.1;
}

Now we need to restart ISC:

sudo /./etc/init.d/isc-dhcp-server restart

And now we need to check if everything worked in the client. It's easy this time, we just ask for an IP:

sudo dhclient
ifconfig

If everything went fine, we should now have an IP in the 100-200 range, as well as the DNS server in /etc/resolv.conf. We have now setup a very basic router and should be able to server several clients for basic browsing capabilities.

Moving DHCP config files

Since I want to keep everything together for easy administration, I will move the configuration files for DHCP to /home/router/dhcp. Changing the dhcpd.conf file itself is easy, just move the subnets declarations and add this line:

include "/home/router/dhcp/subnets.conf";
include "/home/router/dhcp/static_hosts.conf";

Like we did with bind, we need to configure apparmor. vim /etc/apparmor.d/usr.sbin.dhcpd and add this two lines:

/home/router/dhcp/ rw,
/home/router/dhcp/** rw,

Restart apparmor service, then restart dhcpd. Everything should work just fine.

Setting up port forwardings

In any LAN you'll probably want to expose some services to the outer world, be it for a bittorrent connection or because you have internal servers you need to access from outside your internal LAN. To do this, you'll have to tell your router to forward some external port to an internal one, like this:

iptables -t nat -A PREROUTING -i eth0 -p tcp 
	--dport PORT -j DNAT --to INTERNAL_IP:INTERNAL_PORT
# This rule may not be needed, depending on other chain confings
iptables -A INPUT -i eth0 -p tcp -m state --state NEW 
	--dport PORT -j DNAT --to INTERNAL_IP:INTERNAL_PORT

This is enough to expose a private server to the world, but it will not be very useful when your dynamic IP changes. You need to set INTERNAL_IP to be a static IP.

Setting up static IPs

Remember the static_hosts file we created before? We can use that to define a static IP. Add the following to set a static IP host:

host HostName {
	hardware ethernet 00:00:00:00:00:00
	fixed-address 192.168.10.50;
}

After that, just restart the DHCP service and renew your client's IP. Done, it's static now!

Wait a minute: how do you find the MAC for your host? I'm to lazy to copy and type it, so I did the following:

# cd /home/router/dhcp
# ln -s /var/lib/dhcp leases

Then you can check the hardware address in the leases/dhcpd.leases file. I created a symlink to keep this directory at hand, since it gives you a status of the current leases.

Warm restart for the router and easy administration

We have quite a few configuration files now, with different settings for iptables, the DHCP and the DNS. If we are aiming for an easy to administer setup, we should add a restart script like this:

#!/bin/bash

./set_forwarding.sh
/./etc/init.d/bind9 restart
/./etc/init.d/isc-dhcp/server restart

Now anyone who changes a config file can run this script to have their new rule applied.

Non standard UML: decomposing inheritance in sequence diagrams

TL;DR: In sequence diagrams, separating an object into (some of) its parent classes improves readibility, at the cost of a not so accurate structure description.

A sequence diagram is supposed to display interactions between objects, making enphasis on its sequentiality. The important part here is "between objects"; as far as I know, the standard states that you should display interaction between entities, or the interaction of an entity with itself through a public interface.

Usually this works just fine, since a sequence diagram provides a way of understanding a program through the interaction of its entities, however I found that sometimes you need to express the interaction of an object with his own public interface with a little bit more of detail, namely when there's inheritance involved.

When there's an extension relationship between two objects, and a dependency (i.e. a call) in methods of this two classes, the code for each method usually "lives" far appart, most usually in two different files. Portraying these two methods as a reflexive call is sintactically accurate for a sequence diagram, but I found it very poor semantically. Representing this dependency as a call between two different objects might not be a sintactically correct representation, as you are treating a single entity as two different objects, but I have found it results in much clearer and cleaner diagrams.

Of course spliting an entity into multiple objects has the disadvantage of inducing the reader to believe these are indeed different objects, but since the purpose of a sequence diagram is not to display a static structure I believe this is an acceptable tradeoff, one that can be diminished by just using a note and a reference to a class diagram, where the accurate structure of the classes can be displayed.

Using masqd for custom subdomains

Some time ago I had to work on a project with many, and dynamic, subdomains. Things like sub1.domain, sub2.domain, subN.domain. It turns out /etc/hosts doesn't support wildcards. That sucks, but its understandable, that's not why the hosts file is there; that's the reason we have DNS, isn't it?

Of course installing a DNS server just to resolve a couple of subdomains is much too work for lazy people like me, so it's time to install dnsmasq instead. dnsmasq is a DNS and DHCP proxy, and has a ton of configuration options I don't even know about. If you really want to learn how to use masqd you should read its manpage, if you just want to solve your DNS issues just add "address=/.DOMAIN/IP" to your /etc/dnsmasq.conf (notice the . before the domain. That's the wildcard).

CLI music FTW!

Does your music collection suck? Did you have to delete all your mp3 because you were facing a major lawsuit by the MPAA? Make your own console-music! Just open a terminal and type this for hours of endless fun:

cat /dev/urandom | aplay

Aditional tip: if you ever get tired of the random static, you can have fun playing your boot images like this:

cat /boot/initrd.img-3.0.0-12-generic | aplay

I wonder if that has copyrights....

Checking connectivity for port forwardings

This is a little bit outside of what I normally post, but when I find such a terribly useful site I need to share it (so I won't forget next time I need it).

Have you ever had to set up a forwarding to expose a LAN service to the outside world? It kind of sucks not knowing if you set up everything correctly, and it's not easy to test, since you can only test if the service is working inside your LAN. Normally you would need to either bounce on a proxy outside, or ask a friend to nmap you. Alternatively, you can use canyouseeme.org, a website that will probe a specific port on the IP you specify, and tell you if it times out or if it's open.

A great time saver.

g++ has a (mean) personaility

Did you ever get a message like "undefined reference to `__gxx_personality_v0'"? It means you are trying to link c++ code with a c linker, just change gcc with g++. But what is gxx_personality?

Basically, gxx_personality is a global pointer used for stack unwinding. You could make your code compile (assuming you don't have other problems with vtables and such) by defining it as a NULL ptr, and everything should work until an exception is thrown.

Tailing Google app engine's logs on realtime

Knowing in real time when an error occurs on your GAE app is not as easy as it sounds. You can create a custom error handler and then mail you the exception log, but you have to develop that yourself (and if you do it wrong you may end up slowing down the whole app... yes, I know it first hand). You can also keep the GAE log page open and set your browser to refresh it every X seconds, but that's quite cumbersome.

So, seeing there are a bunch of acceptable but not quite nice solutions out there, I decided to add yet another "solution" that works, but looks ugly. This script will probably break in a few months, or whenever any part of the auth protocol Google uses changes (since I think very little of it is supposed to be public) but until then you can use it to tail GAE's logs on real time.

Note it will work polling a logs webservice @ Google (just like appcfg does); if you set the frequency too high you may see your daily cost increasing but if you set it too low then the script will only get the last error between intervals (so you might miss errors in between). If too many errors go undetected, though, your problem is likely a too-high error rate and not a low update frequency.

#!/usr/bin/python

# Configure your account here
GOOGLE_AUTH = ('foobar@gmail.com', 'gmailpass')
# The app version should be in your GAE control panel (Main> Version)
APP_VERSION = "Your_App_version"
# Your app ID (You can see the ID in the "Application" list on every GAE
# control panel page)
APP_ID = "your app id"
# Frequency of updates; if it's too often, you might get a noticeable increase
# in your cost per day, if it's too sparse you might loose an error in between
# updates (though this probably means your error rate is too high)
UPDATE_FREQ_SECS = 60



import time
from datetime import datetime
import urllib2, urllib
from httplib2 import Http

class Google_Authenticator(object):

    # We use this to keep urllib2 from following redirs
    class _RedirectHandler(urllib2.HTTPRedirectHandler):
        def handler(self, req, fp, code, msg, headers):
            infourl = urllib.addinfourl(fp, headers, req.get_full_url())
            infourl.status = code
            infourl.code = code
            return infourl

        http_error_301 = handler
        http_error_302 = handler
    opener = urllib2.build_opener(_RedirectHandler)
    urllib2.install_opener(opener)


    def __init__(self, email, passwd):
        self.auth = self.__class__._get_auth_token(email, passwd)
        self.cookie = self.__class__._convert_auth_cookie(self.auth)


    @classmethod
    def _get_auth_token(cls, email, passwd):
        GOOG_LOGIN_URL = "https://www.google.com/accounts/ClientLogin"
        data = {
                'Email': email,
                'Passwd': passwd,
                'source': 'Google-appcfg-1.7.2',
                'accountType': 'HOSTED_OR_GOOGLE',
                'service': 'ah',
            }

        try :
            post_data = urllib.urlencode(data)   
            res = urllib2.urlopen(GOOG_LOGIN_URL, post_data)
        except Exception:
            return None

        for ln in res.readlines():
            if ln.startswith('Auth='):
                pos = len('Auth=')
                return ln[pos:].strip()

        return None


    @classmethod
    def _convert_auth_cookie(cls, auth):
        GOOG_COOKIE_URL = "https://appengine.google.com/_ah/login?"\
                          "continue={0}&auth={1}"
        redir = 'http%3A%2F%2Flocalhost%2F' # Anywhere (that's listening)...
        url = GOOG_COOKIE_URL.format(redir, auth)

        try:
            res = urllib2.urlopen(url)
            cookie = res.headers['set-cookie']
        except Exception:
            return None

        pos = cookie.find(' ') - 1
        return cookie[:pos]



class GAE_Logs_Fetcher(object):

    GAE_LOGS_URL = "https://appengine.google.com/api/request_logs?" \
                   "include_vhost=False&version={0}&limit={1}&"\
                   "severity={2}&app_id={3}"

    DEBUG=0
    ERROR=3
    CRITICAL=4

    def __init__(self, auth_cookie, app_id, app_version, limit, severity):
        self.url = self.__class__.GAE_LOGS_URL.\
                        format(app_version, limit, severity, app_id)
        self.opener = urllib2.build_opener()
        self.opener.addheaders = [('cookie', auth_cookie),
                                  ('X-appcfg-api-version', 1)]


    def fetch(self):
        res = self.opener.open(self.url)
        msg = ""
        for ln in res.readlines():
            if not ln.startswith('# next_offset='):
                msg += ln

        return msg


    def watch(self, freq, callback):
        try:
            last_msg = self.fetch()
            while True:
                time.sleep(freq)
                msg = self.fetch()
                if msg != last_msg:
                    last_msg = msg
                    callback(msg)

        except KeyboardInterrupt:
            pass



def main():
    def printmsg(msg):
        print msg

    auth = Google_Authenticator(*GOOGLE_AUTH)
    logs_fetcher = GAE_Logs_Fetcher(auth.cookie, app_version=APP_VERSION,
                                    limit=1, severity=GAE_Logs_Fetcher.ERROR, app_id=APP_ID)

    print "Auth OK, starting watch on {0} error log".format(APP_ID)
    logs_fetcher.watch(UPDATE_FREQ_SECS, printmsg)



if __name__ == '__main__':
    main()

Boot Linux in single user mode

Sooner or later, you'll need a safe boot mode for Linux. Maybe you forgot your password, maybe you need to recover some files of a really really broken system (shame on you for not using a different partition for /home). Luckily in any Linux server you can probably interrupt Lilo or Grub and add to the kernel line the following parameter init="/bin/sh". This will give you a root command shell from which you can do other fun recovery tasks.

Admin tip: Just don't forget to fill up on coffee before starting.