Google Test: Quarantine for tests?

Google Test: Putting a test under quarantine

GTest works wonders for c++ testing, even more so when combined with GMock. I've been using these frameworks for a few side projects. I've seen them used in large scale projects too. In all cases, there is a very common problem for which (I think) there is no elegant solution: managing temporarily disabled tests.

It may be because you found a flaky piece of code or a test that exposes a heisenbug. Maybe the test itself is just unstable, or perhaps you are using TDD and want to submit a test to your CI before its implementation is ready. In these cases, you can choose to disable the offending test or let it run, possible halting your CI because of it. When that happens, you maybe masking other, real, problems.

Most people would stick a "DISABLED_" before the test name, to let GTest know not to run it. Maybe even stick a "// TODO: reenable" in there too. When run, GTest will generate a message to let you know there is a disabled test. Even so, I find that people -myself included- tend to forget to re-enable the disabled tests.

For one of my side projects, I hacked GTest to quarantine tests up to a certain date:

TEST(Foo, Bar) {
    QUARANTINE_UNTIL("16/8/22");
    EXPECT_EQ(1,2);
}

In my CI setup, that test will be showing a happy green (and a warning, which I will probably ignore) until the 22nd of August. By the 23rd the test will run again and fail if I haven't fixed the code. If I have indeed fixed it, it will print a warning to remind me that it's safe to delete the quarantine statement.

Is there any advantage in this approach over the usual _DISABLE strategy? In my opinion, there is: if you ignore warnings in your test, for whatever reason, a _DISABLE might go unnoticed and it may hide a real problem. In the same scenario, for a quarantined test, nothing bad happens: the warning just says "you should delete this line" but the quarantined test is again part of your safety net.

How does it work? The first caveat in my article mentions it: hackishly. There are a few facilities missing in GTest to make this implementation production-ready but, ugly as it looks, it should work as intended:

#include <ctime>
#include <string>
#include <sstream>
std::string now() {
    time_t t = time(0);
    struct tm *now = localtime(&t);

    std::stringstream formatted_date;
    formatted_date << (now->tm_year+1900) << '/'
                   << (now->tm_mon+1) << '/'
                   << now->tm_mday;

    return formatted_date.str();
}

#define QUARANTINE_UNTIL(date_limit)                                     \
        if (now() < date_limit) {                                        \
            GTEST_LOG_(WARNING) << "Test under quarantine!";             \
            return;                                                      \
        } else {                                                         \
            GTEST_LOG_(WARNING) << "Quarantine expired on " date_limit;  \
        }

If I find there is interest in this approach for real world applications, I may try to come up with a nicer interface for it.

Vimcrypt

Have you ever been working on your plans for world domination but got scared someone else might find them? It happens to me all the time. Or maybe you are so paranoid that you need to encrypt your grocery list. Perhaps you are sharing a semi-private text file through a public service? Good news, Vim has you covered. Just type ":X". Vim will ask you for a password. Save your file again and voila, your file is now encrypted. Open the same file with Vim to decrypt it. Your plans for world domination are now safe!

Things you should never do

I think I may start a new series in my "Rants" category: things you shouldn't do. First one: Never ever use "strange" characters on your wifi's AP name, where strange is defined as non-ascii. I made the huge mistake of choosing a name with an ñ on it, then had to spend an entire evening hacking on a printer driver with no unicode support. No, I couldn't have changed the AP's name. That would have required me to physically connect a computer to my router, and I was too lazy to get up from the couch.

Self reminder: setting the default boot option in UEFI

Bought a new laptop (*) and I'm 100% sure I'll forget this if I don't put it here:

From http://askubuntu.com/questions/291905/how-can-i-make-ubuntu-the-default-boot-option-on-a-newer-laptop-uefi

To set Ubuntu as the default boot OS in a multi-OS setup (ie, dual boot Windows) with UEFI, goto Windows and exec (as admin) bcdedit /set {bootmgr} path \EFI\ubuntu\grubx64.efi

Why am I using Windows, you may ask? I'm still in the process of discovering which features will be broken and which hardware will work out of the box. So far I'm actually quite surprised, with only the video card and the touchpad not working. Luckily bash doesn't use either of those. Who needs a mouse anyway?

Simple vim plugin IV: project greping

I recently wrote about some of the utilities I created for my Vim setup. Using someone else's Vim scripts is not nearly as fun as writing your own, so I decided to also write a short summary on what it takes to get started writting Vim plugins. For this task, I decided to start with greping.

Greping can be improved a bit: if you do it a lot in a project, you might find it's useful to also grep the results themselves, to further refine your search. If you have your grep results in vim itself, that is trivial.

Let's start hacking something in our .vimrc file. Try this:

function! FG_Search()
    let needle = input("Search for: ")
    tabnew
    setlocal buftype=nofile bufhidden=wipe nobuflisted noswapfile nowrap
    let grepbin = 'grep -nri '
    let cmd = grepbin . ' "' . needle . '" *'
    execute '$read !' . cmd
    setlocal nomodifiable
endfunction

map <leader>s :call FG_Search()<CR>

This function should be pretty clear: it will map <leader>s (in my case, ",s") to FG_Search(). FG_Search will prompt the user for a term to grep, then search for it executing the command. In the end the results are written to a new tab, which is declared as a temp non-modifiable buffer.

Just paste that in your .vimrc and you're good to grep.

Extra tip: integrate this with my fast grep cache and you have a nice and quick project search integration for vim that works even for very large projects with tools available in most default Linux installs.