Bash script preamble

All background Bash scripts should start with this preamble:

set -euo pipefail exec > ~/log.log 2>&1

There are countless articles explaining why, and the main purpose of this one is a reminder for myself, so I won't go into the details. For reference:

• -e halts the script on error
• -u errors when using an undefined variable
• -o pipefail makes pipe error return value sane
• exec > ~/log.log 2>&1 redirect all output to ~/log.log

jq: grep and prettify json

If you don't use jq, you are missing a very important utility in your bash toolset. jq let's you query and filter json files from a cli. Just like awk or sed, js's "language" is basically write only, meaning whenever you need to do something there's a 99% chance you'll just be copy-pasting recipes from Stackoverflow until you find the one that works for you. Here are a couple of recipes I found most useful:

cat a json file - with pretty print

jq . /path/to/json_file

Select a single key

jq '.path.to.key'

The command above will return "42" for a json that looks like "{path: {to: {key: 42}}}"

Delete all entries in an object, except for one

jq '.foo|=bar'

The command above will return "{foo: {bar:''}}" for a json that looks like "{foo: {bar:'', baz: ''}}"

This is probably not even enough to get started. Luckily there's plenty of docs to read @ https://stedolan.github.io/jq/manual/

Bash tip: Default value for a variable

In my Bash scripts, I used to hack my way around default values for variables. Turns out there is a very simple way to give your variables a default value while also letting other override them if they want to:

FOO=${BAR-bar}

If someone export's BAR, then FOO will equals the already exported value of $BAR, if $BAR doesn't exist then FOO will have the value of the literal 'bar'.

Deobfuscate your bash

Who hasn't written some read-only magical Bash voodoo, only to find you need to decrypt your own creation later on? Luckily, Explain Shell can help with that. Here's an example from my .bash_history file:

for fn in *; do echo cat $fn | sed "s|' '$URL'||g" | sed "s|curl -X POST -d '||g" ; done

And Explain Shell's explanation: it's no substitute for knowing Bash but it sure helps.

Bonus: while reading my bash history file, I realized I accidentally copy&paste a lot of code to my terminals. There are way too many "template <FOO>" entries in there...

Bonus II: It's a good thing they wrote "shell" in a different color. I was wondering why I had "explains hell" in my bookmarks.

Quickly sharing files in Linux via HTTP

Isn't it awful when you have to share a file too big for email and don't know how? You'd think by 2016 we'd have that figured out. Actually we do, many times over. Just pick a standard that works for you!

If you don't want to read many pages on file transfer standards (Samba? What's that?) you can try this little snippet:

python -m SimpleHTTPServer $PORT

This will create an http server sharing the current directory. HTTP, luckily, is one of those things that tend to work everywhere, always.

Bonus: some other ways of doing the same thing at https://gist.github.com/willurd/5720255

Public service announcement: searching your terminal's output

Short tip today, but a life-changer one: you don't need to copy&paste your terminal's scrollback to search on it, you can do it in place. At least in terminator that's possible (and I hear it's also doable in Gnome's default terminal application). Just press Ctrl+Shift+F. No more copy and pasting to vim!

Bash tip: idiom to get the first error code from a pipe

When writing a bash script, often times you'll end up with something like this:

real_command | filter_stuff | prettify | do_something_else

The problem arises when you try to figure out if your command succeeded or not. If you `echo $?` you'll get the return code for the last chain in the pipe. You don't really care about the output value of do_something_else, do you?

I haven't found a solution I really like to this problem, but this idiom is handy:

out=`real_command` && echo $out | filter_stuff | prettify | do_something_else
echo $?

Now $? will hold the value of real_command, and you can actually use it to diagnose the real problem.

Globing in bash

There is a pretty common and unnecessary pattern used by bash scripts: whenever you need to loop through a list of file names in a path, you might tempted to write something like this.

for fname in $(ls | grep foo); do echo $fname; done

You can save some typing by using bash-globbing:

for fname in *foo*; do echo $fname; done

Not only the script should be cleaner and faster, bash will take care of properly expanding the file names and you won't have to worry about things like filenames with spaces. This should also be portable to other shells too. Want to know more about bash globbibg? Check out http://www.linuxjournal.com/content/bash-extended-globbing

Bash traps: almost like RAII for bash

Everywhere, but specially in bash, cleaning up is annoying and error prone. Resource leaks can be common if your bash script is interrupted half-way. Do you need to execute something always, even if your script fails or gets killed? Try using traps:

#!/bin/bash
foobar() {
    echo "See ya!"
}
trap "foobar" EXIT

It doesn't mater how you end this script, "foobar" will always be executed. Want to read more about bash traps? Check http://linuxcommand.org/wss0160.php

Force a program to output to stdout

Silly but handy CLI trick on Linux: Some programs don't have an option to output to stdout. Gcc comes to mind. In that case the symlink '/dev/stdout' will come in handy: /dev/stdout will be symlinked to stdout for each process.

With this trick you could, for example, run "gcc -S foo.cpp -o /dev/stdout", to get the assembly listing for foo.cpp.

You probably shouldn't use this trick on anything other than CLI scripting stuff (keep in mind /dev/stdout might be closed or not accessible for some processes).

A random slideshow in Ubuntu

The other day I wanted to use my tv for a slideshow of my travel pictures. Something simple, just select a folder and have a program like Shotwell create a slideshow with a random order on my tv. Of course, Ubuntu and double screen equals fail. For some reaason all the programs I tried either were incapable of using the tv as the slideshow screen (even after cloning screens... now that's a wtf) or where not able to recursively use all the pictures in a folder.

feh to the rescue. It's not pretty, but feh seems to be exactly what I was looking for. It's a CLI application for Linux and after some RTFM I came up with this script:

feh ~/Pictures \
     --scale-down \
     --geometry 1920x760 \
     --slideshow-delay 9 \
     --recursive \
     --randomize \
     --auto-zoom \
     --draw-filename \
     --image-bg black

You can probably figure out by yourself what each option means. If not, just man feh.

Counting lines per second with bash

The other day I wanted to quickly monitor the status of a production nginx after applying some iptables rules and changing some VPN stuff. It's easy to know if you completely screwed up the server: the number of requests per second will drop to zero, all requests will have an httpstatus different from 200, or some other dramatic and easy to measure side effect.

What happens if you broke something in a slightly more subtle way? Say, you screwed up something in ipsec (now, I wonder how that can happen...) and now networking is slow. Or iptables now enforces some kind of throttling in a way you didn't expect. To detect this type of errors I wrote a quick bash script to output how many lines per second are added to a file. This way I was able to monitor if the throughput of my nginx install didn't decrease after my config changes, without installing a full fledged solution like zabbix.

I didn't find anything like this readily available, so I'm posting it here in case someone else finds it useful.

#!/bin/bash

# Time between checks
T=5

# argv[1] will be the file to check
LOG_FILE=$1

while true; do
    tmp=`mktemp`
    # tail a file into a temp. -n0 means don't output anything at the start so
    # we can sleep $T seconds and we don't need to worry about previous entries
    tail -n0 -f $LOG_FILE > $tmp 2>/dev/null & sleep $T;
    kill $! > /dev/null 2>&1;
    echo "Requests in $LOG_FILE in the last $T seconds: `cat $tmp | wc -l`";
    rm $tmp;
done

Bash scripting and getops

Did you ever write a bash script and thought it looked too clean? Yeah, me neither. Anyway, now you can make it look even worse by using getopt. As an upside, you'll be able to read command line options from a user without having to resort to nasty hacks, like hardcoding the switch position into the argv.

getopt should be installed by default in most Linux distros, and you can even run it as a command line program. It's quite easy to use on a bashcript. For example, something like:

while getopts "bar" opt; do
    case "$opt" in
        b) echo "Option b is set"
           ;;
        a) echo "Option a is set"
           ;;
        r) echo "Option r is set"
           ;;
    esac
done

It won't look pretty but it does get the job done. According to "man getopt" it supports things like short & long options and defaults; if you need something more complex, you should probably be using a proper language instead of a bash script.

Vim tips: my github's vimrc

It's been a while since I posted my Vimrc. That's probably because I don't really need to back it up in my blog anymore, now I just keep it in my github repo.

Now whenever I get a new computer I just clone my repo and get for free my vimrc, my terminator rc and a bunch of useful scripts I regularly use on bash.

Monitor file changes on a CLI

The other day I had a problem with a config file being overwritten. Some process, I did not know which one, was overwriting a configuration file I manually changed. Annoyed by this, I started looking for the culprit. lsof was no good, because that would list the open files; this process would most likely just open the file, write to it and then close it again. My chances of ever catching this process in the act were nil. Luckily I found auditd. Install it like this:

sudo apt-get install auditd

Then to monitor a file you can use the following command:

sudo auditctl -w $FILE -p war

Wait until $FILE has changed, then execute this command to get the results:

ausearch -f $FILE

Voila, now you have your culprit. Kill -9 at will.

Quick C/C++ reference on the console

What do you usually do when you don't remember whether fp is the first argument for fprintf or for fwrite? (Hint: it's the first argument for one, the last for the other. Handy, huh?). Well, there's no need to go and check the answer on Google, just sudo apt-get install manpages-dev and then do "man fprintf" or "man fwrite".

Note: you might also want posix-dev to check stuff like htons.

Printing broken in Ubuntu? Try lpr

Lately I've seen quite a few Ubuntu machines with broken printing (more broken than usual, that is). Sometimes printers are detected, sometimes not. When it doesn't work you usually get a warning saying "Getting printer information failed" even though the printer is connected and you can even ping it. Searching around a little bit I read a couple of threads and bug reports that make me think that it's actually a Gnome 3 migration problem, but I'm not sure. It doesn't matter much anyway: screw UIs, just use lpr < foo.pdf from a CLI and keep on killing forests!

On a cheerful closing note: how funny is it that when the regular printing applet in Gnome is broken we have to use a utility by "Apple Inc"?

CLI music FTW!

Does your music collection suck? Did you have to delete all your mp3 because you were facing a major lawsuit by the MPAA? Make your own console-music! Just open a terminal and type this for hours of endless fun:

cat /dev/urandom | aplay

Aditional tip: if you ever get tired of the random static, you can have fun playing your boot images like this:

cat /boot/initrd.img-3.0.0-12-generic | aplay

I wonder if that has copyrights....

Starting Ubuntu in console mode

Like it or not, Ubuntu is so easy to install that even for servers is very comfortable to just mount the iso and create as many virtual machines as you may need. Sometimes you already have an iso for Ubuntu, but are too lazy to download the server version. For those occasions you can either decide to waste precious RAM running a GUI for a server that will never need it, or you can remove all traces of the graphical login. Like this:

sudo update-rc.d -f gdm remove

This will remove GDM from the startup scripts, meaning you can still fire up the graphical interface (*) if you want, but it will start Ubuntu without loading any graphics stuff. This is very useful to save on RAM, startup time and processing power, which even if not that useful for a desktop machine is incredible beneficial when you have several virtual machines running in a single physical server.

(*) More precisely, if you have users that need it. Remember though, if it can't be done in console mode, it ain't worth doing.

Running commands on Windows from Linux, through ssh

Running Windows is something I don't usually like (running of Windows is a different story) but having to run something on Windows command line interface is something I wouldn't wish even to my worst enemies. I was stuck in that situation, don't remember why, but I needed to run a command in a Windows machine, automatically, and I only had ssh (is there a better way of automating scripted tasks in Windows, remotely and without a GUI?). Well, this is what I came up with:

ssh host cmd /c dir

Running that in a bash shell will show the directory listing of C: in machine "host". Ugly as hell, but it's a good way of kickstarting a batch script.