Wednesday, 5 June 2013

Six useful things I have learnt at Software Carpentry boot camp

Monday and Tuesday this week, I attended a Software Carpentry boot camp at the University of Southampton. Topics covered included bash, shell scripting, Git and writing/testing reproducible code. The event was well-attended and whilst making an event like this relevant to everyone is obviously a challenge, they did a great job. I certainly learnt some useful tips even for the topics that I knew a fair bit about.

As well as the SWC presentations themselves, it was good to get tips from fellow programmers as well as "UNIX and Perl to the Rescue!: A Field Guide for the Life Sciences (and Other Data-rich Pursuits)", which I brought along for some browsing. Here is half a dozen random things that I picked up over the two days...

1. Useful grep flags. I've used grep to pull out matches from text files a fair amount and was familiar with a few of the flags that I use a lot:
-i = case-insensitive search
-v = inverse search
-A n = return n lines after match
-B n = return n lines before match
I learnt a few more useful ones, though:
-w = match whole words only
-n = show line numbers
--color = highlight matches in colour
-r = search subdirectories (recursive)
2. Useful ls flags. ls is another must-have part of the UNIX users toolkit. There were still a few useful flags with which I was unfamiliar and am likely to use in future, though:
-G = colour-code directory contents
-F = appends / to directories
-R = recursive ls including subdirectory contents
-1 = one entry per line
-a = show all files including hidden files
Along with my old favourites, of course:
-l = "long" mode (more info)
-r = reverse sort
-t = sort by time
-S = sort by size
-h = user-friendly file sizes
3. Catching the standard error and standard output. I've seen this a few times but for some reason this was the first time it really sunk in. Most UNIX users will be familiar with redirecting the standard output from a command into a file using > file (or >> to append). Catching the standard error is less obvious/common. This can be done using 2> to catch the stderr alone, or &> to catch both stdout and stderr at the same time. If you want to redirect them both into different files, do something like this:
[cmd] 2> error.txt > stdout.txt
The stdout will slip by the first redirect and then get caught by the second. Of course, you can have those the other way round if you wish!

4. Navigating the terminal. This one actually came from "UNIX and Perl to the Rescue" but I discovered it at the boot camp:
Ctrl+a = move to start of line
Ctrl+e = move to end of line
Ctrl+w = delete previous word
Ctrl+l = clear screen (clear works too)
Ctrl+r = search through previous commands one letter at a time (this was an SWC revelation)
Using Ctrl+a and Ctrl+e seems to work in a number of other Mac editors too - including this blogger HTML window! If you are a Mac user, just make sure that you don't use cmd+w by mistake - this will close the current terminal window with much wailing and gnashing of teeth!

5. Reversing Python strings. In the past, I have reversed a python string by converting into a list, using the list.reverse() method and then string.join() to convert it back. But there is a better way!
I've used string slicing loads before but never realised that you could add a step to this syntax as with the range() method! (string[::-1] translates as string[start=end:end=start:step=-1].) Now I am wondering where else in my code I can use this knowledge!

6. Changing the command prompt to $. Sometimes, the command prompt is too long (or the terminal window too narrow) so that almost every command wraps around in an annoying fashion. To replace with a simple $ character, just type:
PS1="\$ "

I must admit that I do not (yet) really understand exactly what this does and how it works but it does! (And if you now get lost as to where you are, just remember that pwd gives you the full path to the working directory.

There were more but I think that's enough for now! If this is the sort of thing that floats your boat, you can find useful stuff like this plus a whole bunch of lessons at the Software Carpentry website:

If you get the chance to attend a boot camp yourself, my advice is: do!


  1. Nice collection of the many things we learned during the past two days, some of those were also new for me and are really useful. It seems I usually stick to the reduced set of options I have readily available in mind, making strange combinations to make it work, but there were a couple of tips there that I have immediately and effortlessly internalized (the shortcut is my usual friend to search bash history but I've been doing 'history' followed by '!number_from_history' since the Monday session).

  2. Ctrl-U and Ctrl-K for editing (cut from start to cursor, or cursor to end) and Ctrl-R for recursive search through the history are ones I use a lot.

    1. Thanks for those extra tips. That's "cut" as in "delete" rather than "cut to clipboard", right?

  3. Hi Richard,
    Glad you found it useful --- you might find Haddock and Dunn's "Practical Computing for Biologists" useful as well ( And if you're interested in helping us teach future boot camps, please give us a shout (
    Thanks again,
    Greg Wilson
    Mozilla Foundation / Software Carpentry

    1. I think one of my students has that book. Looks useful. When I move to Oz later this year, I am probably going to make the Mac move and might invest in a copy. Could be interested in helping run a boot camp Down Under!

  4. Recommendation: Don't go down the PERL rabbit hole.... That train has left the station!

    1. Don't worry, I cut my teeth on Perl but I'm a Python man through and through now!

  5. Thanks for sharing a six useful things and shortcut keys about camp software.


Thanks for leaving a comment! (Unless you're a spammer, in which case please stop - I am only going to delete it. You are just wasting your time and mine.)