Linux refresher

Linux refresher

This is more a cram-session on Linux, like learning to drive if you had to drive your younger sister to the hospital when you're 10 because you just stuck a knife in her leg accidentally and didn't want your parents to know.

All the Exercises listed below will work on Lonestar.

Getting to a remote computer

ssh is an executable program that runs on your local computer. It connects securely (i.e. encrypted) to a remote computer that you specify. ssh programs exist for all major operating systems - Windows, Mac, and linux. Mac and linux come with these commands built-in; Windows needs some help. If you're using a Windows box and are part of UT Austin, Bevoware provides two free ssh programs, "ssh secure client" and "putty". We won't describe how to use these here.

If you're on a linux box or a mac, open a terminal window and log in to lonestar with your TACC login credentials using ssh:

ssh <your user ID>@lonestar.tacc.utexas.edu

When you log in to a linux computer, the operating system checks your login credentials and if they're OK it sets up some configuration for you and then runs a program called a "shell" which acts like your fast-food drive-thru window to the rest of the operating system. You type commands and hit "enter" to send something into the drive-thru window, and then the OS passes output back through the drive-thru window.

Every time you exchange stuff through this window, it's within a context, like one specific drive-thru window at one restaurant. The directory within the file system is one part of that context; the programs and environment variables available to you are other parts of that context. When you log in, the system and shell agree that you'll start off in your home directory on the system.

Essential command-line tricks to look like an expert quickly, or figure out what's going on.

Hit the tab key twice - it's almost always magic. This instructs the shell to try to guess what you're doing and finish the typing for you. On most modern linux shells, it works for commands (like "ls" or "scp") and for completing file or directory names.

This is really useful if you can't remember whether fasta2fastq.sh is fastaToFastq or fastaToFastq.sh or Fasta2fastq or Fasta2Fastq.sh or something else. It's also helpful for reconstructing directory paths or filenames on-the-fly.

Exercise

Type "modu" then hit tab twice - it presents two choices, module and modutil. Type the next character l, hit tab twice and it will complete the rest of the typing. If you hit tab twice again, the OS will show you all the files in your current working directory which doesn't make any sense for the command "module" - it's smart, but not smart enough to figure out that the next word in the command needs to be one of module's built-in commands.

Man pages - linux had built-in help files somewhere around the 1500's, way before Macs or PCs thought of such things. In linux they're called man pages - short for "manual"; it's not a gender thing (I assume).

Exercise:

Try "man grep", or "man du", or "man sort" - you'll want these sometime.

Basic linux commands you need to know like breathing air

• ls - list the contents of the current directory
• pwd - print the present working directory - which restaurant am I at right now - the format is something like /home/myID - just like on most computer systems, this represents leaves on the tree of the file system structure, also called a "path".
• cd <whereto> - change the present working directory to <whereto> - pick up my drive-thru window (shell) and move it so that I'm now looking thru to the
  directory <whereto>
  • Some special <wheretos>: .. (period, period) means "up one level". ~ (tilde) means "my home directory". ~myfriend (tilde "myfriend) means "myfriend's home directory".
    df shows you the top level of the directory structure of the system you're working on, along with how much disk space is available
    head <file> and tail <file> shows you the top or bottom 10 lines of a file <file>
more <file> and less <file> both display the contents of <file> in nice ways.
    file <file> tells you what kind of file <file> is.
    cat <file> outputs all the contents of <file> - CAUTION - only use on small files.

TODO: rm, cp, mkdir, rmdir, scp
Exercise: scavenger hunt

Exercise:

Use variables to store where you are, move away, and then back. Try this and see if you can figure out what the shell is doing for you:

pwd
here=`pwd`
cd /scratch/01057
pwd
cd $here
pwd

Options: the lifeblood of linux commands

Sitting at the computer, you should have some idea what you need to do. There's probably a command to do it. If you have some idea what it starts with, you can type a few characters and hit tab twice to get some help. If you have no idea, you google it or ask someone else. But soon you want those commands to do a bit more - like seeing the sizes of files in addition to their names.

Most commands in linux use a common syntax to ask more of a command; they usually add a dash "-" followed by a code letter that means "do the basic command, but with a bit more..."

Try these - they're useful:

ls -l
ls -lh
ls -t

These little toggle-like things are often called "command line switches"; there can be other options, like filenames, that aren't switches.

Almost all commands, and especially NGS tools, use options heavily.

Like dialects in a language, there are at least three basic schemes commands/programs accept options in:

1. One letter options which can sometimes be combined, or other single options like:

   head -10
   ls -lhtS (equivalent to ls -l -h -t -S)
   

2. Word options, like -d64 and -Xms512m in this command, that are never combined (this is the GATK command to call SNPs):

   java -d64 -Xms512m -Xmx4g -jar /work/01866/phr254/gshare/Tools_And_Programs/bin/GenomeAnalysisTK.jar -glm BOTH -R $reference -T UnifiedGenotyper -I
   $outprefix.realigned.recal.bam --dbsnp $dbsnp -o $outprefix.snps.vcf -metrics snps.metrics -stand_call_conf 50.0 -stand_emit_conf 10.0 -dcov 1000
   -A DepthOfCoverage -A AlleleBalance
   

3. "Long option" forms, using the convention that a single dash - precedes single-letter options, and double dashes- - precede word options, like this command to run the mira assembler:

   mira --project=ct --job=denovo,genome,accurate,454 -SK:not=8
   

man pages should detail all options available for a command.

More help please

Sometimes man lets you down - no man page. Don't fret, try one of these:

1. Just type in the command and hit return - it will usually try to help you.
2. Type the command followed by one of: -h -help --help -? and may give you some help.
   Sometimes the command by itself will give you short help, and will list the magic option for full help.

Exercise

First do:

module load blast

Now figure out how to run some kind of blast program on lonestar with options. Hints: try <tab><tab>, man, running some blast command, use options to figure out other options.

Piping and redirection

It's a pain to have to order for your kids at the drive-thru; sometimes you'd like them to order directly and have the food go directly to them instead of through you. In a linux shell, this is called redirection. It uses a familiar metaphor: "pipes".

The linux operating system expects some "standard input pipe" and gives output back through a "standard output pipe". These are called "stdin" and "stdout" in linux. There's also a special "stderr" for errors; we'll ignore that for now.

Usually, your shell is filling the operating system's stdin with stuff you type - the commands with options. The shell passes responses back from those commands to stdout, which the shell usually dumps to your screen.

The ability to switch stdin and stdout around is one of the key reasons linux has existed for decades and beat out many other operating systems.

The syntax for doing this switching around can be confusing because it uses codes.

Exercise

Redirect stdout of the ls -1 command to the file whatsHere

ls -1 > whatsHere
cat whatsHere

Redirect stdout of ls -1 to the head -2 command, then to the file whatsHere

ls -1 | head -2 > whatsHere
cat whatsHere

So: the redirect output (stdout) character is >, and the "pass output along as input" is the pipe character |.

Redirect stdout of ls -1 to the head -2 command, then to the file whatsHere, then use that file to list the sizes of those two files

ls -1 | head -2 > whatsHere
cat whatsHere
ls -l `cat whatsHere`

(You may have to copy and paste that last command - it uses the backtick character ` to tell the shell to execute the command cat whatsHere and then use the result as an option to the ls -l command).

Perfect - now my kids (i.e. other commands) can order (stdin) through the same drive-thru window and get their kid-pack (stdout) directly!

Not all shells are equal - the bash shell lets you redirect stdout with either > or 1>; stderr can be redirected with 2>; you can redirect both stdout and stderr using &>. If these don't work, use google to try to figure it out. The web site stackoverflow is a usually trustworthy and well annotated site for OS and shell help.

A few more bits you need for this class

wget <url> fetches a file with a valid URL. We'll use this to pull data from one of TACC's web-based storage devices.

Confused about commands vs. programs?

A command is something entered to stdin that the OS understands how to run. That command might be a list of other commands, a command built-in to the linux operating system, or an executable program (sometimes called a binary).

Command is the general term, but, for example, when we run samtools we're entering the command samtools to run the executable binary program samtools.

The operating system must have previously been told where the actual executable file called samtools exists in the filesystem, so when a user enters the command samtools it knows where to find the executable samtools and run it.