Linux fundamentals

Basic linux commands you need to know

Here's a copy of the cheat sheet we passed out.

File system navigation

• ls - list the contents of the current directory
• pwd - print the present working directory - which restaurant am I at right now - the format is something like /home/myID - just like on most computer systems, this represents leaves on the tree of the file system structure, also called a "path".
• cd <whereto> - change the present working directory to <whereto> - pick up my drive-thru window (shell) and move it so that I'm now looking thru to the directory <whereto>
  • Some special <wheretos>: .. (period, period) means "up one level". ~ (tilde) means "my home directory". ~myfriend (tilde "myfriend) means "myfriend's home directory".
• file <file> tells you what kind of file <file> is.
• mkdir <dirname> and rmdir <dirname> make and remove the directory "dirname". This only removes empty directories - "rm -r <dirname>" will remove everything.
• rm <file> deletes a file. This is permanent - not a "trash can" deletion.

Displaying file contents

• head <file> and tail <file> shows you the top or bottom 10 lines of a file <file>
• more <file> and less <file> both display the contents of <file> in nice ways. Read the bit above about man to figure out how to navigate and search when using less
• cat <file> outputs all the contents of <file> - CAUTION - only use on small files.

Copying files and directories

• cp <source> <destination> copies the file source to the location and/or file name destination}. Using . (period) means "here, with the same name". * cp -r <dirname> <destination> will recursively copy the directory dirname and all its contents to the directory destination.
• scp <user>@<host>:<source> <destination> works just like cp but copies source from the user user's directory on remote machine host to the local file destination
• wget <url> fetches a file from a valid URL.

Miscellaneous commands

df shows you the top level of the directory structure of the system you're working on, along with how much disk space is available

Getting around in the shell

Important keyboard shortcuts

Type as little and as accurately as possible by using keyboard shortcuts!

Tab key

The tab key is your best friend. Hit the tab key once or twice - it's almost always magic! This instructs the shell to try to guess what you're doing and finish the typing for you. On most modern Linux shells, Tab "completion" will:

• completing file or directory names up to any ambiguous part
• it works for commands (like "ls" or "scp") and for completing file or directory names.

Up arrow

Use "up arrow" to retrieve any of the last 500 commands you've typed. You can then edit them and hit enter (even in the middle of the command) and the shell will use that command.

Ctrl-a, Ctrl-e

You can use control-a (holding down the "control" key and "a") to jump the cursor right to the beginning of the line. The omega to that alpha is control-e, which jumps the cursor to the end of the line. Arrow keys work, and control-arrow will skip by word forward and backward.

Wildcards and special file names

The shell has shorthand to refer to groups of files by allowing wildcards in file names. * (asterisk) is the most common; it is a wildcard meaning "any length of any characters". Other useful ones are [] to allow for any character in the set <characters>> and {{[] for a range of characters.

For example: ls *.bam lists all files in the current directory that end in .bam; ls [A-Za-z]*.bam does the same, but only if the first character of the file is a capital letter.

Three special file names:

1. . (single period) means "this directory".
2. .. (two periods) means "directory above current." So ls -l .. means "list contents of the parent directory."
3. ~ (tilde) means "my home directory".

Environment variables

Environment variables are just like variables in a programming language (in fact bash is a complete programming language), they are "pointers" that reference data assigned to them. In bash, you assign an environment variable like this:

export varname="Some value, here it's a string"

You then refer to the environment variable with its name, preceded by a dollar sign, like this:

echo $varname

In your .profile_user we set a couple of environment variables to

Getting help

Man pages - linux has had built-in help files since the mid-1500's, way before Macs or PCs thought of such things. In linux they're called man pages - short for "manual"; it's not a gender thing (I assume). man intro will give you an introduction to all user commands.

Exercise:

Try "man grep", or "man du", or "man sort" - you'll want these sometime.

Tip: Type the letter q to quit man, j and k/<CR> to move up and down by line, b or spacebar up/down by page. Want to search? Just hit the slash key /, enter the search word and hit enter. These are actually the tools of the less command which man is using.

Exercises:

Scavenger hunt practice; on Lonestar issue the following commands:

cp -r /corral-repl/utexas/BioITeam/linuxpractice .
cd linuxpractice
cd what
cat readme

and follow the instructions. Hints: use <tab><tab> to fill in filenames as much as you can.

pwd
here=`pwd`
cd /scratch/01057
pwd
cd $here
pwd

Command options

Sitting at the computer, you should have some idea what you need to do. There's probably a command to do it. If you have some idea what it starts with, you can type a few characters and hit tab twice to get some help. If you have no idea, you google it or ask someone else. But soon you want those commands to do a bit more - like seeing the sizes of files in addition to their names.

Most commands in linux use a common syntax to ask more of a command; they usually add a dash "-" followed by a code letter that means "do the basic command, but with a bit more..."

ls -l
ls -lh
ls -t

These little toggle-like things are often called "command line switches"; there can be other options, like filenames, that aren't switches.

Almost all commands, and especially NGS tools, use options heavily.

Like dialects in a language, there are at least three basic schemes commands/programs accept options in:

1. One letter options which can sometimes be combined, or other single options like:

   Examples of different option types

   head -10
   ls -lhtS (equivalent to ls -l -h -t -S)
   

2. Word options, like -d64 and -Xms512m in this command, that are never combined (this is the GATK command to call SNPs):

   Examples of word options

   java -d64 -Xms512m -Xmx4g -jar /work/01866/phr254/gshare/Tools_And_Programs/bin/GenomeAnalysisTK.jar -glm BOTH -R $reference -T UnifiedGenotyper -I
   $outprefix.realigned.recal.bam --dbsnp $dbsnp -o $outprefix.snps.vcf -metrics snps.metrics -stand_call_conf 50.0 -stand_emit_conf 10.0 -dcov 1000
   -A DepthOfCoverage -A AlleleBalance
   

3. "Long option" forms, using the convention that a single dash - precedes single-letter options, and double dashes- - precede word options, like this command to run the mira assembler:

   Example of long options

   mira --project=ct --job=denovo,genome,accurate,454 -SK:not=8
   

man pages should detail all options available for a command. Unless there's no man page.

More help please

Sometimes man lets you down - no man page. Don't fret, try one of these:

1. Just type in the command and hit return - it will usually try to help you.
2. Type the command followed by one of: -h-help--help-? and may give you some help.
   Sometimes the command by itself will give you short help, and will list the magic option for full help.

.profile_user

Let's take a quick look at what is being done by the .profile_user login script, using cat (concatenate):

cd
cat .profile_user

#!/bin/bash
# Change the command line prompt to contain the current directory name
PS1='stamp:\w$ '

# Ensure all created files can be read/written by group members
umask 002

# Use yellow for directories, not that horrible blue
dircolors .dircolors > /dev/null

# Make common useful software available
module load python
module load launcher

# Environment variables for useful locations
export BI=/corral-repl/utexas/BioITeam
export CLASSDIR="$BI/core_ngs_tools"

# Add current directory and $HOME/local/bin to PATH
export PATH=.:$HOME/local/bin:$PATH

Now that you know a bit more about Linux commands, can you tell what some of these directives do?