Page History

...

Overview

Image RemovedImage Added

After raw sequence files are generated (in FASTQ format), quality-checked, and pre-processed in some way, the next step in many NGS pipelines is mapping to a reference genome.

...

Even though many mapping tools exist, a few individual programs have a dominant "market share" of the NGS world. In this section, we will primarily focus on two of the most versatile general-purpose ones: BWA and Bowtie2 (the latter being part of the Tuxedo suite which includes the transcriptome-aware RNA-seq aligner Tophat2 as well as other downstream quantifiaction quantification tools).

Stage the alignment data

...

idev -m 180 -N 1 -A OTH21164 -r CoreNGSday4CoreNGS-Thu

Then stage the sample datasets and references we will use.

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa     $SCRATCH/core_ngs/references/fasta/

# Copy the FASTQ files that will be used for alignment
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/
cd $SCRATCH/core_ngs/alignment/fastq

...

Reference Genomes

...

Here are the four reference genomes we will be using today, with some information about them. These are not necessarily the most recent versions of these references (e.g. the newest human reference genome is hg38 and the most a recent miRBase annotation is  version is v21. (See here for information about many more genomes.)

...

We've discovered a pattern (also known as a regular expression) to use in searching, and the command line tool that does regular expression matching is grep (general regular expression parser). (Read more about grep here: Advanced commands: grep.and regular expressions)

Regular expressions are so powerful that nearly every modern computer language includes a "regex" module of some sort. There are many online tutorials for regular expressions, and several slightly different "flavors" of them. But the most common is the Perl style (http://perldoc.perl.org/perlretut.html), which was one of the fist and still the most powerful (there's a reason Perl was used extensively when assembling the human genome). We're only going to use simple regular expressions here, but learning more about them will pay handsome dividends for you in the future.

...

# If you haven't stagedStage the fastaFASTA files
cds
mkdir -p core_ngs/references/fasta
cd core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa .

cd $SCRATCH/core_ngs/references/fasta
grep -P '^>' sacCer3.fa | more

...

• The -P option tells grep to Perl-style regular expression patterns. 
  
  • This makes including special characters like Tab ( \t  ), Carriage Return carriage return ( \r ) or Linefeed linefeed ( \n ) much easier that the default POSIX paterns.
  • While it is not required here, it generally doesn't hurt to include this option.

• '^>' is the regular expression describing the pattern we're looking for (described below)

• sacCer3.fa is the file to search. 
  
  • lines with text that match our pattern will be written to standard output
  • non matching lines will be omitted
• We pipe to more just in case there are a lot of contig names.

Now down to the nuts and bolts of the pattern: '^>'

First, the single quotes around the pattern – this tells the bash shell to pass the exact string contents to grep.

As part of its friendly we have seen, during command line parsing and evaluation , the shell will often look for special characters metacharacters on the command line that mean something to it (for example, the the $ in front of an environment variable name, like in $SCRATCH). Well, regular expressions treat the $ specially too – but in a completely different way! Those Those single quotes tell tell the shell "don't look inside here for special characters – treat this as a literal string and pass it to the program". The shell will obey, will strip the single quotes off the string, and will pass the actual pattern,   ^>, to the grep program. (Note that the shell does look inside double quotes ( " ) for certain special signals, such as looking for environment variable names to evaluate. Read more about  program. (Read more about Quoting in the shell.)

So what does ^> mean to grep? We know that contig name lines always start with a > character, so > is a literal for grep to use in its pattern match.

We might be able to get away with just using this literal alone as our regex, specifying '>' as as the command line pattern argument. But for for grep, the more specific the pattern, the better. So we constrain where the > can appear on the line. The special carat ( ^ )  metacharacter represents "beginning of line". So ^> means "beginning of a line followed by a > character".

Exercise: How many contigs are there in the sacCer3 reference?

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

Exercise: How many contigs are there in the sacCer3 reference?

cd $SCRATCH/core_ngs/references/fasta
grep -P '^>' sacCer3.fa | wc -l

grep -P -c '^>' sacCer3.fa

...

1. Trim the FASTQ sequences down to 50 with fastx_clipper
   • this removes most of any 5' adapter contamination without the fuss of specific adapter trimming w/cutadapt
2. Prepare the sacCer3 reference index for bwa using bwa index
   
   • this is done once, and re-used for later alignments
3. Perform a global bwa alignment on the R1 reads (bwa aln) producing a BWA-specific binary .sai intermediate file
4. Perform a global bwa alignment on the R2 reads (bwa aln) producing a BWA-specific binary .sai intermediate file
5. Perform pairing of the separately aligned reads and report the alignments in SAM format using bwa sampe
6. Convert the SAM file to a BAM file (samtools view)
7. Sort the BAM file by genomic location (samtools sort)
8. Index the BAM file (samtools index)
9. Gather simple alignment statistics (samtools flagstat and samtools idxstatidxstats)

We're going to skip the trimming step for now and see how it goes. We'll perform steps 2 - 5 now and leave samtools for a later exercise since steps 6 - 10 are common to nearly all post-alignment workflows.

...

Like other tools you've worked with so far, you first need to load bwa. Do that now, and then enter bwa with no arguments to view the top-level help page (many NGS tools will provide some help when called with no arguments). bwa is available as a BioContainers. module.

...

Make sure you're in

...

an idev session

idev -m 

180 -N 1 -A OTH21164 -r 

CoreNGS-Thu
# or
idev -m 90 -N 1 -A OTH21164 -p development

module 

module load biocontainers  # takes a while
module load bwa
bwa

...

mkdir -p $SCRATCH/core_ngs/alignment/fastq
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/alignment/*fastq.gz   $SCRATCH/core_ngs/alignment/fastq/
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

...

mkdir -p $SCRATCH/core_ngs/references/bwa/sacCer3
cd $SCRATCH/core_ngs/references/bwa/sacCer3
ln -ssf ../../fasta/sacCer3.fa
ls -l

...

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

# Copy a pre-built references bwa index for sacCer3
mkdir -p $SCRATCH/core_ngs/references/bwa/sacCer3
cp $CORENGS/references/bwa/sacCer3/*.fa* $SCRATCH/core_ngs/references/bwa/fastasacCer3/

# Get the FASTQ to align
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

...

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -s -f .sf ../fastq
ln -s -fsf ../../references/bwa/sacCer3

...

bwa aln

Required arguments are a <prefix> of the bwa index files, and the input FASTQ file. There are lots of options, but here is a summary of the most important ones.

...

# If not already loaded:
module load biocontainers
module load bwa

cd $SCRATCH/core_ngs/alignment/yeast_bwa
bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R1.cat.fastq.gz > yeast_pe_R1.sai
bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pe_R2.sai

When all is done you should have two .sai files: yeast_pe_R1.sai and yeast_pe_R2.sai.

...

bwa aln -t 60 sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pe_R2.sai

Exercise: How much of a speedup did you seen when aligning the R2 file with 20 60 threads?

[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
[bwa_aln_core] calculate SA coordinate... 266.70 sec
[bwa_aln_core] write to the disk... 0.04 sec
[bwa_aln_core] 262144 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 268.94 sec
[bwa_aln_core] write to the disk... 0.03 sec
[bwa_aln_core] 524288 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 72.26 sec
[bwa_aln_core] write to the disk... 0.01 sec
[bwa_aln_core] 592180 sequences have been processed.
[main] Version: 0.7.17-r1188
[main] CMD: /usr/local/bin/bwa aln -t 60 sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz
[main] Real time: 5.013 sec; CPU: 142.813 sec

...

Here is the command line statement you need. Just execute it on the command line.

bwa sampe sacCer3/sacCer3.fa yeast_R1.sai yeast_R2.sai \
  fastq/Sample_Yeast_L005_R1.cat.fastq.gz \
  fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pairedend.sam

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

# Copy a pre-built bwa index for sacCer3
mkdir -p $SCRATCH/core_ngs/references/bwa/sacCer3
cp $CORENGS/references/bwa/sacCer3/*.* $SCRATCH/core_ngs/references/bwa/sacCer3/

# Get the FASTQ to align
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

# Stage the BWA .sai files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -sf ../fastq
ln -sf ../../references/bwa/sacCer3
cp $CORENGS/catchup/yeast_bwa/*.sai .

cd $SCRATCH/core_ngs/alignment/yeast_bwa
bwa sampe sacCer3/sacCer3.fa yeast_pe_R1.sai yeast_pe_R2.sai \
  fastq/Sample_Yeast_L005_R1.cat.fastq.gz \
  fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pe.sam

You should now have a SAM file (yeast_pe.sam) that contains the alignments.

Exercise: How many lines does the SAM file have? How does this compare to the number of input sequences (R1+R2)?

You should now have a SAM file (yeast_pairedend.sam) that contains the alignments. It's just a text file, so take a look with head, more, less, tail, or whatever you feel like. Later you'll learn additional ways to analyze the data with samtools once you create a BAM file.

...

grep -P -c '^HWI' yeast_pairedendpe.sam

grep -P -v -c '^@' yeast_pairedendpe.sam

Exercise: How many sequences were in the R1 and R2 FASTQ files combined?

Exercises:

...

Exercises:

• Does the SAM file contain both mapped and un-mapped reads?
• What is the order of the alignment records in this SAM file?
  
  

...

Suppose you wanted to look only at field 3 (contig name) values in the SAM file. You can do this with the handy cut command. Below is a simple example where you're asking cut to display the 3rd column value for the last 10 alignment records.

# Stage the aligned SAM file
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sam .

tail yeast_pairedendpe.sam | cut -f 3

By default cut assumes the field delimiter is Tab, which is the delimiter used in the majority of NGS file formats. You can specify a different delimiter with the -d option.

...

tail -20 yeast_pairedendpe.sam | cut -f 2-6,9

You may have noticed that some alignment records contain contig names (e.g. chrV) in field 3 while others contain an asterisk ( * ). The * means the record didn't map. We're going to use this heuristic along with cut to see about how many records represent aligned sequences. (Note this is not the strictly correct method of finding unmapped reads because not all unmapped reads have an asterisk in field 3. Later you'll see how to properly distinguish between mapped and unmapped reads using samtools.)

...

# the ^@ pattern matches lines starting with @ (only header lines), 
# and -v says output lines that don't match
grep -v -P '^@' yeast_pairedendpe.sam | head

Ok, it looks like we're seeing only alignment records. Now let's pull out only field 3 using cut:

...

grep -v -P '^@' yeast_pairedendpe.sam | cut -f 3 | grep -v '*' | head

...

grep -v -P '^@' yeast_pairedendpe.sam | cut -f 3 | grep -v '*' | wc -l

...

idev -m 120 -N 1 -A OTH21164 -r CoreNGSday4 CoreNGS-Thu
# or
idev -m 90 -N 1 -A OTH21164 -p development

# If not already loaded
module load biocontainers  # takes a while

module load samtools
samtools

...

In this exercise, we will explore five utilities provided by samtools: view, sort, index, flagstat, and idxstats. Each of these is executed in one line for a given SAM/BAM file. In the SAMtools/BEDtools sections tomorrow we will explore samtools in capabilities more in depth.

...

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedendpe.sam .

cd $SCRATCH/core_ngs/alignment/yeast_bwa
cat yeast_pairedend.sam | samtools view -b -oyeast_pe.sam > yeast_pairedendpe.bam 

• the -b option tells the tool to output BAM format
• the -o option specifies the name of the output BAM file that will be created
• we pipe the entire SAM file to samtools view so that the header records are included (required for SAM → BAM conversion)
  • samtools view reads its input from standard input by default
• the tool to output BAM format

The BAM file is a binary file, not a text file, so how do you look at its contents now? How do you look at the BAM file contents now? That's simple. Just use samtools view without the -b option. Remember to pipe output to a pager!

samtools view yeast_pairedendpe.bam | more

Notice that this does not show us the header record we saw at the start of the SAM file.

Exercise: What samtools view option will include the header records in its output? Which option would show only the header records?

...

Looking at some of the alignment record information (e.g. samtools view yeast_pairedendpe.bam | cut -f 1-4 | more), you will notice that read names appear in adjacent pairs (for the R1 and R2), in the same order they appeared in the original FASTQ file. Since that means the corresponding mappings are in no particular order, searching through the file very inefficient. samtools sort re-orders entries in the SAM file either by locus (contig name + coordinate position) or by read name.

...

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.bampe.[bs]am .

To sort the paired-end yeast BAM file by position, and get a BAM file named yeast_pairedendpe.sort.bam as output, execute the following command:

cd $SCRATCH/core_ngs/alignment/yeast_bwa
samtools sort -O bam -T yeast_pairedendpe.tmp yeast_pairedendpe.bam > yeast_pairedendpe.sort.bam

• The -O options says the Output format should be BAM
• The -T options gives a prefix for Temporary files produced during sorting
  • sorting large BAMs will produce many temporary files during processingprocessing
  • make sure the temporary file prefix is different from the input BAM file prefix!
• By default sort writes its output to standard output, so we use > to redirect to a file named yeast_pairedend.sort.bam

Exercise: Compare the file sizes of the yeast_pariedend pe .sam, .bam, and .sort.bam files and explain why they are different.

ls -lh yeast_pairedendpe.*

...

samtools index yeast_pairedendpe.sort.bam

This will produce a file named yeast_pairedendpe.bam.bai.

Most of the time when an index is required, it will be automatically located as long as it is in the same directory as its BAM file and shares the same name up until the .bai extension.

...

ls -lh yeast_pairedendpe.sort.bam*

samtools flagstat

...

Here's how to run samtools flagstat and both see the output in the terminal and save it in a file – the samtools flagstat standard output is piped to tee, which both writes it to the specified file and sends it to its standard output:to the specified file and sends it to its standard output:

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sort.bam* .

samtools flagstat yeast_pairedendpe.sort.bam | tee yeast_pariedendpe.flagstat.txt

You should see something like this:

...

More information about the alignment is provided by the samtools idxstats report, which shows how many reads aligned to each contig in your reference. Note that samtools idxstats must be run on a sorted, indexed BAM file.

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sort.bam* .

samtools idxstats yeast_pairedendpe.sort.bam | tee yeast_pairedendpe.idxstats.txt

Here we use the tee command which reports its standard input to standard output before also writing it to the specified file.

...