Page History

Table of Contents

Environment setup

Anchor
Catchup_symlinks
Catchup_symlinks
Directories and symlinks

Directories and links needed in your home directory.

Code Block

language	bash

cd 
ln -s -f $SCRATCH scratch
ln -s -f $WORK2 work2
ln -s -f /work2/projects/BioITeam
ln -s -f /work2/projects/BioITeam/projects/courses/Core_NGS_Tools CoreNGS

mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /work2/projects/BioITeam/common/bin/launcher_creator.py
ln -s -f /work2/projects/BioITeam/common/bin/launcher_maker.py

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

If you already have a .bashrc set up, make a backup copy first. You can restore your original login script after class is over.

Code Block

language	bash

cd
cp .bashrc .bashrc.beforeNGSTools

Copy and configure the login profile for this class

Code Block

language	bash

cd
cp /work2/projects/BioITeam/projects/courses/Core_NGS_Tools/tacc/bashrc.corengs.stampede2  .bashrc
chmod 600 .bashrc

# or, using your symlink
cd
cp ~/CoreNGS/tacc/bashrc.corengs.stampede2  .bashrc
chmod 600 .bashrc

Source it to make it active (if this doesn't work, log off then log back in):

Code Block

language	bash
title	Copy a pre-configured login script

source ~/.bashrc

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

General

Code Block

language	bash

export ALLOCATION=UT-2015-05-18
export BIWORK=/work2/projects/BioITeam
export CORENGS=$BIWORK/projects/courses/Core_NGS_Tools

export PATH=.:$HOME/local/bin:$PATH

Turn on coloring by file type in the shell:

Code Block

language	bash

# For better colors using a dark background terminal, un-comment this line:
export LS_COLORS=$LS_COLORS:'di=1;33:fi=01:ln=01;36:'

# For better colors using a white background terminal:
export LS_COLORS=$LS_COLORS:'di=1;34:fi=01:ln=01;36:'

# May or may not be needed
export LS_OPTIONS='-N --color=auto -T 0

TACC intro

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Simple commands

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/slurm/simple
cd $SCRATCH/core_ngs/slurm/simple
cp $CORENGS/tacc/simple.cmds .

Wayness commands

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/slurm/wayness
cd $SCRATCH/core_ngs/slurm/wayness
cp $CORENGS/tacc/wayness.cmds .

Anchor
Catchup_idev
Catchup_idev
Start an idev session

To start an a 3-hour idev (interactive development) session:

Code Block

language	bash
title	Start an idev session

idev -p normal -m 120 -N 1 -n 2468 -A UT-2015-05-18 --reservation=CCBBBIO_DATA_week_1

Tip
You can tell you're in a idev session because the hostname command will return a compute node name (e.g. nid00438 c401-041.stampede2.tacc.utexas.edu) instead of a login node name (e.g. login5. login2.stampede2.tacc.utexas.edu).

The n idev session will terminate when the requested time has expired, or you use the exit command.

Working with FASTQ

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Working with some yeast ChIP-seq FASTQ data:

Code Block

language	bash

# Create a $SCRATCH area to work on data for this course,
# with a sub-direct[1ory for pre-processing raw fastq files
mkdir -p $SCRATCH/core_ngs/fastq_prep

# Make symbolic links to the original yeast data:
cd $SCRATCH/core_ngs/fastq_prep
ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz
ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz

# Copy over a small FASTQ file
cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/misc/small.fq .

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Get some FastQC reports for MultiQC:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq
cd $SCRATCH/core_ngs/multiqc/fqc.atacseq
cp $CORENGS/multiqc/fqc.atacseq/*.html .

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

For command-line cutadapt exploration:

Code Block

language	bash

cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .
zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2000 > miRNA_test.fq

For batch cutadapt processing:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/cutadapt
cd $SCRATCH/core_ngs/cutadapt
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R1.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R2.fastq.gz .
cp $CORENGS/tacc/cuta.cmds .

Alignment workflow

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Starting files:

Code Block

language	bash

# FASTA (for building references)
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/*.* $SCRATCH/core_ngs/references/fasta/

# FASTQ (to align)
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

Anchor
Catchup_references
Catchup_references
References

Get a copy of all references we build in the exercises (including FASTA):

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/references
rsync -ptlvrP $CORENGS/references/ $SCRATCH/core_ngs/references/

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

To jump into aligning PE yeast data with BWA

Code Block

language	bash

# Pre-built references
mkdir -p $SCRATCH/core_ngs/references
rsync -avrP $CORENGS/references/ $SCRATCH/core_ngs/references/

# FASTQ (to align)
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

# Alignment directory
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -s -f ../fastq
ln -s -f ../../references/bwa/sacCer3

module load biocontainers  # takes a while
module load bwa
module load samtools

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

To jump into post-alignment manipulation of the yeast_pairedend.bam with samtools:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.bam .

module load biocontainers  # takes a while
module load samtools

# If the sorted, indexed BAM is needed:
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.sort* .

SAMTools and BEDTools

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools

Code Block

language	bash
title	Setup for samtools exercises

mkdir -p $SCRATCH/core_ngs/samtools
cd $SCRATCH/core_ngs/samtools
cp $CORENGS/catchup/for_samtools/* .

module load biocontainers  # takes a while
module load samtools

Page tree

Versions Compared

Old Version 1

New Version Current

Key

Environment setup

Anchor
Catchup_symlinks
Catchup_symlinks
Directories and symlinks

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

TACC intro

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Anchor
Catchup_idev
Catchup_idev
Start an idev session

Working with FASTQ

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

Alignment workflow

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Anchor
Catchup_references
Catchup_references
References

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

SAMTools and BEDTools

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools

Page tree

Page History

Versions Compared

Old Version 1

New Version Current

Key

Environment setup

AnchorCatchup_symlinksCatchup_symlinksDirectories and symlinks

AnchorCatchup_bashrcCatchup_bashrc.bashrc setup

AnchorCatchup_env_varsCatchup_env_varsEnvironment variables

TACC intro

AnchorCatchup_cmdsCatchup_cmdsCommands files

AnchorCatchup_idevCatchup_idevStart an idev session

Working with FASTQ

AnchorCatchup_yeast_dataCatchup_yeast_dataYeast data

AnchorCatchup_multiqcCatchup_multiqcATACseq data for MultiQC

AnchorCatchup_cutadaptCatchup_cutadaptFASTQ files for cutadapt

Alignment workflow

AnchorCatchup_align_setupCatchup_align_setupAlignment workflow setup

AnchorCatchup_referencesCatchup_referencesReferences

AnchorCatchup_bwa_pe_yeastCatchup_bwa_pe_yeastBWA PE alignment of yeast data

AnchorCatchup_bwa_pe_yeastCatchup_bwa_pe_yeastsamtools manipulation of aligned yeast data

SAMTools and BEDTools

AnchorCatchup_samtoolsCatchup_samtoolsSetup for samtools

Anchor
Catchup_symlinks
Catchup_symlinks
Directories and symlinks

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Anchor
Catchup_idev
Catchup_idev
Start an idev session

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Anchor
Catchup_references
Catchup_references
References

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools