...
| Code Block |
|---|
|
cd
ln -s -f $SCRATCH scratch
ln -s -f $WORK work$WORK2 work2
ln -s -f /work2/projects/BioITeam
ln -s -f /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools CoreNGS
mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /workwork2/projects/BioITeam/common/bin/launcher_makercreator.py
ln -s -f /workwork2/projects/BioITeam/ls5/opt/cutadapt-1.10/bin/cutadapt
ln -s -f /work/projects/BioITeam/ls5/opt/multiqc-1.0/multiqc
ln -s -f /work/projects/BioITeam/ls5/opt/samstat-1.09/samstatcommon/bin/launcher_maker.py
|
| Anchor |
|---|
| Catchup_bashrc |
|---|
| Catchup_bashrc |
|---|
|
.bashrc setup
...
| Code Block |
|---|
|
cd
cp .bashrc .bashrc.beforeNGSbeforeNGSTools |
Copy and configure the login profile for this class
| Code Block |
|---|
|
cd
cp /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/tacc/bashrc.corengs.stampede2 .bashrc
chmod 600 .bashrc
# or, using your symlink
cd
cp ~/CoreNGS/tacc/bashrc.corengs.ls5stampede2 .bashrc
chmod 600 .bashrc |
...
| Code Block |
|---|
|
export ALLOCATION=UT-2015-05-18
export BI=/corral-repl/utexas/BioITeam
export BIWORK=/workwork2/projects/BioITeam
export CORENGS=$BIWORK/projects/courses/Core_NGS_Tools
exportexport PATH=.:$HOME/local/bin:$PATH
# For cutadapt support:
export PYTHONPATH=$BIWORK/ls5/lib/python2.7/site-packages:$PYTHONPATH
# For MultiQC support:
export PYTHONPATH=$BIWORK/ls5/lib/python2.7/annab-packages:$PYTHONPATH
|
Turn on coloring by file type in the shell:
| Code Block |
|---|
|
export LS_OPTIONS='-N --color=auto -T 0'
# For better colors using a whitedark background terminal, un-comment this line:
export LS_COLORS=$LS_COLORS:'di=1;33:fi=01:ln=01;36:'
# For better colors using a white background terminal:
export LS_COLORS=$LS_COLORS:'di=1;34:fi=01:ln=01;36:'
# May or may not be needed
export LS_OPTIONS='-N --color=auto -T 0 |
TACC intro
Commands files
...
| Code Block |
|---|
|
mkdir -p $SCRATCH/core_ngs/slurm/simple
cd $SCRATCH/core_ngs/slurm/simple
cp $CORENGS/tacc/simple.cmds . |
Wayness commands
| Code Block |
|---|
|
mkdir -p $SCRATCH/core_ngs/slurm/wayness
cd $SCRATCH/core_ngs/slurm/wayness
cp $CORENGS/tacc/wayness.cmds . |
Start an idev session
To start an a 3-hour idev (interactive development) session:
| Code Block |
|---|
| language | bash |
|---|
| title | Start an idev session |
|---|
|
idev -p normal -m 120 -N 1 -n 2468 -A UT-2015-05-18 --reservation=CCBBBIO_DATA_week_1 |
| Tip |
|---|
You can tell you're in a idev session because the hostname command will return a compute node name (e.g. nid00438. c401-041.stampede2.tacc.utexas.edu) instead of a login node name (e.g. login5. login2.stampede2.tacc.utexas.edu). |
The n idev session will terminate when the requested time has expired, or you use the exit command.
...
Working with some yeast ChIP-seq FASTQ data:
| Code Block |
|---|
|
# Create Areaa for$SCRATCH "original"area sequencingto data
mkdir -p $WORK/archive/original/2018_05.core_ngs
cd $WORK/archive/original/2018_05.core_ngs
wget http://web.corral.tacc.utexas.edu/BioITeam/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz
wget http://web.corral.tacc.utexas.edu/BioITeam/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz
# Create a $SCRATCH area for FASTQ prep and link the yeast data therework on data for this course,
# with a sub-direct[1ory for pre-processing raw fastq files
mkdir -p $SCRATCH/core_ngs/fastq_prep
# Make symbolic links to the original yeast data:
cd $SCRATCH/core_ngs/fastq_prep
ln -s -f $WORK/archive/original/2018_05.core_ngs$CORENGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz
ln -s -f $WORK/archive/original/2018_05.core_ngs$CORENGS/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz
# Copy over a small FASTQ file
cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/misc/small.fq . |
...
| Code Block |
|---|
|
mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq
cd $SCRATCH/core_ngs/multiqc/fqc.atacseq
cp $CORENGS/multiqc/fqc.atacseq/*.html . |
| Anchor |
|---|
| Catchup_cutadapt |
|---|
| Catchup_cutadapt |
|---|
|
FASTQ files for cutadapt
...
| Anchor |
|---|
| Catchup_references |
|---|
| Catchup_references |
|---|
|
References
Get a copy of all references we build in the exercises (including FASTA):
| Code Block |
|---|
|
mkdir -p $SCRATCH/core_ngs/references
rsync -ptlvrP $CORENGS/references/ $SCRATCH/core_ngs/references/
|
| Anchor |
|---|
| Catchup_bwa_pe_yeast |
|---|
| Catchup_bwa_pe_yeast |
|---|
|
BWA PE alignment of yeast dataTo jump into aligning PE yeast data with BWA
| Code Block |
|---|
|
# Pre-built references
mkdir -p $SCRATCH/core_ngs/references
rsync -avrP $CORENGS/references/ $SCRATCH/core_ngs/references/
# FASTQ (to align)
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/
# Alignment directory
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -s -f ../fastq
ln -s -f ../../references/bwa/sacCer3
module load biocontainers # takes a while
module load bwa
module load samtools |
| Anchor |
|---|
| Catchup_bwa_pe_yeast |
|---|
| Catchup_bwa_pe_yeast |
|---|
|
samtools manipulation of aligned yeast dataTo jump into post-alignment manipulation of the yeast_pairedend.bam with samtools:
| Code Block |
|---|
|
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.bam .
module load biocontainers # takes a while
module load samtools
# If the sorted, indexed BAM is needed:
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.sort* . |
SAMTools and BEDTools
| Anchor |
|---|
| Catchup_samtools |
|---|
| Catchup_samtools |
|---|
|
Setup for samtools| Code Block |
|---|
| language | bash |
|---|
| title | Setup for samtools exercises |
|---|
|
mkdir -p $SCRATCH/core_ngs/samtools
cd $SCRATCH/core_ngs/samtools
cp $CORENGS/catchup/for_samtools/* .
module load biocontainers # takes a while
module load samtools |