Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

TACC storage areas and Linux commands to access data
(all commands to be executed at TACC except
laptop-to-TACC copies, which must be executed on your laptop)

Image RemovedImage Added

Local file systems

There are 3 local file systems available on any TACC compute cluster (stampede2, lonestar5 lonestar6, etc.), each with different characteristics. All these local file systems are very fast and set up for parallel I/O (Lustre file system).

On stampede2 lonestar6 these local file systems have the following characteristics:


HomeWork2WorkScratch
quota10 GB1024 GB = 1 TB2+ PB (basically infinite)
policybacked upnot backed up,
not purged
not backed up,
purged if not accessed recently (~10 days)
access commandcdcdw2cdwcds
environment variable$HOME

$WORK (different sub-directory for each cluster)

$STOCKYARD (root of the shared Work file system

)
$WORK2 (different sub-directory for each cluster

)

$SCRATCH
root file system/home/work2work/scratch
use forSmall files such as scripts that you don't want to lose.Medium-sized artifacts files you don't want to copy over all the time. For example, custom programs you install (these can get large), or annotation file used for analysis.Large files accessed from batch jobs. Your starting files will be copied here from somewhere else, and your final results files will be copied elsewhere (e.g. stockyard, corral, or your BRCF POD), or your organization's storage area.

When you first login, the system gives you information about disk quota quotas and your compute allocation quota:balance in "SU" (system units).

Code Block
--------------------- Project balances for user abattenh ----------------------
| Name           Avail SUs     Expires | Name           Avail SUs     Expires |
| OTH21095 genomeAnalysis       673  2021-03-31 | BioinformaticsResour 1001000  20202022-0609-30 |
| UT-2015-05-18 MCB21106       1000  2021-03-31 | DNAdenovo  995  2022-12-31 |
| OTH21164     4969  2021-03-31 |
| CancerGenetics   1000   4856  2020-09-302022-12-31 | A-cm10OTH21037            4224  8867  20202022-12-31 |
------------------------ Disk quotas for user abattenh ------------------------
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /home1              0.0      1011.07     0.1000           52      153     10000000    0.0200 |
| /work2work             614169.50    1024.0    6016.0150        6109479362     3000000    2.04 |
| /scratch         2676.6       0.0     0.00        32442           0    0.00 |
65 |
------------------------------------------------------------------------------------------------

changing TACC file systems

When you first login, you start in your home Home directory. Use the cd, cdw2 cdw and cds commands to change to your other file systems. Notice how your command prompt helpfully changes to show your location.

Code Block
languagebash
titleChanging file systems at TACC
cdw2cdw
cds
cd


Tip

The cd (change directory) command with no arguments takes you to your home directory on any Linux/Unix system.

The cdw2 cdw and cds commands are specific to the TACC environment.

...

TACC compute clusters now share a common Work file system called stockyard. So files in your Work area do not have to be copied, for example from to stampede2 to ls5 ls6 – they can be accessed directly from either cluster.

...

  • $STOCKYARD - This refers to the root of your shared Work area
    • e.g. /work/01063/abattenh  (should be changed to /work2/01063/abattenh soon)
  • $WORK or $WORK2 - Refers to a sub-directory of the shared Work area that is different for different clusters, e.g.:
    • /work/01063/abattenh/lonestarls6 on ls5 lonestar6
    • /work2work/01063/abattenh/stampede2 on stampede2

...

Code Block
languagebash
titleThe shared BioITeam directory
ls /work2work/projects/BioITeam

Files we will use in this course are in a sub-directory there. The $CORENGS environment variable set in your login profile refers to this path.

Code Block
languagebash
titleOur shared class directory
echo $CORENGS
ls /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools

...

ranch is a gigantic (multiple PB) tape archive system where researchers can archive data. All TACC users have an automatice automatic 2 TB ranch allocation. UT researchers may request larger (multi-TB) ranch storage allocations through the normal TACC allocation request process.

...

Once that data is staged to the ranch disk it can be copied to other places. However, the ranch file system is not mounted as a local file system from the stampede2 or ls5 ls6 clusters. So remote copy commands are always needed to copy data to and from ranch (e.g. scp, rsync). ranch (e.g. scp, rsync).

About file systems

File systems are storage areas where files and directories are arranged in a hierarchy. Any computer can have one or more file systems "mounted" (accessible as local storage). The df command can be used on any Unix system to show all the "top-level" mounted file systems. TACC has a lot of temporary file systems, so lets just look at the first 15 and tell df to use "human readable" size formatting with the -h option:

Code Block
languagebash
df -h | head -15

The rightmost "Mounted on" column give the top-level access path. Find /home1, /work, and /scratch and note their sizes!

What do we mean by "hierarchy"? It is like a tree, with the root file system as the trunk, sub-directories as branches, sub-sub-directories as branches from branches (and so forth), with files as leaves off any branch.

your TACC Home directory
using home-directory-relative (~) path syntax

Image Added

But everyone has a Home directory, so you must only be seeing a part of the Home directory hierarchy. To see the absolute path of a directory you're in, use the pwd -P command. Note that absolute paths always start with a forward slash ( / ) denoting its root file system.

Code Block
languagebash
pwd -P
# will show something like this
# /home1/01063/abattenh

That shows you that your Home directory (~) is actually 3 levels down in the /home1 hierarchy:

part of the TACC Home file system
using absolute paths

Image Added

Here's a depiction of the three file systems as seen from your Home directory ( ~ ), showing where the path-valued environment variables represent, and where the three symbolic links (~/CoreNGS, ~/scratch, ~/work) you created in your Home directory point. Notice that both the Work and Scratch file systems have a top-level hierarchy like we saw in Home above.

Image Added

Staging your data

So, your sequencing center has some data for you. They may send you a list of web or FTP links to use to download the data.

...

  • original – for original sequencing data (compressed FASTQ files)
    • sub-directories named, for example, by year_month.<project_<sequencing run/job or project name>
  • aligned – for alignment artifacts data (BAM files, etc)
    • sub-directories named, e.g.,  by year_month.<project_name>
  • analysis – further downstream analysis
    • reasonably named sub-directories, often by project
  • refs – reference genomes and other annotation files used in alignment and analysis
    • sub-directories for different reference genomes and aligners
    • e.g. ucsc/hg19hg38/star, ucsc/sacCer3/bwa, mirbase/v20/v20bowtie2
  • code – for scripts and programs you and others in your organization write
    • ideally maintained in a version control system such as git, subversion or cvs.
    • easiest to name can have separate sub-directories for people, or various shared repositories.

Download from a link – wget

...

The cp command copies one or more files from a local source to a local destination. It has many options, but the most common form is:

cp [options] <source_file_1> <source_file_2> ... <destination_directory>/

Make a directory in your Scratcharea and copy a single file to it. The trailing slash ( / ) on the destination says the destination is a directory.

Code Block
languagebash
titleSingle file copy with cp
mkdir -p $SCRATCH/data/test1
cp $CORENGS/misc/small.fq  $SCRATCH/data/test1/
ls $SCRATCH/data/test1

# or..
mkdir -p ~/scratch/data/test1   # use the symbolic link in your Home directory
cd ~/scratch/data/test1
cp $CORENGS/misc/small.fq  .
ls

...

Code Block
languagebash
titleDirectory copy with cp
mkdir -p $SCRATCH/data 
cds
cd data
cp -r $CORENGS/general/ general/

...

Expand
titleHint
ls general
# or
tree $SCRATCH/data/general


Expand
titleAnswer
BEDTools-User-Manual.v4.pdf  SAM1.pdf  SAM1.v1.4.pdf

...

rsync is a very complicated program, with many options (http://rsync.samba.org/ftp/rsync/rsync.html). However, if you use the recipe shown here for directories, it's hard to go wrong:

rsync -avW local/path/to/source_directory/ local/path/to/destination_directory/

Both the source and target directories are local (in some file system accessible directly from stampede2 lonestar6). Either full or relative path syntax can be used for both. The -avW options above stand for:

...

Tip
titleAlways add a trailing slash ( / ) after directory names

The trailing slash ( / ) on the source and destination directories are very important for rsync (and for other Linux copy commands also)!

rsync will create the last directory level for you, but earlier levels must already exist.

...

Code Block
languagebash
titlersync (local directory)
mkdir -p $SCRATCH/data
cds
rsync -avWavrW $CORENGS/custom_tracks/ data/custom_tracks/

...

Expand
titleHint
ls $SCRATCH/data/custom_tracks
# or
ls ~/scratch/data/custom_tracks
# or
cds; cd data/custom_tracks; ls_tracks; ls
# or
tree $SCRATCH/data

Now repeat the rsync and see the difference.

...

Code Block
languagebash
rsync -avWavrW /work/projects/BioITeam/projects/courses/Core_NGS_Tools/custom_tracks/ data/custom_tracks/

...

Tip

The bash shell has several convenient line editing features:

  • use the Up arrow to scroll back through the command line history; Down arrow goes forward
  • use Ctrl-a to move the cursor to the beginning of a line; Ctrl-e to the end
  • use Backspace to remove text before the cursor; Delete to remove text after the cursor
  • Ctrl-k ("kill") to delete all text on your command line after the cursor
  • Ctrl-y ("yank") to copy the last killed text to where the cursor is

Copy from a remote computer - scp or rsync

...

Code Block
titlesingle remote file copy with scp
cat $CORENGS/tacc/dragonfly_access.txt
cds
mkdir -p data/test2
scp corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/progeria_ctcf.vcf.gz ./data/test2/
lstree .$SCRATCH/data/test2

Notes:

  • The 1st time you access a new host the SSH security prompt will appear
  • You will be prompted for your remote host password
  • The  -r recursive argument works for scp also, just like for cp

...

  • The tilde ( ~ ) at the start of the path means "relative to my home directory"
  • We use the tilde ( ~ ) in the destination to traverse the ~/scratch symbolic link in your home directory.

Code Block
languagebash
titlersync (remote directory)rsync (remote directory)
cat $CORENGS/tacc/dragonfly_access.txt
rsync -avWavrW corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/ ~/scratch/data/custom_tracks/

...

Expand
titleAnswer

No, because all the source files were already present in the destination directory (you copied the same files earlier) with the same names, file sizes and timestamps. So rsync had nothing to do!

...

Here's a fun scavenger hunt for more practice . Issue of the following commands to get practice what you've learned so far:.

Expand
titleHint

Hit Tab Tab as much as possible to save typing!

...

Code Block
titlePlay a scavenger hunt for more practice
cd
cp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what
# or using the $CORENGS environment variable
cp -r $CORENGS/linuxpractice/what what
cd what
cat readme

Where are you when you're all done?

Expand
titleAnswer

stamp2ls6:~/what/starts/here/changes/the/world

...

Expand
titleStep 1 answer

From inside your ~/what directory:

Code Block
titlePlay a scavenger hunt for more practice
mkdir starts
cd starts
cp /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/steps/nextInstr .
cat nextInst


...

Expand
titleStep 3 answer

From inside your ~/what/starts/here directory:

Code Block
titlePlay a scavenger hunt for more practice
scp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# or
rsync -ptrvP /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# Note: rsync -avPavrP ... will also work, but will report an error because the destination file and
# directory ownership cannot be changed to match the source. But the files will be copied, and
# ownership assigned to you.
 
# Then
cd changes 
more largeFile.txt


...

Expand
titleStep 4 answer

From inside your ~/what/starts/here/changes directory:

Code Block
titlePlay a scavenger hunt for more practice
rsync -avPavrP corengstools@dragonfly.icmb.utexas.edu:~/the/ the/
# or
scp -r corengstools@dragonfly.icmb.utexas.edu:~/the/ the/

cd the
cat instr5.txt
cd world
cat instr6.txt

The path to the directory you're in now should be:

~/what/starts/here/changes/the/world