Page History

...

There are 3 local file systems available on any TACC compute cluster (stampede2, lonestar5 lonestar6, etc.), each with different characteristics. All these local file systems are very fast and set up for parallel I/O (Lustre file system).

On stampede2 lonestar6 these local file systems have the following characteristics:

When you first login, the system gives you information about disk quota quotas and your compute allocation quota:balance in "SU" (system units).

--------------------- Project balances for user abattenh ----------------------
| Name           Avail SUs     Expires | Name           Avail SUs     Expires |
| genomeAnalysisOTH21095       673  2021-03-31 | BioinformaticsResour 1001000  20202022-0609-30 |
| UT-2015-05-18MCB21106       1000  2021-03-31 | DNAdenovo  995  2022-12-31 |
| OTH21164     4969  2021-03-31 |
| CancerGenetics   1000   4856  2020-09-302022-12-31 | A-cm10OTH21037            4224  8867  20202022-12-31 |
------------------------ Disk quotas for user abattenh ------------------------
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /home1              0.0      1011.07     0.1000           52      153     10000000    0.0200 |
| /work2work             614169.50    1024.0    6016.0150        6109479362     3000000    2.04 |
| /scratch         2676.6       0.0     0.00        32442           0    0.00 |
-------65 |
-------------------------------------------------------------------------------------------

changing TACC file systems

When you first login, you start in your home Home directory. Use the cd, cdw2 and cds commands to change to your other file systems. Notice how your command prompt helpfully changes to show your location.

cdw2cdw
cds
cd

...

TACC compute clusters now share a common Work file system called stockyard. So files in your Work area do not have to be copied, for example from to stampede2 to ls5 ls6 – they can be accessed directly from either cluster.

...

• $STOCKYARD - This refers to the root of your shared Work area
  • e.g. /work/01063/abattenh  (should be changed to /work2/01063/abattenh soon)
    
• $WORK or $WORK2 - Refers to a sub-directory of the shared Work area that is different for different clusters, e.g.:
  
  • /work/01063/abattenh/lonestarls6 on ls5 ls6
  • /work2/01063/abattenh/stampede2 on stampede2

...

ls /work2work/projects/BioITeam

Files we will use in this course are in a sub-directory there. The $CORENGS environment variable set in your login profile refers to this path.

...

ranch is a gigantic (multiple PB) tape archive system where researchers can archive data. All TACC users have an automatice automatic 2 TB ranch allocation. UT researchers may request larger (multi-TB) ranch storage allocations through the normal TACC allocation request process.

...

Once that data is staged to the ranch disk it can be copied to other places. However, the ranch file system is not mounted as a local file system from the stampede2 or ls5 ls6 clusters. So remote copy commands are always needed to copy data to and from ranch (e.g. scp, rsync)..g. scp, rsync).

About file systems

File systems are storage areas where files and directories are arranged in a hierarchy. Any computer can have one or more file systems "mounted" (accessible as local storage). The df command can be used on any Unix system to show all the "top-level" mounted file systems. TACC has a lot of temporary file systems, so lets just look at the first 15 and tell df to use "human readable" size formatting with the -h option:

df -h | head -15

The rightmost "Mounted on" column give the top-level access path. Find /home1, /work, and /scratch and note their sizes!

What do we mean by "hierarchy"? It is like a tree, with the root file system as the trunk, sub-directories as branches, sub-sub-directories as branches from branches (and so forth), with files as leaves off any branch.

But everyone has a Home directory, so you must only be seeing a part of the Home directory hierarchy. To see the absolute path of a directory you're in, use the pwd -P command. Note that absolute paths always start with a forward slash ( / ) denoting its root file system.

pwd -P
# will show something like this
# /home1/01063/abattenh

That shows you that your Home directory (~) is actually 3 levels down in the /home1 hierarchy:

Staging your data

So, your sequencing center has some data for you. They may send you a list of web or FTP links to use to download the data.

...

• original – for original sequencing data (compressed FASTQ files)
  • sub-directories named, for example, by year_month.<project_<sequencing run/job or project name>
• aligned – for alignment artifacts data (BAM files, etc)
  • sub-directories named, e.g.,  by year_month.<project_name>
• analysis – further downstream analysis
  • reasonably named sub-directories, often by project
• refs – reference genomes and other annotation files used in alignment and analysis
  
  • sub-directories for different reference genomes and aligners
  • e.g. ucsc/hg38/hg19star, ucsc/sacCer3/bwa, mirbase/v20/bowtie2
• code – for scripts and programs you and others in your organization write
  • ideally maintained in a version control system such as git, subversion or cvs.
  • easiest to name can have separate sub-directories for people, or various shared repositories.

Download from a link – wget

...

The cp command copies one or more files from a local source to a local destination. It has many options, but the most common form is:

cp [options] <source file 1> <source file 2> ... <destination directory>/

...

Here's a fun scavenger hunt for more practice . Issue of the following commands to get practice what you've learned so far:.

...

cd
cp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what
# or using the $CORENGS environment variable
cp -r $CORENGS/linuxpractice/what what
cd what
cat readme

Where are you when you're all done?

...

mkdir starts
cd starts
cp /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/steps/nextInstr .
cat nextInst

...

scp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# or
rsync -ptrvP /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# Note: rsync -avP ... will also work, but will report an error because the destination file and
# directory ownership cannot be changed to match the source. But the files will be copied, and
# ownership assigned to you.
 
# Then
cd changes 
more largeFile.txt

...