...
| Home | Work | Scratch | |
|---|---|---|---|
| quota | 10 GB | 1024 GB = 1 TB | 122+ PB (basically infinite) |
| policy | backed up | not backed up, not purged | not backed up, purged if not accessed recently (~10 days) |
| access command | cd | cdw | cds |
| environment variable | $HOME | $STOCKYARD (root of the shared Work file system) $WORK (different sub-directory for each cluster) | $SCRATCH |
| root file system | /home | /work | /scratch |
| use for | Small files such as scripts that you don't want to lose. | Medium-sized artifacts you don't want to copy over all the time. For example, custom programs you install (these can get large), or annotation file used for analysis. | Large files accessed from batch jobs. Your starting files will be copied here from somewhere else, and your final results files will be copied elsewhere (e.g. stockyard, corral, or your BRCF POD). |
...
| Code Block |
|---|
--------------------- Project balances for user abattenh ---------------------- | Name Avail SUs Expires | Name Avail SUs Expires | | CancerGeneticsgenomeAnalysis 4856673 20182021-0903-3031 | A-cm10BioinformaticsResour 100 1096 2018-12-312020-06-30 | | UT-2015-05-18 21001000 20192021-03-31 | DNAdenovo genomeAnalysis 25004969 20192021-03-31 | -----------| CancerGenetics 4856 2020-09-30 | A-cm10 8867 2020-12-31 | ------------------------ Disk quotas for user abattenh ------------------------- | Disk Usage (GB) Limit %Used File Usage Limit %Used | | /home1 0.0 10.0 0.1210 91153 1000000 0.0102 | | /work 538614.5 1024.0 5260.5901 6105361094 3000000 2.04 | | /scratch 37252676.96 0.0 0.00 413732442 0 0.00 | ------------------------------------------------------------------------------- |
...
| Tip |
|---|
The cd (change directory) command with no arguments takes you to your home directory on any Linux/Unix system. The cdw and cds commands are specific to the TACC environment. |
...
A mechanism for purchasing larger stockyard allocations (above the 1 TB basic quota) from TACC are in development.
The UT Austin BioInformatics Team, a loose group of bioinformatics researchers, maintains a common directory area on stockyard.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
echo $CORENGS
ls /work/projects/BioITeam/projects/courses/Core_NGS_Tools |
...
ranch is a gigantic (multiple PB) tape archive system where researchers can archive data. All TACC users have an automatice 2 TB ranch allocation. UT researchers may request large larger (multi-TB) ranch storage allocations through the normal TACC allocation request process.
There is currently no charge for ranch storage. However, since the data is stored on tape it is not immediately available – robots find and mount appropriate tapes when the data is requested, and it can take minutes to hours for the data to appear on disk. ( The metadata about your data – the directory structures and file names – is always accessible, but the actual data in the files is not on disk until "staged". See the ranch user guide for more information: https://www.tacc.utexas.edu/user-services/user-guides/ranch-user-guide.
Once that data is staged to the ranch disk it can be copied to other places. However, the ranch file system is not mounted as a local file system from the stampede2 or ls5 clusters. So remote copy commands are always needed to copy data to and from ranch (e.g. scp, rsync).
...
The first task is to get this sequencing data to a permanent storage area. This should not be your laptop or one of the TACC local file systems! corral (or stockyard) is a great place for it, or a server maintained by your lab or company.
We're going to pretend – just for the sake of this class – that your permanent storage area is in your TACC work area. Execute these commands to make your "archive" directory and some sub-directories.
...
| language | bash |
|---|---|
| title | Create a pretend "archive" directory |
...
or company.
Here's an example of a "best practice". Wherever your permanent storage area is, it should have a rational sub-directory structure that reflects its contents. It's easy to process a few NGS datasets, but when they start multiplying like tribbles, good organization and naming conventions will be the only thing standing between you and utter chaos!
...
Get ready to run wget from the directory where you want to put the data.
Don't press Enter after the wget command – just put a space after it.
| Code Block | ||||
|---|---|---|---|---|
| ||||
mkdir -p $WORK/archive/original/2020.core_ngs cd $WORK/archive/original/2018_052020.core_ngs wget |
Here are two web links:
...
Suppose you have a corral allocation or stockyard area where your organization keeps its data, and that the sequencing data has been downloaded there. You can use various Linux commands to copy the data locally from there to your $SCRATCH area.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
mkdir -p $SCRATCH/data
cds
cd data
cp -r $CORENGS/general/ general/ |
...
rsync is a very complicated program, with many options (http://rsync.samba.org/ftp/rsync/rsync.html). However, if you use the recipe shown here for directories, it's hard to go wrong:
rsync -avP avW local/path/to/source_directory/ local/path/to/destination_directory/
...
- -a means "archive mode", which implies the following options (and a few others)
- -p – preserve file permissions
- -t – preserve file times
- -l – copy symbolic links as links
- -r – recursively copy sub-directories
- -v means verbose
- -P means show ProgressW means transfer Whole file only
Since these are all single-character options, they can be combined after one option prefix dash ( - ). You could also use options -ptlrvPptlrvW, separately, instead of using -a for "archive mode".
| Tip | ||
|---|---|---|
| ||
The trailing slash ( / ) on the source and destination directories are very important for rsync (and for other Linux copy commands also)! rsync will create the last directory level for you, but earlier levels must already exist. |
...