...
TACC storage areas and Linux commands to access data (all commands to be executed at TACC except laptop-to-TACC copies, which must be executed on your laptop) |
Local file systems
...
- $STOCKYARD - This refers to the root of your shared Work area
- e.g. /work/01063/abattenh (should be changed to /work2/01063/abattenh soon)
- e.g. /work/01063/abattenh (should be changed to /work2/01063/abattenh soon)
- $WORK or $WORK2 - Refers to a sub-directory of the shared Work area that is different for different clusters, e.g.:
- /work/01063/abattenh/lonestar on ls5
- /workwork2/01063/abattenh/stampede2 on stampede2
...
corral is a gigantic (multiple PB) storage system (spinning disk) where researchers can store data. UT researchers may request up to 5 TB of corral storage through the normal TACC allocation request process. Additional space on corral can be rented for ~$80/TB/year.The UT/Austin BioInformatics Team also has an older, common directory area on corral.
...
...
ls /corral-repl/utexas/BioITeam
A couple of things to keep in mind regarding corral:
...
Code Block | ||||
---|---|---|---|---|
| ||||
mkdir -p $WORK2$SCRATCH/archive/original/2021.core_ngs cd $WORK2$SCRATCH/archive/original/2021.core_ngs wget |
Here are two web links:
- httphttps://web.corral.tacc.utexas.edu/BioITeamBioinformaticsResource/CoreNGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gzhttp
- https://web.corral.tacc.utexas.edu/BioinformaticsResource/BioITeamCoreNGS/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz
Right-click (Windows) or Control+click (Mac) on the 1st link in your browser, then select "Copy link location" from the menu. Now go back to your Terminal. Put your cursor after the space following the wget command then either right-click (Windows), or Paste (Command-V on Mac, Control-V on Windows). The command line to be executed should now look like this:
Code Block | ||||
---|---|---|---|---|
| ||||
wget http://web.corral.tacc.utexas.edu/BioITeamBioinformaticsResource/CoreNGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz |
Now press Enter to get the command going. Repeat for the 2nd link. Check that you now see the two files (ls).
Tip |
---|
By default wget creates a file in the current directory matching the last component of the URL (e.g. Sample_Yeast_L005_R1.cat.fastq.gz here). You can change the copied file name with wget's -O option. Also note that if you execute the same wget more than once, subsequent local files will be named with a .1, .2, etc. suffix. |
Copy from a corral location - cp or rsync
...
cp [options] <source file 1> <source file 2> ... <destination directory>/
Make a directory in your scratch Scratcharea and copy a single file to it. The trailing slash ( / ) on the destination says it the destination is a directory.
Code Block | ||||
---|---|---|---|---|
| ||||
mkdir -p $SCRATCH/data/test1 cp $CORENGS/misc/small.fq $SCRATCH/data/test1/ ls $SCRATCH/data/test1 # or.. mkdir -p ~/scratch/data/test1 cd ~/scratch/data/test1 cp $CORENGS/misc/small.fq . ls |
Copy an entire directory to your scratch Scratch area. The -r argument option says "recursive".
Code Block | ||||
---|---|---|---|---|
| ||||
mkdir -p $SCRATCH/data cds cd data cp -r $CORENGS/general/ general/ |
...
Both the source and target directories are local (in some file system accessible directly from ls5 stampede2). Either full or relative path syntax can be used for both. The -avP avW options above stand for:
- -a means "archive mode", which implies the following options (and a few others)
- -p – preserve file permissions
- -t – preserve file times
- -l – copy symbolic links as links
- -r – recursively copy sub-directories
- -v means verbose
- -W means transfer Whole file only
- Normally the rsync algorithm compares the contents of files that need to be copied and only transfers the different parts.
- For large files and binary files, figuring out what has changed (diff-ing) can take more time than just copying the whole file.
- The -W option disables file content comparisons (skips diff-ing).
...
Code Block | ||||
---|---|---|---|---|
| ||||
mkdir -p $SCRATCH/data cds rsync -avWPavW $CORENGS/custom_tracks/ data/custom_tracks/ |
...
Code Block | ||
---|---|---|
| ||
rsync -avWPavW /work/projects/BioITeam/projects/courses/Core_NGS_Tools/custom_tracks/ data/custom_tracks/ |
Tip |
---|
The bash shell has several convenient line editing features:
|
Copy from a remote computer - scp or rsync
...
Code Block | ||
---|---|---|
| ||
mkdir -p $SCRATCH/data/test1 cds cat $CORENGS/tacc/dragonfly_access.txt cds mkdir -p data/test2 scp corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/progeria_ctcf.vcf.gz ./data/test1test2/ ls ./data/test1test2 |
Notes:
- The 1st time you access a new host the SSH security prompt will appear
- You will be prompted for your remote host password
- The -r recursive argument works for scp also, just like for cp
...
Code Block | ||||
---|---|---|---|---|
| ||||
rsync -avWPavW corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/ ~/scratch/data/custom_tracks/ |
...
Code Block | ||
---|---|---|
| ||
cd cp -r /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what # or using the $CORENGS environment variable cp -r $CORENGS/linuxpractice/what what cd what cat readme |
Where are you when you're all done?
Expand | ||
---|---|---|
| ||
|
...
Expand | |||||
---|---|---|---|---|---|
| |||||
From inside your ~/what directory:
|
Expand | |||||
---|---|---|---|---|---|
| |||||
From inside your ~/what/starts directory:
|
Expand | |||||
---|---|---|---|---|---|
| |||||
From inside your ~/what/starts/here directory:
|
...