...
TACC storage areas and Linux commands to access data
(all commands to be executed at TACC except
laptop-to-TACC copies, which must be executed on your laptop) |
 Image Removed Image Added
|
Local file systems
...
- $STOCKYARD - This refers to the root of your shared Work area
- e.g. /work/01063/abattenh (should be changed to /work2/01063/abattenh soon)
- $WORK or $WORK2 - Refers to a sub-directory of the shared Work area that is different for different clusters, e.g.:
- /work/01063/abattenh/lonestar on ls5
- /workwork2/01063/abattenh/stampede2 on stampede2
...
| Code Block |
|---|
| language | bash |
|---|
| title | Get ready to wget |
|---|
|
mkdir -p $WORK2$SCRATCH/archive/original/2021.core_ngs
cd $WORK2$SCRATCH/archive/original/2021.core_ngs
wget |
...
Copy an entire directory to your scratch Scratch area. The -r argument option says "recursive".
| Code Block |
|---|
| language | bash |
|---|
| title | Directory copy with cp |
|---|
|
mkdir -p $SCRATCH/data
cds
cd data
cp -r $CORENGS/general/ general/ |
...
Both the source and target directories are local (in some file system accessible directly from ls5 stampede2). Either full or relative path syntax can be used for both. The -avP avW options above stand for:
- -a means "archive mode", which implies the following options (and a few others)
- -p – preserve file permissions
- -t – preserve file times
- -l – copy symbolic links as links
- -r – recursively copy sub-directories
- -v means verbose
- -W means transfer Whole file only
- Normally the rsync algorithm compares the contents of files that need to be copied and only transfers the different parts.
- For large files and binary files, figuring out what has changed (diff-ing) can take more time than just copying the whole file.
- The -W option disables file content comparisons (skips diff-ing).
...
| Code Block |
|---|
| language | bash |
|---|
| title | rsync (local directory) |
|---|
|
mkdir -p $SCRATCH/data
cds
rsync -avWPavW $CORENGS/custom_tracks/ data/custom_tracks/ |
...
| Code Block |
|---|
|
rsync -avWPavW /work/projects/BioITeam/projects/courses/Core_NGS_Tools/custom_tracks/ data/custom_tracks/ |
| Tip |
|---|
The bash shell has several convenient line editing features: - use the Up arrow to scroll back through the command line history; Down arrow goes forward
- use Ctrl-a to move the cursor to the beginning of a line; Ctrl-e to the end
- use Backspace to remove text before the cursor; Delete to remove text after the cursor
- use Ctrl-a and then Ctrl-k to delete all text on your command line
|
Copy from a remote computer - scp or rsync
...
| Code Block |
|---|
| title | single remote file copy with scp |
|---|
|
mkdir -p $SCRATCH/data/test1
cds
cat $CORENGS/tacc/dragonfly_access.txt
cds
mkdir -p data/test2
scp corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/progeria_ctcf.vcf.gz ./data/test1test2/
ls ./data/test1test2 |
Notes:
- The 1st time you access a new host the SSH security prompt will appear
- You will be prompted for your remote host password
- The -r recursive argument works for scp also, just like for cp
...
| Code Block |
|---|
| language | bash |
|---|
| title | rsync (remote directory) |
|---|
|
rsync -avWPavW corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/ ~/scratch/data/custom_tracks/ |
...
| Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
|
cd
cp -r /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what
# or using the $CORENGS environment variable
cp -r $CORENGS/linuxpractice/what what
cd what
cat readme |
Where are you when you're all done?
| Expand |
|---|
|
ls5stamp2:~/what/starts/here/changes/the/world
|
...
| Expand |
|---|
|
From inside your ~/what directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| mkdir starts
cd starts
cp /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/steps/nextInstr .
cat nextInst |
|
| Expand |
|---|
|
From inside your ~/what/starts directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| mkdir here
cd here
wget http://web.corral.tacc.utexas.edu/BioITeamBioinformaticsResource/CoreNGS/step3.txt
cat step3.txt |
|
| Expand |
|---|
|
From inside your ~/what/starts/here directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| scp -r /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# or
rsync -ptrvP /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# Note: rsync -avP ... will also work, but will report an error because the destination file and
# directory ownership cannot be changed to match the source. But the files will be copied, and
# ownership assigned to you.
# Then
cd changes
more largeFile.txt |
|
...