...
Copy an entire directory to your scratch Scratch area. The -r argument option says "recursive".
| Code Block |
|---|
| language | bash |
|---|
| title | Directory copy with cp |
|---|
|
mkdir -p $SCRATCH/data
cds
cd data
cp -r $CORENGS/general/ general/ |
...
Both the source and target directories are local (in some file system accessible directly from ls5 stampede2). Either full or relative path syntax can be used for both. The -avP avW options above stand for:
- -a means "archive mode", which implies the following options (and a few others)
- -p – preserve file permissions
- -t – preserve file times
- -l – copy symbolic links as links
- -r – recursively copy sub-directories
- -v means verbose
- -W means transfer Whole file only
- Normally the rsync algorithm compares the contents of files that need to be copied and only transfers the different parts.
- For large files and binary files, figuring out what has changed (diff-ing) can take more time than just copying the whole file.
- The -W option disables file content comparisons (skips diff-ing).
...
| Code Block |
|---|
| language | bash |
|---|
| title | rsync (local directory) |
|---|
|
mkdir -p $SCRATCH/data
cds
rsync -avWPavW $CORENGS/custom_tracks/ data/custom_tracks/ |
...
| Code Block |
|---|
|
rsync -avWPavW /work/projects/BioITeam/projects/courses/Core_NGS_Tools/custom_tracks/ data/custom_tracks/ |
...
| Code Block |
|---|
| title | single remote file copy with scp |
|---|
|
mkdir -p $SCRATCH/data/test1
cds
cat $CORENGS/tacc/dragonfly_access.txt
cds
mkdir -p data/test2
scp corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/progeria_ctcf.vcf.gz ./data/test1test2/
ls ./data/test1test2 |
Notes:
- The 1st time you access a new host the SSH security prompt will appear
- You will be prompted for your remote host password
- The -r recursive argument works for scp also, just like for cp
...
| Code Block |
|---|
| language | bash |
|---|
| title | rsync (remote directory) |
|---|
|
rsync -avWPavW corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/ ~/scratch/data/custom_tracks/ |
...
| Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
|
cd
cp -r /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what
# or using the $CORENGS environment variable
cp -r $CORENGS/linuxpractice/what what
cd what
cat readme |
Where are you when you're all done?
| Expand |
|---|
|
ls5stamp2:~/what/starts/here/changes/the/world
|
...
| Expand |
|---|
|
From inside your ~/what directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| mkdir starts
cd starts
cp /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/steps/nextInstr .
cat nextInst |
|
| Expand |
|---|
|
From inside your ~/what/starts directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| mkdir here
cd here
wget http://web.corral.tacc.utexas.edu/BioinformaticsResource/BioITeamCoreNGS/step3.txt
cat step3.txt |
|
| Expand |
|---|
|
From inside your ~/what/starts/here directory: | Code Block |
|---|
| title | Play a scavenger hunt for more practice |
|---|
| scp -r /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# or
rsync -ptrvP /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# Note: rsync -avP ... will also work, but will report an error because the destination file and
# directory ownership cannot be changed to match the source. But the files will be copied, and
# ownership assigned to you.
# Then
cd changes
more largeFile.txt |
|
...