Versions Compared

Old Version 39

changes.mady.by.user Anna M Battenhouse

Saved on

compared with

New Version Current

changes.mady.by.user Anna M Battenhouse

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • wget – retrieves the contents of an Internet URL
  • cp – copies directories or files located on any local file system
  • scp – copies directories or files to/from a remote system
  • rsync – copies directories or files on either local or remote systems

(Read more about Copying files and directories)

TACC storage areas and Linux commands to access data
(all commands to be executed at TACC except
laptop-to-TACC copies, which must be executed on your laptop)

...

Code Block
--------------------- Project balances for user abattenh ----------------------
| Name           Avail SUs     Expires | Name           Avail SUs     Expires |
| OTH21095             905  2023-09-30 | MCB21106            1496  2023-09-30 |
| OTH21164             215  2024-05-31 | OTH21180             899  2024-03-31 |
------------------------ Disk quotas for user abattenh ------------------------ 
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /scratch            0.7       0.0     0.00          567           0    0.00 |
| /home1              0.0      11.7     0.01          232           0    0.00 |
| /work             169.0    1024.0    16.50        79361     3000000    2.65 |
-------------------------------------------------------------------------------

changing TACC file systems

When you first login, you start in your Home directory. Use the cd, cdw and cds commands to change to your other file systems. Notice how your command prompt helpfully changes to show your location.

...

Well, you don't have a desktop at TACC to "Save as" to, so what to do with a link? The wget program knows how to access web URLs such as http, https and ftp. Anchorwget yeast datawget yeast data

wget

Get ready to run wget from the directory where you want to put the data.

...

Code Block
languagebash
titleSingle file copy with cp
mkdir -p $SCRATCH/data/test1
cp $CORENGS/misc/small.fq  $SCRATCH/data/test1/
ls $SCRATCH/data/test1

# or..
cds
mkdir -p data/test1
cd data/test1
cp $CORENGS/misc/small.fq .

# or..
mkdir -p ~/scratch/data/test1   # use the symbolic link in your Home directory
cd ~/scratch/data/test1
cp $CORENGS/misc/small.fq  .
ls

(Read more about using Absolute or Relative pathname syntax)

Now copy Copy an entire directory to your Scratch area. The -r option says "recursive" recursive.

Code Block
languagebash
titleDirectory copy with cp
mkdir -p $SCRATCH/data 
cds
cd data
cp -r $CORENGS/general/ general/

...

Expand
titleHint
ls general
# or
tree $SCRATCH/data/general


Expand
titleAnswer
BEDTools-User-Manual.v4.pdf  SAM1.pdf  SAM1.v1.4.pdf

...

Tip
titleAlways add a trailing slash ( / ) after directory names

The trailing slash ( / ) on the source and destination directories are very important for rsync( – and for other Linux copy commands also)!

rsync will create the last directory level for you, but earlier levels must already exist.

...

Tip

The bash shell has several convenient line editing features:

  • use the Up arrow to scroll back through the command line history; Down arrow goes forward
  • use Ctrl-a to move the cursor to the beginning of a line; Ctrl-e to the end
  • use Backspace to remove text before the cursor; Delete to remove text after the cursor
  • Ctrl-k ("kill") to delete all text on your command line after the cursor
  • Ctrl-y ("yank") to copy the last killed text to where the cursor is

Once the cursor is positioned where you want it:

  • Just type in any additional text you want
  • To delete text after the cursor, use:
    • Delete key on Windows
    • Function-Delete keys on Macintosh
  • To delete text before the cursor, use:
    • Backspace key on Windows
    • Delete key on Macintosh

(Read more about Command line history and editing)

Copy from a remote computer - Copy from a remote computer - scp or rsync

Provided that the remote computer is running Linux and you have ssh access to it, you can use various Linux commands to copy data over a secure connection.

...

Code Block
titlesingle remote file copy with scp
cat $CORENGS/tacc/dragonfly_access.txt
cds
mkdir -p data/test2
scp -p corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/progeria_ctcf.vcf.gz ./data/test2/
tree $SCRATCH./data/test2

Notes:

  • The 1st time you access a new host the SSH security prompt will appear
  • You will be prompted for your remote host password
    • for security reasons characters will not be echoed
  • The  -r recursive argument works for scp also, just like for cp
  • The  -p argument says to preserve the file's last modification time
    • otherwise the last modification time of the local copy will be when the copy was done

remote rsync

rsync can be run just like before, but using the remote-host syntax. Here we use two tricks:

  • The tilde ( ~ ) at the start of the path means "relative to my home Home directory"
  • We use the tilde ( ~ ) in the destination to traverse the ~/scratch symbolic link created in your home directory.

Code Block
languagebash
titlersync (remote directory)
cat $CORENGS/tacc/dragonfly_access.txt
rsync -avrW corengstools@dragonfly.icmb.utexas.edu:~/custom_tracks/ ~/scratch/data/custom_tracks/

...