...
- Recall that there are 24 physical (48 virtual) cores and 64 GB of memory on each compute node
- so technically you can run up to 48 commands on a node, each with ~1.3 GB available memory
- you can run fewer tasks, and if you do, each task gets more resources
- Because bioinformatics programs generally require more memory and fewer cores, launcher_makercreator.py sets a 24 cores/node maximum.
| tasks per node (wayness) | cores available to each task | memory available to each task | |||
|---|---|---|---|---|---|
| 1 | 24 | 64 GB | |||
| 2 | 12 | 32 GB | |||
| 3 | 8 | 21.3 GB | |||
| 4 | 6 | 16 GB | |||
| 6 | 4 | 10.6 GB | |||
| 8 | 3 | 8 GB | |||
| 12 | 2 | 5.3 GB | |||
| 24 | 1 | 2.6 GB | 48 | 1/2 (one virtual core) | 1.3GB |
- In launcher_creator.py, wayness is specified by the -w argument.
- the default is 24 (one task per core)
- A special case is when you have only 1 command in your job.
- In that case, it doesn't matter what wayness you request.
- Your job will run on one compute node, and have all 24 cores available.
...
Wayness example
Let's use launcher_makercreator.py to explore wayness options. First copy over the wayness.cmds commands file:
| Code Block | ||||
|---|---|---|---|---|
| ||||
cds mkdir -p $SCRATCH/core_ngs/slurm/wayness cd $SCRATCH/core_ngs/slurm/wayness cp $CORENGS/tacc/wayness.cmds . |
Exercise: How many tasks are specified in the wayness.cmds file?
| Expand | ||
|---|---|---|
| ||
wc --help |
| Expand | |||||||
|---|---|---|---|---|---|---|---|
| |||||||
Find the number of lines in the wayness.cmds commands file:
The file has 24 lines, representing 24 tasks. |
The wayness.cmds commands file consists of a number of 24 identical lines that look like this:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
launcher_makercreator.py -nj wayness.cmds -n wayness -w 8 -t 0:10 -v00:00:20 -a UT-2015-05-18 sbatch -q dev sbatch -reservation=intro_NGS wayness.slurm showq -u |
Exercise: With 24 tasks requested and wayness of 8, how many nodes will this job require? How much memory will be allocated to each task?
...
| Code Block | ||
|---|---|---|
| ||
Command 1 of 24 (8 per node) ran on node nid00023nid00020 core 210. Command 2 of 24 (8 per node) ran on node nid00023nid00020 core 191. Command 3 of 24 (8 per node) ran on node nid00023nid00020 core 172. Command 4 of 24 (8 per node) ran on node nid00023nid00020 core 183. Command 5 of 24 (8 per node) ran on node nid00023nid00020 core 164. Command 6 of 24 (8 per node) ran on node nid00023nid00020 core 225. Command 7 of 24 (8 per node) ran on node nid00023nid00020 core 206. Command 8 of 24 (8 per node) ran on node nid00023nid00020 core 237. Command 9 of 24 (8 per node) ran on node nid00022nid00021 core 148. Command 10 of 24 (8 per node) ran on node nid00022nid00021 core 109. Command 11 of 24 (8 per node) ran on node nid00022nid00021 core 1210. Command 12 of 24 (8 per node) ran on node nid00022nid00021 core 1511. Command 13 of 24 (8 per node) ran on node nid00022nid00021 core 1312. Command 14 of 24 (8 per node) ran on node nid00022nid00021 core 1113. Command 15 of 24 (8 per node) ran on node nid00022nid00021 core 814. Command 16 of 24 (8 per node) ran on node nid00022nid00021 core 915. Command 17 of 24 (8 per node) ran on node nid00021nid00022 core 416. Command 18 of 24 (8 per node) ran on node nid00021nid00022 core 217. Command 19 of 24 (8 per node) ran on node nid00021nid00022 core 518. Command 20 of 24 (8 per node) ran on node nid00021nid00022 core 019. Command 21 of 24 (8 per node) ran on node nid00021nid00022 core 320. Command 22 of 24 (8 per node) ran on node nid00021nid00022 core 121. Command 23 of 24 (8 per node) ran on node nid00021nid00022 core 622. Command 24 of 24 (8 per node) ran on node nid00021nid00022 core 723. |
Notice that there are 3 different host names, each of which ran 8 tasks:
...
should produce this output (read more about piping commands to make a histogram)
| Code Block | ||
|---|---|---|
| ||
8 nid00021nid00020 8 nid00022nid00021 8 nid00023nid00022 |
Some best practices
Redirect task output and error streams
...
Another really good way to work is to "bundle" a complex set of steps into a shell script that sets up its own environment, loads its own modules, then executes a series of program steps. You can then just call that script, probably with data-specific arguments, in your commands file. This multi-program script is sometimes termed a pipeline, although complex pipelines may involve several such scripts.
For example, you might have a script called align_bwa.sh (a bash script) or align_bowtie2.py (written in python) that performs multiple steps needed during the alignment process:
...
Because a single job can create a lot of files, it is a good idea to use a different directory for each job or set of closely related jobs, maybe with a name similar to the job being performed. This will help you stay organized.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
idev -p development -m 20 -A UT-2015-05-18 -N 1 -n 24 --reservation=CCBBintro_NGS |
Notes:
- -p development requests nodes on the development queue
- -m 20 asks for a 20-minute session (120 minutes is the maximum for development)
- -A UT-2015-05-18 specifies the TACC allocation/project to use
- -N 1 asks for 1 node and -n 24 requests access to 24 cores
- --reservation=CCBBintro_NGS gives us priority access to TACC nodes for the class. You normally won't use this argument.
When you ask for an idev session, you'll see output as shown below. Note that the process may pause while it waits for available nodes.
...
- the hostname command on a login node will return a login server name like login2
- while in an idev session hostname returns a compute node name like nid00011
- you cannot submit a batch job from inside an idev session, only from a login node
- your idev session will end when the requested time has expired
- or you can just type exit to return to a login node session
...