...
- 2U, Gigabyte G291 enclosures https://www.gigabyte.com/GPU-Server/G291-Z20-rev-100#ov
- 8 AMD Radeon Instinct MI50 GPUs (https://www.amd.com/en/products/professional-graphics/instinct-mi50)
- each with 32 GB HBM2 memory
- 48-core/96-hyperthread EPYC 7642 CPU
- 512 GB RAM
- 1.9 TB Samsung NVMe SSD
Resources
ROCm GPU-enabling framework
...
- Introduction_to_AMD_7002_processor.pdf
- Radeon_Instinct_HPC_Training_2020.pdf
- Radeon_Instinct_ML_Training_2020.pdf
Command-line diagnostics
- GPU usage: rocm-smi
- CPU and GPU details: rocminfo
- What ROCm modules are installed: dpkg -l | grep rocm
- GPU ↔ GPU/CPU communication bandwidth test
- between GPU2 and CPU: rocm-bandwidth-test -b2,0
- between GPU3 and GPU4: rocm-bandwidth-test -b3,4
Sharing Resources
Since there's no batch system on BRCF POD compute servers, it is important for users to monitor their resource usage and that of other users in order to share resources appropriately.
- Use top to monitor running tasks (or top -i to exclude idle processes)
- commands while top is running include:
- M - sort task list by memory usage
- P - sort task list by processor usage
- N - sort task list by process ID (PID)
- T - sort task list by run time
- 1 - show usage of each individual hyperthread
- they're called "CPUs" but are really hyperthreads
- this list can be long; non-interactive mpstat may be preferred
- Use mpstat to monitor overall CPU usage
- mpstat -P ALL to see usage for all hyperthreads
- mpstat -P 0 to see specific hyperthread usage
- Use free -g to monitor overall RAM memory and swap space usage (in GB)
- Use rocm-smi to see GPU usage