Is there any way i can see what the swap is full of, or why it is full when there is plenty memory available?
It might relate to what VM technology your provider is using. Many VM technologies use a balloon driver which allows the system to steal memory from your VM to allocate it to others. Other things like tmpfs can also use it.
On one of our systems, we have some big VMs, and if I look for a line called VmSwap in the /proc/pid/status file, I can see this is responsible for almost all my swap useage:
#ps v -u qemu PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 7444 ? Sl 19393:04 4066 2818 34398217 33652364 25.4 /usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 32768 -smp 2,sockets=2,cores=1,threads=1 8380 ? Sl 13157:03 5288 2818 68921177 66595700 50.3 /usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 65536 -smp 8,sockets=8,cores=1,threads=1
#cat /proc/8380/status | grep ^VmSwap VmSwap: 646248 kB
# free total used free shared buffers cached Mem: 132142972 131448352 694620 0 257232 26698436 -/+ buffers/cache: 104492684 27650288 Swap: 262143984 673900 261470084
Can anyone come up with a script to sort this output for all processes by named process\pid?