If you must use swap (you probably don't need to), then at least set zswap.enabled=1 in the kernel boot options, typically in grub. This will enable lzo compression of swap in memory so there is less writing to disk. Some newer kernels use lz4.

Either way, what most folks are actually missing is the correct kernel settings for the amount of memory they have. Sadly, the kernel does not dynamically adjust these based on your total amount of memory. Below is based loosely on Redhat suggestions. Please note, that if you fall below the min free, the kernel will decide what to do next, based on your oom and panic settings. The settings below will have the kernel free cache and other memory earlier so that you do not hit those stalling conditions and can even prevent some OOM race conditions. Some might suggest tuned, but use tuned with caution or at least read up on everything it does.

    MEM=`grep ^MemTotal /proc/meminfo | awk {'print $2'}`
    if   [ ${MEM} -gt 1129241478 ] ; then
        sysctl -q -w vm.min_free_kbytes=16384000
    elif [ ${MEM} -gt 564620739 ] ; then
        sysctl -q -w vm.min_free_kbytes=8192000
    elif [ ${MEM} -gt 352887962 ] ; then
        sysctl -q -w vm.min_free_kbytes=4096000
    elif [ ${MEM} -gt 176443981 ] ; then
        sysctl -q -w vm.min_free_kbytes=1024000
    elif [ ${MEM} -gt 88221990 ] ; then
        sysctl -q -w vm.min_free_kbytes=524288
    else
        sysctl -q -w vm.min_free_kbytes=262144
    fi

If you have small VM's, then perhaps set the default above to something a little smaller. You can of course free up about 128MB on default installations by removing "crashkernel" from your grub config and rebooting.

Then do this regardless, because overcommit set to 0 does not mean off, so we set the ration to 0 as well. Overcommit is good for developers testing code and finding the correct ways to manage memory in their applications during development.

    sysctl -q -w vm.overcommit_ratio=0

And of course, cache pressure plays into early evacuation of the right cache based on your usage:

    ## default is 100 (optimal for file servers).  4000+ for in memory databases.
    ## 10000 means always prefer page cache.
        sysctl -q -w vm.vfs_cache_pressure=1000

And if you have people oversubscribing a lot: (adjust based on your memory capacity)

    sysctl -q -w vm.admin_reserve_kbytes=131072
    sysctl -q -w vm.user_reserve_kbytes=262144

Please do read up on all of these before testing on your test machines. [1]

[1] https://www.kernel.org/doc/Documentation/sysctl/

Then finally, make sure you have Transparent Huge Pages disabled unless you know for sure you need it. THP can leak a lot of memory and it is nearly impossible to see without extensive kernel debugging.

In grub, set this and reboot:

    transparent_hugepage=madvise

Or to manually disable THP during run-time,

    echo -n "madvise" > /sys/kernel/mm/transparent_hugepage/enabled
    echo -n "never" > /sys/kernel/mm/transparent_hugepage/defrag

Then restart your applications. THP defrag can also cause stalling and lag spikes, especially in large memory java deployments (it will look like FGC's) and in MongoDB, Cassandra, others.

If you did this manually, stop your apps, flush cache, compact memory, then start your apps.

    sync;sync;sync
    echo 3 > /proc/sys/vm/drop_caches
    echo 1 > /proc/sys/vm/compact_memory

Some will say 3 sync's is not required. This is mostly true, but some old raid controllers treat this differently.