mysqld using swap, not ram

I have 4GB of swap space on my server, and 1GB of ram. According to “htop” (a nicer version of “top”), the system rarely uses more than about 40-50% of the ram, but about 300-500MB of swap space.

My mysqld server is causing a lot of lag at busy times, due to the size of the database I have… But when I increase the cache sizes, it just uses more swap, not ram… Which is a waste of time, as it’s still on disk.

How can I stop it using swap space, and make it use the other unused half of my ram?