Have you tried looking into HTTP? That was my problem that I mentioned earlier. I had a poorly written PHP script :o for sending mail. Someone was able to inject their own email into it and used my server to send out a bunch of spam.
I would look at your HTTP logs and also do a “top” to see which process are being used that should give you some more clues.
Possibly do a netstat to see which ports are open too?