Server Optimization and Security Hardening

General Interworx Control Panel Info General Inquries
1

We work with a local company of Linux gurus that helps out with network security and they’re also very good with web hosting environments. Here are some suggestions they gave us but since they’re unfamiliar with Iworx they’re not sure what potential problems these changes would have. I’d love to get some feedback so I can decide how to proceed:

SECURITY CONCERNS

  1. Users have UNIX accounts, though they seem to be configured well, from a security standpoint, they are unnecessary.

  2. phpMyAdmin 2.6.2 is installed on the system.

  3. The system could potentially be vulnerable to a DoS attack, in theory, if one were to deliver a vast quantity of messages to the postmaster@ abuse@ or other RFC-required account that contained files that were large, long message bodies, etc. This would cause ClamAV to scan each message (taking more memory and CPU time), and SpamAssassin scanning them (which takes more memory and CPU time.) While observing, we noticed load tends to reach >= 1.00 (5 min avg.) unnecessarily at times.

OPTIMIZATION SUGGESTIONS

  1. Apache may see a potential performance gain by using mapped vhosts as opposed to user-based vhosts.

  2. Apache should be using the Worker MPM model given the type of processor present. The default config for this module is untuned. There would be a potential performance increase by reconfiguring apache in this instance.

  3. PHP Version 5.1.1 is installed. There may be benefits by upgrading to the latest stable release (5.2.1).

  4. MySQL should be updated to the latest release (5.x) for optimal performance. (4.1.20 is being used by users and 4.0.21 by iworx)

  5. Replace syslogd & klogd with the more-optimized syslog-ng

  6. If all domains will utilize Guardian MX then SpamAssassin and ClamAV are a waste of resources. (Note: Guardian MX is a service they provide for some of our domains. Is there an easy way to completely disable SpamAssassin and ClamAV?)

  7. Vixie-cron is more versatile than atd.

  8. Utilizing tpop3d would incur less overhead than vpopmail.

  9. The latest version of BIND 9 would use less overhead and would be more secure then tinydns.

  10. PHP sometimes uses massive amounts of CPU time (~20-30%) & SpamAssassin eats about 30% CPU each scan. (If either of these were hammered, it could cause a DoS condition)

  11. iworx consumes vast amounts of memory on the system: It runs large numbers of Apache (iworx-web) and mySQL (iworx-db) processes in addition to those that are already on the system. This results in unnecessary load. (This could also result in DoS if hammered)

  12. iworks uses its own separate application of apache and mysql. Optimization could occur if these could be merged to use a single version of the applications.

I already realize some of their suggestions and findings would not be possible since I’m sure some of them would truly mess up Iworx. But, I’d like to know which ones and why Iworx is running certain services that are less secure. Overall I like the control panel really well but I’d hate to think my server is less secure and less robust simply because I’m running Iworx.

2

[quote=whoisjb;11636]We work with a local company of Linux gurus that helps out with network security and they’re also very good with web hosting environments. Here are some suggestions they gave us but since they’re unfamiliar with Iworx they’re not sure what potential problems these changes would have. I’d love to get some feedback so I can decide how to proceed:

SECURITY CONCERNS

  1. Users have UNIX accounts, though they seem to be configured well, from a security standpoint, they are unnecessary.

  2. phpMyAdmin 2.6.2 is installed on the system.

  3. The system could potentially be vulnerable to a DoS attack, in theory, if one were to deliver a vast quantity of messages to the postmaster@ abuse@ or other RFC-required account that contained files that were large, long message bodies, etc. This would cause ClamAV to scan each message (taking more memory and CPU time), and SpamAssassin scanning them (which takes more memory and CPU time.) While observing, we noticed load tends to reach >= 1.00 (5 min avg.) unnecessarily at times.

OPTIMIZATION SUGGESTIONS

  1. Apache may see a potential performance gain by using mapped vhosts as opposed to user-based vhosts.

  2. Apache should be using the Worker MPM model given the type of processor present. The default config for this module is untuned. There would be a potential performance increase by reconfiguring apache in this instance.

  3. PHP Version 5.1.1 is installed. There may be benefits by upgrading to the latest stable release (5.2.1).

  4. MySQL should be updated to the latest release (5.x) for optimal performance. (4.1.20 is being used by users and 4.0.21 by iworx)

  5. Replace syslogd & klogd with the more-optimized syslog-ng

  6. If all domains will utilize Guardian MX then SpamAssassin and ClamAV are a waste of resources. (Note: Guardian MX is a service they provide for some of our domains. Is there an easy way to completely disable SpamAssassin and ClamAV?)

  7. Vixie-cron is more versatile than atd.

  8. Utilizing tpop3d would incur less overhead than vpopmail.

  9. The latest version of BIND 9 would use less overhead and would be more secure then tinydns.

  10. PHP sometimes uses massive amounts of CPU time (~20-30%) & SpamAssassin eats about 30% CPU each scan. (If either of these were hammered, it could cause a DoS condition)

  11. iworx consumes vast amounts of memory on the system: It runs large numbers of Apache (iworx-web) and mySQL (iworx-db) processes in addition to those that are already on the system. This results in unnecessary load. (This could also result in DoS if hammered)

  12. iworks uses its own separate application of apache and mysql. Optimization could occur if these could be merged to use a single version of the applications.

I already realize some of their suggestions and findings would not be possible since I’m sure some of them would truly mess up Iworx. But, I’d like to know which ones and why Iworx is running certain services that are less secure. Overall I like the control panel really well but I’d hate to think my server is less secure and less robust simply because I’m running Iworx.[/quote]

UNIX shell accounts are disabled and bound to a noshell by default, which is exactly the same as not having one at all… moot point.

phpmyadmin is yes, a concern, but at the same time this should not be an issue if it is kept outside of the normal user view. If you get compromised there, you should be compromised entirely… because the full access there is well, mysql root access. I’d hope you were protecting that aspect with a secure enough password anyways - so in this area, if you get “hacked” its because of a weak password, NOT because of version.

The clamAV situation is also moot because if you are getting a significant # of emails to the abuse/postmaster accounts you should already have rules protecting against clamAV scanning and just simply deleting them.

All the software recommendations are simply opinions…

And, just fyi…

Tinydns is in fact more secure (and more efficient) than BIND ever has been =/ This has been proven on many levels, in just about every security paper released in hosting environments.

PHP version is moot…

MySQL - this issue is also moot, but I will address it. MySQL 4.1 and 5.0 are in tandem in terms of security. 5.0 is a completely new version and 90% of all scripts that utilize 5.0 are broken due to different data storage types, password storage, protocol interaction and furthermore… undeveloped plugins in interaction between applications produced. PHP support is still not fully out of beta, as well as well… most other web-application languages. MySQL 4.1 is slightly slower, but not by much. The only added performance to MySQL 5.0 is the cache changes, which you wont notice unless you’re running an oracle-level size database. This feature was put in to reduce the time between MASSIVELY executied queries, on the level of thousands per second.

Sorry but, what about the syslog? That… really does not make any sense. That is kernel level and completely… well, yea.

“GuardianMX?” I’ve never heard of it, and really… although I’m not a fan of spamassasin, I’m really not a fan of external services :slight_smile:

If you have your firewall properly configured (which, is something that is not too hard to do), a DoS can be completely avoided on any port, any web client. Its rather simple to do, its exactly the same as blocking consecutive failures in SSH.

Also…

Just FYI…

The reason I chose iworx is because of the seperation of applications between what it uses and what everything else uses. In cpanel, combining these applications is what drove me away. I could not update php to my own desires, i could not optimize apache to my own desires, and i could not take the aspects that I desire into my own hands to properly configure the way that I wanted. With iworx, I can pretty much create a server to my own standards, security preferences, and optimizations… without having to worry about my panel application dying on me. This alone pretty much makes the entire situation you provide a little absent =/ You can configure all the things you said, from the command line, with a little work and a little research. I highly recommend this if you are using an outside administration source, as it could make you more knowledgeable about the system and in the end save you some well earned money.

I’ll leave it at this…

It is easier to DoS a Plesk or cPanel system, than it is to DoS the php4 running, “high resource usage” iworx system… and this is a fact that I proved in my own testing.

cpanel still uses apache 1.3.3.7, which, as much as people keep saying, is nowhere near being updated. That… is alone an issue.

Plesk is a java heavy, database heavy, very inefficiently coded well (mess). Java alone in that aspect… I hope you know the security issue in that being within a panel. If you dont, then the DoS issues you relate to need a little research done first :slight_smile:

3

Thanks for the detailed response. I’d like to hear something from the Iworx team as well. I just don’t want to make any suggested changes that could “break” Iworx or screw up any future updates to the control panel. I’m not trying to start any kind of argument for or against Iworx. I’ve used both Plesk and cPanel and so far, I like Iworx much better. So, the intention here is not to get devoted Iworx users on the defensive. Just looking for some honest feedback.

Some of their suggestions are simply OS specific and not anything to do with the control panel. For example, they prefer syslog-ng for whatever reason, but your response doesn’t clarify whether changing this would have any impact on Iworx.

Thanks in advance for everyone’s input!

4

[quote=whoisjb;11647]Thanks for the detailed response. I’d like to hear something from the Iworx team as well. I just don’t want to make any suggested changes that could “break” Iworx or screw up any future updates to the control panel. I’m not trying to start any kind of argument for or against Iworx. I’ve used both Plesk and cPanel and so far, I like Iworx much better. So, the intention here is not to get devoted Iworx users on the defensive. Just looking for some honest feedback.

Some of their suggestions are simply OS specific and not anything to do with the control panel. For example, they prefer syslog-ng for whatever reason, but your response doesn’t clarify whether changing this would have any impact on Iworx.

Thanks in advance for everyone’s input![/quote]

From what I have understood from my own questions about customizing my server, iworx depends on the following things:

apache
mysql

Beyond that, the sky is the limit.

BTW, I meant no disrespect to you or your friends, I was just providing some constructive criticism. Sometimes software opinions can be factually wrong… like in the BIND case. Recently a number of the larger managed hosting companies have switched from BIND to tinydns due to its better handling of larger DNS databases, and the fact that in BIND it is easier to well, DoS a system yourself with very large DNS databases for 10000’s of servers.

5

I asked him about the tinydns issue and he said it will spawn a number of processes where BIND only creates one. So, he must see tinydns as using more resources than necessary. Thanks again for the quick reply!

6

The more processes spawned does not mean more resources, it is just simply child processes within the same parent process. This is in fact more secure and uses less resources, because the maintenance overhead for the system is quite a bit lower. The reason for this is because each child process can be closed and reopened on its own without having to restart the entire application, and idle processes can be closed and reopened at any time. When an entire process runs the entire application, the overhead for that application is always at its maximum because it is never idle, so in reality the entire server is constantly running. With tinydns, this is not the case. For a domain that say, goes almost entirely inactive at non-peak hours, tinydns basically shuts off the service for that domain and only responds when needed, usually with a seperate child process that provides recursive features :slight_smile:

The same rule applies to apache child processes as well as mysql child processes.

Forking multiple processes under the same parent is one of the reasons windows fails at keeping its resources low - the system doesnt allow for this, thus everything is running at 100% 100% of the time :slight_smile:

7

Just to give you a second opinion independent of anytihng previously said (otherwise, it wouldn’t really be 2nd, would it?)

[QUOTE=whoisjb;11636]SECURITY CONCERNS

  1. Users have UNIX accounts, though they seem to be configured well, from a security standpoint, they are unnecessary.[/quote]
    For serving up web sites, yes. BUT BUT BUT! It lets CGI scripts run in a more isolated fashion. A common problem is some code wanting web server writable files/directories. If all code is run as the same user as the web sever (the case with PHP :frowning: ) then one user’s writable files become writeable by other users! That is a security concern, but just the way it goes with shared web hosting.

With PHP we have to use world writable directories, that any other use on the system can write to, but CGI + suExec allows us to use user/group writable instead, preventing this issue. Locking a user (be it UNIX user or otherwise) to thier own universe is a security boon. Plus, IWorx created users can’t really do anything by default.

Cron scrips get the same isolation, and other goodies.

  1. phpMyAdmin 2.6.2 is installed on the system.

There are a lot of things installed. Software has problems. The benifits of giving phpMyAdmin access to users outweighs the problems of having it installed. Besides, if you want to claim this is an issue, you need to look no further than PHP iself! Ick!

  1. The system could potentially be vulnerable to a DoS attack, in theory, if one were to deliver a vast quantity of messages to the postmaster@ abuse@ or other RFC-required account that contained files that were large, long message bodies, etc. This would cause ClamAV to scan each message (taking more memory and CPU time), and SpamAssassin scanning them (which takes more memory and CPU time.) While observing, we noticed load tends to reach >= 1.00 (5 min avg.) unnecessarily at times.

Very true. But what if someone sends millions of virus-laden e-mails to a mailboxes on your system. You wind up with storage/quota exploiting DoS anyway. Either way, this can be disabled. Again, the good of having it outweighs the problems of dealing with spam and viruses. If you can reject spam and viruses at the MTA level, they don’t get put into permanant storage. Like all computer security issues, you just choose what problems you would rather have.

OPTIMIZATION SUGGESTIONS

  1. Apache may see a potential performance gain by using mapped vhosts as opposed to user-based vhosts.

Absolutely. I’m not a fan of that way subdomains are done, either. BUT, the current method lets you control quite a bit on a per-domain basis instead of one policy across the entire user base. This is better from a business and security standpoint. Note user account isolation above. Again, where do you want your problem. I like being able to control anything I want on a per-domain basis. Its been a lifesaver.

  1. Apache should be using the Worker MPM model given the type of processor present. The default config for this module is untuned. There would be a potential performance increase by reconfiguring apache in this instance.

I haven’t looked into this so I can’t comment.

  1. PHP Version 5.1.1 is installed. There may be benefits by upgrading to the latest stable release (5.2.1).

If you choose PHP 5. PHP is a lot of great things. What it isn’t in a lot of cases is backwards compatable. PHP4 scripts don’t always run in PHP5. Same goes for different PHP5 versions if I recall correctly. Latest isn’t always greatest. The fact is that no matter what version of PHP you are running, chances are there are issues with it. So, choice should be the major and minor version you want, taken to the latest release from there.

  1. MySQL should be updated to the latest release (5.x) for optimal performance. (4.1.20 is being used by users and 4.0.21 by iworx)

Same deal, minus the problems.

  1. Replace syslogd & klogd with the more-optimized syslog-ng

Guys correct me if I’m wrong but IWorx will just use whatever is installed on the system. IWorx takes less control over the rest of the system as other panels do (thankfully). This is really a host OS issue and not an IWorx issue.

  1. If all domains will utilize Guardian MX then SpamAssassin and ClamAV are a waste of resources. (Note: Guardian MX is a service they provide for some of our domains. Is there an easy way to completely disable SpamAssassin and ClamAV?)

Run what you want to run. You can disable spam and virus filtering from IWorx.

  1. Vixie-cron is more versatile than atd.

Same as the syslog issue.

  1. Utilizing tpop3d would incur less overhead than vpopmail.

I can’t comment on the choice of vpopmail. I just don’t know.

  1. The latest version of BIND 9 would use less overhead and would be more secure then tinydns.

I thought TinyDNS had less security problems, and better performance than BIND. This was awhile ago and I haven’t kept up. Also, does BIND do SQL back-end? I don’t really know, so I suppose I shouldn’t comment.

  1. PHP sometimes uses massive amounts of CPU time (~20-30%) & SpamAssassin eats about 30% CPU each scan. (If either of these were hammered, it could cause a DoS condition)

I haven’t seen numbers that high myself. As far as PHP is concerned, its really an issue of what code is being run.

  1. iworx consumes vast amounts of memory on the system: It runs large numbers of Apache (iworx-web) and mySQL (iworx-db) processes in addition to those that are already on the system. This results in unnecessary load. (This could also result in DoS if hammered)

Again, I’ve never seen a problem. Its just as necessary to have a responsive control interface as it is to have a responsive web site.

  1. iworks uses its own separate application of apache and mysql. Optimization could occur if these could be merged to use a single version of the applications.

They are different instances so problems with the web server and database users use won’t kill the control panel side. If a users is killing something, the administrator (or the user) can still go in throguh the control panel and wrestle back control.

I already realize some of their suggestions and findings would not be possible since I’m sure some of them would truly mess up Iworx. But, I’d like to know which ones and why Iworx is running certain services that are less secure. Overall I like the control panel really well but I’d hate to think my server is less secure and less robust simply because I’m running Iworx.

Oh goodness don’t get me started on the “more secure” “less secure” thing. I’m not even going to touch that.

Anyway, over all IWorx rocks. Your guru guys have points, but to me carry little weight. It doubt they considered the other side to each point they made.

8

No offense taken guys. We always welcome feedback and constructive criticism. :slight_smile:

Offbeatadam actually did a pretty good job of summing up most of reasons why we decided what we decided. And, like he said, you’re free to turn off most things you don’t need. For example, you can disable phpMyAdmin, and you can edit the conf files in /home/interworx/etc/httpd to change how many running threads iworx-web uses.

One thing I should mention though, is that using the Apache Worker MPM isn’t suggested. PHP thread support is still experimental:
http://www.php.net/manual/en/install.unix.apache2.php

Also, if all your clients are going to be using this Guardian MX, then you should be able to shutdown SpamAssassin and ClamAV.

Socheat

9

Yes, this is true. We made a few of our packages for a few things (qmail, simscan, etc), but much of the underlying system stuff, like syslog, we leave to the distribution.

10

I’ll go ahead and respond to each of this issues in turn to define our “official” stance. I’m going to steal, I mean, copy some responses from OffBeatAdam, CMI, and Socheat since they did a good job responding :slight_smile:

  1. Users have UNIX accounts, though they seem to be configured well, from a security standpoint, they are unnecessary.

UNIX shell accounts are disabled and bound to a noshell by default, which is exactly the same as not having one at all, in terms of “login ability”. Second, some hosts give shell accounts to some (trustworthy) customers, so this is necessary. Finally, CGI+suexec uses linux accounts to work. This isn’t a feature we’re going to get rid of, nor should we.

  1. phpMyAdmin 2.6.2 is installed on the system.

We’re updating the phpMyAdmin version in the next release if the version number was the concern, and phpMyAdmin is a vital offering for folks using databases, so it’s not going anywhere.

  1. The system could potentially be vulnerable to a DoS attack, in theory, if one were to deliver a vast quantity of messages to the postmaster@ abuse@ or other RFC-required account that contained files that were large, long message bodies, etc. This would cause ClamAV to scan each message (taking more memory and CPU time), and SpamAssassin scanning them (which takes more memory and CPU time.) While observing, we noticed load tends to reach >= 1.00 (5 min avg.) unnecessarily at times.

They were correct when they said “in theory”. Lots of things can happen in theory :). SpamAssassin and ClamAV can be disabled of course if an issue arises.

  1. Apache may see a potential performance gain by using mapped vhosts as opposed to user-based vhosts.

We consider any potential gain minimal, and the benefits of doing it our way outweigh it.

  1. Apache should be using the Worker MPM model given the type of processor present. The default config for this module is untuned. There would be a potential performance increase by reconfiguring apache in this instance.

PHP is considered “experimental” when running under the Worker MPM, and some commonly used PHP extensions are NOT thread safe, and could bring down the webserver if used under the MPM model. (GD for example). See: PHP: Apache 2.x on Unix systems - Manual

  1. PHP Version 5.1.1 is installed. There may be benefits by upgrading to the latest stable release (5.2.1).

PHP is a lot of great things. What it isn’t in a lot of cases is backwards compatible. PHP4 scripts don’t always run in PHP5. Same goes for different PHP5 versions if I recall correctly. Latest isn’t always greatest. The fact is that no matter what version of PHP you are running, chances are there are issues with it. So, choice should be the major and minor version you want, taken to the latest release from there.

  1. MySQL should be updated to the latest release (5.x) for optimal performance. (4.1.20 is being used by users and 4.0.21 by iworx)

If it ain’t broke (don’t rush) to fix it. You can install whatever MySQL version you decide to for the users on your system. We’ve kept the (completely separate) iworx mysql instance at 4.0.21 because it has been stable, and hasn’t caused problems. We very well may update it (but probably not before we see a good reason to). We’d hate to update the iworx-mysql version just to cause unexpected problems for folks.

  1. Replace syslogd & klogd with the more-optimized syslog-ng

InterWorx doesn’t care one bit about syslogd vs klogd vs syslog-ng. Whatever is installed with the distro is what gets used. I suspect changing it will result in 0 noticeable difference, but if it makes you happy, by all means go for it.

  1. If all domains will utilize Guardian MX then SpamAssassin and ClamAV are a waste of resources. (Note: Guardian MX is a service they provide for some of our domains. Is there an easy way to completely disable SpamAssassin and ClamAV?)

Yes, they can be completely disabled in NodeWorx under Mail Settings->Virus/Spam Settings respectively.

  1. Vixie-cron is more versatile than atd.

Ok. As long as the cron file syntax is the same, InterWorx won’t care what system you use.

  1. Utilizing tpop3d would incur less overhead than vpopmail.

InterWorx uses qmail and vpopmail for it’s mail infrastructure. This isn’t going to change. “Less overhead” is very vague, I can only guess at what they are referring to. It sounds like they just prefer and use tpop3d - and as far as I know, tpop3d is not a “drop in replacement” for vpopmail, so this doesn’t really compute.

  1. The latest version of BIND 9 would use less overhead and would be more secure then tinydns.
  • say the guys that use BIND 9 and not tinydns :). Nothing can come from this statement except a flame war. I’ll just say I kindly disagree with that statement. Folks see all those tinydns processes (one for each IP address a DNS server is listening on) and assume that is somehow “more overhead.” That’s just not the case. Sometimes folks are also turned off by the number of “supervise” processes running on InterWorx servers. They think they are using a lot of resources. This is entirely incorrect though, it’s just “different” from what they’re used to seeing. InterWorx uses and expects tinydns, so changing this isn’t an option.

I asked him about the tinydns issue and he said it will spawn a number of processes where BIND only creates one.

This statement is just based on a false assumption - that each tinydns process is somehow equivalent to one BIND process. That’s incorrect.

  1. PHP sometimes uses massive amounts of CPU time (~20-30%) & SpamAssassin eats about 30% CPU each scan. (If either of these were hammered, it could cause a DoS condition)

If anything “gets hammered” it can, by definition, cause a problem. If it didn’t cause a problem, you wouldn’t say it was getting hammered :).

  1. iworx consumes vast amounts of memory on the system: It runs large numbers of Apache (iworx-web) and mySQL (iworx-db) processes in addition to those that are already on the system. This results in unnecessary load. (This could also result in DoS if hammered)

We’re always working to decreases InterWorx’s footprint on the system. As it turns out, there’s a lot of stuff that goes on when managing a server, so it can be an uphill battle, but it is of course important and we’re aware of that. The number of child processes a process starts alone is not a direct connection to unnecessary load, as mentioned above.

  1. iworx uses its own separate application of apache and mysql. Optimization could occur if these could be merged to use a single version of the applications.

The decision was made early on to remain as flexible as possible folks using InterWorx. Most folks consider it one of the strongest things about InterWorx - they can change the server’s webserver and database server and never worry about InterWorx having a problem. We believe this benefit outweighs the other scenario.

Hope that helps,

Paul

11

Thanks everyone for the thorough responses! This helps a great deal.