Clustering And You

[Keiro]

Hi,

I’m posting here instead of asking InterWorx directly as I’m curious and interested in hearing about others’ implementation of InterWorx’s clustering features. I’d love to implement it and we do have tentative plans of implementing it.

The question(s) I have are: As far as I understand it, the clustering features allow one to split server functionality out to several servers. Since we already know what the clustering features do, I won’t repeat them here. That said…

1. What is best practices for implementing such a feature? For example… is it wise to have such clustering features set up locally as it were, in a given datacenter rack, as it were?

2. Is it possible to use those clustering features across several datacenters?

3. Building upon #2, I currently have the Cluster Manager (currently unlicensed, functioning as a standalone InterWorx instance but will be relicensed for clustering functionality) as a VPS. I intend to build out physical servers that contain lots of storage, either for SQL use or for customer data use.

4. If working across different datacenters is possible, it’s obviously going to be bandwidth-intensive for private communications between the CM and servers doing the CM’s bidding. How difficult would this be?

As far as I understand it, through the InterWorx documentation, it doesn’t matter how one implements it, it’ll all be transparent to the end-user anyway. Yes?

[d2d4j]

Hi kerio

I hope you don’t mind but here’s my views

I’m not too sure what you mean by setup locally in a datacentre - I’m thinking this is part of your Q2 perhaps. You can set it locally behind your internal network (saves on bandwidth usage) or set it up on external connections (bandwidth will be counted on your agreement with provisioner)

I do not believe you can use clustering across several datacentres

You can currently achieve seperate build outs without clustering, ie have seperate MySQL, storage etc but would be better to cluster this.

Please remember, clustering groups services, so you can have more servers serving for apache, email etc… And the load balance is a breeze to use. Also, please remember you do not need high end servers, with a lot of space for clustering, but I would advise your cluster manager does have large fast storage, or your storage server is fast.

I’m sorry I do not understand your last question, my understanding is the end user would not be aware if you were clustered or not, and if clustered, would not know which slave was serving their request. I do think though, any user with good understanding would be able to work out if clustered or not, but then this has never bothered me sorry.

I’m sorry if I am wrong though, and I would strongly advise you follow IW for future announcement, which I will not say anything but I’m sure you would be interested in.

I hope that helps

Many thanks

John

[Licensecart]

InterWorx clustering system is just for load balancing not for splitting up so when a server gets full it moves onto another server. There is HA, which copies everything so if a server goes offline the sites stay online.

[Keiro]

[QUOTE=d2d4j;27035]Hi kerio

I hope you don’t mind but here’s my views

I’m not too sure what you mean by setup locally in a datacentre - I’m thinking this is part of your Q2 perhaps. You can set it locally behind your internal network (saves on bandwidth usage) or set it up on external connections (bandwidth will be counted on your agreement with provisioner)

I do not believe you can use clustering across several datacentres

You can currently achieve seperate build outs without clustering, ie have seperate MySQL, storage etc but would be better to cluster this.

Please remember, clustering groups services, so you can have more servers serving for apache, email etc… And the load balance is a breeze to use. Also, please remember you do not need high end servers, with a lot of space for clustering, but I would advise your cluster manager does have large fast storage, or your storage server is fast.

I’m sorry I do not understand your last question, my understanding is the end user would not be aware if you were clustered or not, and if clustered, would not know which slave was serving their request. I do think though, any user with good understanding would be able to work out if clustered or not, but then this has never bothered me sorry.

I’m sorry if I am wrong though, and I would strongly advise you follow IW for future announcement, which I will not say anything but I’m sure you would be interested in.

I hope that helps

Many thanks

John[/QUOTE]

By setting it up locally in a datacenter, I mean… let’s say I’m using all three of QuadraNet’s datacenters, right? Now… with my read/understanding of your post, this means I need 3x CMs, one for each datacenter location, ie, LAX, Houston and Tampa if I remember their locations right.

Or… wait, I remember QuadraNet has its own private networks across all three datacenters… so I’d only need one CM, at least for QN. I’ll need additional CMs for other datacenters since we cannot cluster across different datacenters. Hmm. Maybe that’s the best way to go, have a CM for each additional datacenter outside of QN… because I really like QN. And the ability to have your own private network across all three datacenters and see it as one private network.

I’m aware that clustering groups services… and that’s why I want to implement clustering. Because I want to be able to split services out onto several servers. I’ve never really liked cPanel’s reliance on the fact that everything has to be on one box, itself. I mean, yes, cPanel can be clustered but that tends to be headache-inducing in more than one way. But cPanel isn’t really designed for clustering whereas InterWorx is.

Hopefully this clarifies everything for you.

Licensecart ? I know. If I recall correctly though, HA isn’t really possible with InterWorx yet, though.

[d2d4j]

Hi kerio

Many thanks, makes more sense thanks

The only thought which comes to mind re clustering over different datacentre, albeit your capable of seeing just 1 network between, which is good, would be 2 fold, latency and only 1 datacentre would hold your /home or /chroot/home.

I think your thoughts over each datacentre having it’s own CM/cluster would be better, and I suppose you could rsyn your home to each, but perhaps messy.

Glad you understood how cluster works with IW, and HA was what I was referencing to keep an eye on for announcement, which works lovely, and you can lose 1 CM and whole cluster keeps running. However, if running HA CMs, I personally would suggest you have a seperate storage server as I do not think the CM copy the files, the CM just control the CN

I know HA CM is in testing, and hopefully should not be too far away, but did not want to spoil any IW announcement

I’m sorry if I’m wrong though

Many thanks

John

[Keiro]

Latency is fairly low between all three QN datacenters.

As far as HA goes… there’s probably two ways we could go with this. I was thinking of using a HA cloud solution. Something like QuadraNet’s InfraCloud offerings for the CM, for example. This would more or less provide the ability to have HA CM on the hypervisor.

Or having a hot/cold copy of the CM where if the main CM goes down, the spare CM comes up, using the main CM’s resources until the main CM comes back up. Not quite HA… but close enough.

Hopefully someone’ll have a nice hosting solution for HA CM… maybe I should talk to QuadraNet about that, as they seem to be very familiar with InterWorx…

If anyone’s implemented such a clustering system, how’d you handle your storage issues? As far as I understood it, you could split storage service out to specific storage servers via the NFS share but that the preferred method is to have the CM handle storage itself directly rather than the storage servers?

[d2d4j]

Hi kerio

Are you the user on vps board who had wows with another provider then moved to quadrant

You have very nicely described IW HA CM, there is 2 CM, when one goes down, the second takes over and when the downed CM is returned to service, is brought back into CMs.

I personally would use seperate storage myself, but I know you could use either method for storage, but if keeping storage on primary HA CM, I think this is a weakness as if it goes down, you may lose your storage area

As I said, so you fully understand, IW are in testing with HA CM, which we tested last October.

I hope that helps

Many thanks

John

[Keiro]

[QUOTE=d2d4j;27040]Hi kerio

Are you the user on vps board who had wows with another provider then moved to quadrant

You have very nicely described IW HA CM, there is 2 CM, when one goes down, the second takes over and when the downed CM is returned to service, is brought back into CMs.

I personally would use seperate storage myself, but I know you could use either method for storage, but if keeping storage on primary HA CM, I think this is a weakness as if it goes down, you may lose your storage area

As I said, so you fully understand, IW are in testing with HA CM, which we tested last October.

I hope that helps

Many thanks

John[/QUOTE]

Er… wows? We’ve used a few different providers for VPS services in the past. Edit: Same username on WHT. /edit

I’m thinking that we won’t be using the primary HA CM with its own storage. That is a high-failure point and… I’m not too keen on that particular failure point. Much easier to split it out into storage servers that the primary and backup HA CMs can access when one or the other fails and continue operating with nearly no loss of service.

I do indeed fully understand that IW is testing HA CM. Hopefully it’s going well and will allow us to add that as another highly desired feature for our shared and reseller hosting offerings.

[IWorx-Brett]

Hi Keiro – I’m glad to hear you’re interested in building out an InterWorx cluster! I wanted to clarify a couple points brought up in this thread.

• Our HA setup is still in beta, and what we’re looking for with it are big hosts that control their own datacenter to get on board. We’ve reached out to our partners about it, but nothing motivates hosts mote than customer demand, so definitely suggest to Quadranet that they try it

• As others have pointed out, latency has always been an issue with multi-datacenter cluster setups. Perhaps Quadranet’s setup mitigates that, but I can’t say.

• When discussing it with them, make sure they allow IP spoofing – it’s a requirement for clustering

• Out of the box, InterWorx clustering doesn’t support multi-CM setups. Even HA is basically a backup load-balancing server, not a fully-operational CM.

• We generally recommend both an external file system and an external MySQL setup for large clusters – both are actually a requirement for HA.

• Nexcess is our biggest cluster customer, and they put out a whitepaper a few years ago on how they setup their systems. They are optimized for running Magento, so it might not be 100% like what you’re doing, but there’s still a lot of good info here: http://www.nexcess.net/resources/white-papers/magento-best-practices

Happy to answer any other questions you have!

[Keiro]

Your forum is way too aggressive in checking posts for URLs and forbidden words.

Can we please get this fixed? I keep hitting this when quoting or writing words that shouldn’t trigger a check.

[d2d4j]

Hi kerio

Sorry, I use tapatalk on mobile and predictive text kicks in, it should have said woes

I would open support ticket and ask IW over posting, and you should be able to give them examples so they can check.

If you prefer, you can send an email to support@interworx.com

I have to say though, I have never had an posting issue but do understand users who have, but again it depends on the post.

I think what’s nice about IW forums, is it not full of junk spam posts.

Many thanks

John

[Keiro]

Will do, thanks, John!

Edit: Ah, gotcha on the woes part… yes, that’d be me. We did have lots of VPS issues in the past.

[Keiro]

[QUOTE=IWorx-Brett;27045]Hi Keiro – I’m glad to hear you’re interested in building out an InterWorx cluster! I wanted to clarify a couple points brought up in this thread.

• Our HA setup is still in beta, and what we’re looking for with it are big hosts that control their own datacenter to get on board. We’ve reached out to our partners about it, but nothing motivates hosts mote than customer demand, so definitely suggest to Quadranet that they try it

• As others have pointed out, latency has always been an issue with multi-datacenter cluster setups. Perhaps Quadranet’s setup mitigates that, but I can’t say.

• When discussing it with them, make sure they allow IP spoofing – it’s a requirement for clustering

• Out of the box, InterWorx clustering doesn’t support multi-CM setups. Even HA is basically a backup load-balancing server, not a fully-operational CM.

• We generally recommend both an external file system and an external MySQL setup for large clusters – both are actually a requirement for HA.

• Nexcess is our biggest cluster customer, and they put out a whitepaper a few years ago on how they setup their systems. They are optimized for running Magento, so it might not be 100% like what you’re doing, but there’s still a lot of good info here: http://www.nexcess.net/resources/white-papers/magento-best-practices

Happy to answer any other questions you have![/QUOTE]

We’re not quite a big host… but we do know that for quality service, you need to offer the best that you can offer. Hence, we’re looking into this for our own offerings. That, and to be able to advise others in addition to our own clients for how to implement one and slowly scale it out.

I imagine we’ll have lots of teething pains as we scale our clustering solution out… but I think this is worth the teething pains. I’ve had to spend hours working on cPanel boxens that hosed themselves and while I’m aware clustering an IW setup doesn’t stop that from happening… it lowers that chance when we can just spin up fresh instances to take over those that basically craps the bed, you know? It’s the simple fact that we can spin up new instances to take over duties and give us a chance to quietly work on the boxens that’re having trouble quietly and put them back into production quickly that has us looking into clustering.

It’s issues like this that quickly loses you your customers… believe me, I know this from firsthand experience, myself.

As far as IP spoofing goes… I’m not sure if they do? Why would you need IP spoofing? Could you go into more detail about that for the clustering? It’d be easier to get QuadraNet on board if I can explain exactly what I need from them to set up something like this as a test platform… or for them to set up their own.

So, based on your comment re: HA being nothing more than a load-balancing server… wait, that’s basically all a CM is, never mind.

External file system + external mysql is fine. We’ll probably need to go with internal storage for a bit until I figure out what external storage solution we’d want to go with… hopefully the switch won’t be too painful when we go from internal to external. I was fully expecting this, to be honest. What’re your suggestions in this particular aspect? Setting up external mysql is fairly easy… relatively speaking. I’ll be sticking MySQL on SSD-backed storage servers for faster MySQL access, though.

I’m thinking the fastest short-cut as it were is to use a HA VM setup, again as mentioned — such as QuadraNet’s InfraCloud solution, or GigeNET Cloud’s solution. In fact, GigeNET Cloud themselves offers an IW cluster of their own.

Bit much money for my wallet though. But worth it in the end if you need a turn-key solution for IW clustering. Me? I’d like to set it up according to our own needs and I think doing it through QN’s three datacenters would work perfectly for us. It’ll be a bit… messy, I’m sure but it should work in theory based on my experience with having all three datacenters hosting servers in the past.

[d2d4j]

Hi kerio

My thoughts are ip spoofing is required so you can have 2 CM believing they have the same ip, external ip I believe. Which makes sense for no loss of service should the primary CM fail. I believe both CM have live checks to determine if up or down.

I personally would not use HA VM, but would use IW CM, seperate storage and MySQL, then upgrade to HA CM when it’s available. Ofcourse if monies no object, I might

Also, I would advice you set all IW instances to use /chroot/home, which could save a little work when clustering.

I hope that helps a little

Many thanks

John

[IWorx-Brett]

Close, but not exactly IP spoofing is actually a requirement of all clustering, not just HA. Basically, the nodes have different IP’s than the CM, so if they process a request, they need to be able to spoof the CM’s IP so that the requestor recognizes the response.

[Keiro]

@IWorx-Brett;27052

Close, but not exactly IP spoofing is actually a requirement of all clustering, not just HA. Basically, the nodes have different IP’s than the CM, so if they process a request, they need to be able to spoof the CM’s IP so that the requestor recognizes the response.

Ahhh, gotcha. That clears it up… I’m not sure if IP spoofing is possible… I’ll ask QuadraNet.

[Keiro]

Been thinking about the clustering features last night.

You know, I’m thinking it’s possible to do cross-datacenter clustering. But it will be tricky to do, as we’d have to set up our own routes to the other datacenters… but not something I’d be willing to try just yet. We don’t have the necessary hardware for that… but if someone wants to run with it, why not?

Basically, what you’d need is probably a VPN of some sort. Maybe not necessarily a VPN but definitely a dedicated static route that ties into your own network at your main datacenter. Think of the failover features that nearly all datacenters use. Those tend to be multi-homed with multiple redundancies in place.

Hmm… I’d love to try this but, again… we don’t have the necessary hardware to try such a wacky, possibly workable solution for cross-datacenter clustering.

[Keiro]

Spoke with QuadraNet via ticket… they said no, they do not allow IP spoofing. Whoever spoke with me said they spoke with their manager.

I’ve requested to talk to a network engineer, see if this is possible and if not, what can be done to more or less come close enough to full-on cluster.

[d2d4j]

Hi kerio

I’m not sure over your setup so I could be entirely wrong sorry, but you should be able to run ip spoofing on an internal private network, which should not involve QN.

Also, yes your thoughts on datacentre clustering should work, but this has been discussed before, and there are limiting factors, which as time goes by, are reducing.

There are other methods to achieve datacentre outages, which are not full cluster but give HA of websites, such as having seperate clusters in each datacentre and use DNS to trigger a change if detected your primary cluster was down, and it changes the A record to point to your reserve cluster held in another datacentre.

Many thanks

John

[Keiro]

We’ll be setting up our own miniature cloud next week on QN infra.

makes note to talk to InterWorx for demo cluster licenses. I’m pretty sure what I’ve got in mind for implementing this will work just fine.

We talked with QN Network Engineering and were able to determine that what we discussed about in this thread should work although the person I spoke with stated they forgot that the InfraCloud infrastructure they have do not have private network access… but it should work nearly the same either way.