Interworx stuck in maintenance mode

Hello!

I’m really new here, so i’m sorry if i post to the wrong place, but we really need some help.

We are using interworx for a half year now without any problems. Today, from 3 AM, one of our machines started an update via yum, and it got stuck. Half hour ago, i tried to reboot the server, but the problem stayed, we cannot reach nodeworx nor siteworx. The websites are running, and we can log in via SSH, but we dont know what to do.

And i cant ask for support via nodeworx, thats why i wrote here.

We apprecciate any help or advice.

Thank you!

Hi emmosystems

Welcome to IW forums

Please can you ssh into server and run the following commands with root rights

service iworx stop

service iworx start

If this fails, please could you post the result

Also, you should be able to email support and detail the server issue but may take longer

Many thanks and I hope that helps

John

You can try to repair the permissions.

cd ~iworx/bin
./cvspermsfix.pex
service iworx restart

Hi dss

Sorry, yes it could be centos 7, good point.

Emmosystems PM me, and actually rereading his post, it shows updating stalled and server restarted, so the current condition of nodeworx is showing maintenance webpage.

I would ssh into server with root permissions (su or root) and run yum update iworx or yum update and allow server to fully complete on update.

If it fails on some updates, I would post the failures but I’m thinking it is just the restart of server breaking the update process

Then restart server (belts and braces to clear down server) and test

Emmosystems, I’ll pm you now but I do not work for interworx sorry.

Also, IW support are in USA time zone, so hopefully at this time of day here in uk, should not be long hopefully

I hope that helps

Many thanks

John

Hi John, all,

I’ve got the same problem on two servers. Restarting iworx service results in “Binding IP Aliases:[FAILED]”, running yum update iworx won’t help, s yum ist stuck: “Another app is currently holding the yum lock; waiting for it to exit… The other application is: yum”

Any ideas?

Cheers,

Michael

We’re having the same issue this morning. Stuck in maintenance mode. Killed stuck yum processes, rebooted, recompleted the interworx yum update, but still stuck in maintenance mode. Not really sure what to do at this point.

I have the exact same situation as the others mentioned.

The only difference is, that we could complete the failed yum updates, everything is updated. cvspermsfix.pex failing cuz it cannod find the horde.log in the default log directory (possible because horde package just updated, and we never use it).

Now i think its a common error :frowning:

Two other servers updated just fine yesterday (version 5.1.3-1041), but since I can’t acces Nodeworx, I can’t confirm it’s the same update. The changelog unfortunately is out-of-date.

Hi

Sounds ominous

I’ve forwarded to support and just trying on one of my test servers.

Emmosystems, if I understand correctly, your issue is resolved by server restart and yum update iworx.

Is this correct

If so, i guess this maybe the short term fix

If I’m wrong, and your issue persists, please accept my apologies and I’ll see what our test server does now

Many thanks

John

Hi

Jenna has replied and IW are currently looking into it

Hopefully should not be long

Many thanks

John

Problem persists. I just said that we could finish the failed update which causing the whole problem at our servers.

In our situation an update of iworx core started this morning 3 AM. We saw it at 9 AM. So we mentioned something failed. We finished yum transactions by killing and rerunning yum update , and then rebooted server twice. Now yum saying no update, but the problem persists, Maintenance mode. The hosted websites running tho, but we couldnt do any config in the nodeworx, and its starting to be a a bigger problem now cuz our customers needs modifications :frowning:

I hope support will start to solve this .

Hi emmosystems

IW support are currently on with this update issue

It is definitely on release candidate update, so if you have other servers, please set the iworx update service to stable so they will not update

I hope that’s alright

Many thanks

John

Hey all,

It looks like there was an issue with the latest release candidate. The development team is still investigating the issue, but it appears that an rpm command was run during the yum update process, which hung since yum locked the rpm database. Please submit a support ticket with remote assistance and the server IP for all affected servers and we will finish the upgrade steps.

EDIT

To enable remote assistance at the command line:

/usr/local/interworx/bin/enable-support-access.sh

Hi Nathan

Many thanks, much appreciated

I will just reload from scratch our test server I tried on as I need to change the drive configurations anyway

Glad you guys resolved issue, you guys rock

Many thanks

John

They fixed our server already :slight_smile:

But for now, in nodeworx it says our license will expire in 10 hours. Which is not true :slight_smile:

Is there anybody with the same experience after the fix?


EDIT: Now everything looks good, we needed to restart iworx service, and run license.pex cuz it came out from sync somehow.

Hi emmosystems

Glad it’s resolved for you

Your correct, you needed to license.pex to stop warning, although in reality, it would have run by cron anyway before 10 hours.

It would have not been able to sync or the license IP registered may have been showing differently to that of your server due to bind IP aliase failed

I hope that helps

Many thanks

John

Hi Michael

Apologies, you were correct with your first post in highlighting where the issue was, and when I updated our test server, saw the same and checked top, which showed db in use.

Hope your well and business is good, and once again, apologies it just seems to be one of those days for me

Many thanks

John

Sorry meant to say update was 5.1.3 1042

Fix outstanding transactions and remove maintenance file

I fixed it like this:

Repair the broken update:


yum install yum-utils
yum-complete-transaction

Output:
There are 1 outstanding transactions to complete. Finishing the most recent one
The remaining transaction had 11 elements left to run.

After that I deleted (again) the file /home/interworx/etc/env/maintenance-rpm.

rm /home/interworx/etc/env/maintenance*

Hi ceewee

Many thanks but does your maintenance page still work ie did the upgrade replace those files back

Also, do you know you thanked your own post

Many thanks

John