Ubuntu version upgrade bashed Zenoss

4 posts / 0 new
Last post
Gregg hughes ISC
Gregg hughes ISC's picture
Offline
Last seen: 22 hours 47 min ago
Joined: 12/08/2015 - 15:48
Posts: 3
Ubuntu version upgrade bashed Zenoss

Good afternoon, all!

I did a version upgrade for my Ubuntu server from 12.04 to 14.04 LTS. This server has Zenoss 4.2.4 riding on it, and it's tracking network activity on my routers and switches.

After the upgrade, the dashboard came up and I thought all was well.  Of course not! After some checking later I found that there were no traffic recorded on the graphs for the time after I had done the upgrade. Also, the high utilization warnings stopped at the same time as the graphs stopped. The event console also shows that zentrap, zenperfsnmp and zenmodeler all show heartbeat failure.

When I look at the logs in /opt/zenoss/log, I find a couple of interesting things. First, the zeneventserver.log shows some general failures in Java.  Here are some examples:

 2017-02-08T12:53:31.267 [INDEXER_EVENT_SUMMARY] WARN  org.zenoss.zep.index.impl.EventIndexerImpl - General failure indexing events
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

and

2017-02-08T12:53:28.266 [ZEP_HEARTBEAT_PROCESSOR] WARN  org.zenoss.zep.impl.Application - Failed to process heartbeat events
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Running zenoss status shows the three daemons above (zentrap, zenperfsnmp and zenmodeler) not running. However, the zentrap.log shows the daemon listening on SNMP trap port 162. Zenmodeler is running as a daemon, connected to localhost:8789 and connected to zenhub. However, about the time of the reboot after the upgrade zenperfsnmp.log threw a "DeadReferenceError: Calling Stale Broker" error. However, it to is connected to localhost:8789, Zenhub and is performing periodic maintenance.

I suspect the problem is authentication between Zenoss and MySql, as a result of the upgrade. However, I'm not sure where best to begin tracing the fix. I'm going over the installation manual again after this posting, but want to be sure I'm looking in the right place.

Any ideas on why the three daemons aren't starting or logging shutdown events, and where the authentication can be checked between Zenoss and MySql?

Thanks to all for looking!

 

Gregg

 

Jcurry
Jcurry's picture
Offline
Last seen: 1 week 3 days ago
Joined: 01/02/2014 - 13:04
Posts: 380
If you are lucky, it is just

If you are lucky, it is just the index to the events database that is confused.  You can safely delete these and they will be recreated when Zenoss is restarted.  As the zenoss user, go to $ZENHOME/var/zeneventserver/index.  You should find subdirectories for archive and summary.  Delete both those subdirectories and restart Zenoss.

Cheers,

Jane

NB Please note that we are away from Feb 10 to March 10, 2017

Gregg hughes ISC
Gregg hughes ISC's picture
Offline
Last seen: 22 hours 47 min ago
Joined: 12/08/2015 - 15:48
Posts: 3
Update on the events index

Hello, Jane!

Thanks for looking into this. Unfortunately, deleting these directories threw an unrecoverable error that brought up a "send the logs home to Zenoss" page. After restoring these directories I'm back to where I was before.

I see you'll be out on holiday for a month, but if you're checking, I'm still stuck.

Thanks!

 

Gregg

Gregg hughes ISC
Gregg hughes ISC's picture
Offline
Last seen: 22 hours 47 min ago
Joined: 12/08/2015 - 15:48
Posts: 3
Update - there's some segfaulting going on!

So I'm going over some OSSEC logs this morning and I find this:

Feb 13 15:50:29 isc-nagios kernel: [442203.736394] python[30077]: segfault at 8 ip 00007f7ec8eb7d63 sp 00007fff2f9b8878 error 4 in libc-2.19.so[7f7ec8e31000+1ba000]

This looks very suspicious. I'm going to debug this as soon as I have time and see if this is what's biting on my Zenoss installation. Meanwhile, has anyone else seen this particular error?

Thanks to all!

 

 

Log in to post comments