Monday, August 25, 2014

Maximo startup problems!

Our infrastructure utilizes WebSphere MQ as our Maximo queue backend. Our automation framework (consisting mainly of jython scripts injected into wsadmin) sets up CQIN and SEQIN queues, as well as activation specifications and whatnot, with 1 click of a button, so our margin for error is pretty low once it's off the ground running.
So when this error appeared in SystemOut.log on Maximo startup it was quite discomforting:

[8/25/14 15:53:39:542 EST] 000003d9 SystemOut O 25 Aug 2014 15:53:39:510 [ERROR] [MXServer] [] java.lang.NullPointerException at psdi.iface.jms.JMSContQueueProcessor.processMessage(JMSContQueueProcessor.java:253) at psdi.iface.jms.JMSListenerBean.onMessage(JMSListenerBean.java:203) at com.ibm.ejs.container.WASMessageEndpointHandler.invokeJMSMethod(WASMessageEndpointHandler.java:138) at com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1146) at com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invoke(MessageEndpointHandler.java:844) at com.sun.proxy.$Proxy33.onMessage(Unknown Source) at com.ibm.mq.connector.inbound.MessageEndpointWrapper.onMessage(MessageEndpointWrapper.java:131) at com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:125) at com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2747) at com.ibm.mq.jms.MQSession.run(MQSession.java:950) at com.ibm.mq.connector.inbound.ASFWorkImpl.doDelivery(ASFWorkImpl.java:88) at com.ibm.mq.connector.inbound.AbstractWorkImpl.run(AbstractWorkImpl.java:216) at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:668) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

This error was flooding the logs every few milliseconds, and causing CPU starvation!
It seemed to suggest the "JMSContQueue" was trying to "processMessages" (duh). The problem: The WebSphere MQ infrastructure (which we didn't own) did not have the queue yet. For some reason Maximo polls infinitely for the queue...so I changed "intjmsact" to point at a queue which did exist, and voila!

But that wasn't the end of it! When the queue was finally created and "intjmsact" was configured back to point at the original queue, same error message!
This time, the problem was that there were messages already on the queue, which Maximo did not recognize. Maximo picked them up, rejected them, and put them back on the queue, causing yet another infinite cycle. Deleting the messages resolved the issue.

No comments:

Post a Comment