Hello,
We are running CMS Search on a stand alone search server separate from our TeamSite/OpenDeploy base server. After applying 3 separate patches, we finally thought search was ready to be enabled however much like many other 7.2.1 horror stories, we've encountered yet another problem - this time with indexing service which will crash on us at random. We do have a case opened but have not made much headway. The most recent crash was last night around 8:36pm EST.
I'm seeing the following errors in the "iwtsindexwrapper.log" around the same time indexing crashed last night:
INFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | # An unexpected error has been detected by Java Runtime Environment:
INFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | # java.lang.OutOfMemoryError: requested 6464505 bytes for jbyte in C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\prims\jni.cpp. Out of swap space?
INFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | # Internal Error (allocation.inline.hpp:42), pid=3100, tid=3964
INFO | jvm 1 | 2013/02/13 20:36:31 | # Error: jbyte in C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\prims\jni.cpp
INFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode windows-x86)
INFO | jvm 1 | 2013/02/13 20:36:31 | # An error report file with more information is saved as:
INFO | jvm 1 | 2013/02/13 20:36:31 | #
\Interwoven\Search\private\bin\hs_err_pid3100.log
INFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | # If you would like to submit a bug report, please visit:
INFO | jvm 1 | 2013/02/13 20:36:31 | #
http://java.sun.com/webapps/bugreport/crash.jspINFO | jvm 1 | 2013/02/13 20:36:31 | #
INFO | jvm 1 | 2013/02/13 20:36:31 | 20:36:31.804 [InactivityMonitor WriteCheck] DEBUG o.a.a.transport.InactivityMonitor - 10000 ms elapsed since last write check.
ERROR | wrapper | 2013/02/13 20:36:32 | JVM exited unexpectedly.
STATUS | wrapper | 2013/02/13 20:36:36 | Launching a JVM...
INFO | jvm 2 | 2013/02/13 20:36:36 | Wrapper (Version 3.1.0)
http://wrapper.tanukisoftware.orgINFO | jvm 2 | 2013/02/13 20:36:36 |
INFO | jvm 2 | 2013/02/13 20:36:41 | 20:36:41.383 [WrapperSimpleAppMain] DEBUG o.a.a.transport.WireFormatNegotiator - Sending: WireFormatInfo { version=5, properties={CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
STATUS | wrapper | 2013/02/13 20:36:53 | <-- Wrapper Stopped
Here is what I'm seeing iw.tsindex.log - the last entries logged just prior to the service terminating:
[13 Feb 2013 20:36:51,399] FATAL com.interwoven.hunter.index.IndexServer (WrapperSimpleAppMain) - Error in initialization of the Index Manager
java.lang.RuntimeException: Error initializing Listener for Interwoven
at com.interwoven.hunter.index.Listener.(Listener.java:94)
at com.interwoven.hunter.index.IndexServer.startListeners(IndexServer.java:342)
at com.interwoven.hunter.index.IndexServer.work(IndexServer.java:283)
at com.interwoven.hunter.index.IndexServer.main(IndexServer.java:210)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:136)
at java.lang.Thread.run(Thread.java:619)
Caused by: javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1368)
at org.apache.activemq.ActiveMQConnection.setClientID(ActiveMQConnection.java:391)
at com.interwoven.hunter.index.Listener.init(Listener.java:120)
at com.interwoven.hunter.index.Listener.(Listener.java:71)
... 9 more
Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1262)
... 13 more
[13 Feb 2013 20:36:51,399] INFO com.interwoven.hunter.index.Reaper (Thread-0) - Shutdown IndexServer
[13 Feb 2013 20:36:51,399] INFO com.interwoven.hunter.index.Reaper (Thread-0) - Shutdown IndexManager
[13 Feb 2013 20:36:51,415] WARN com.interwoven.hunter.index.BranchMapping (Thread-0) - Initial database is empty, i.e. no indexed branches
[13 Feb 2013 20:36:51,446] INFO com.interwoven.hunter.index.IndexerSupport (Thread-0) - DocumentFilter will be re-used as per configuration
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.IndexerSupport (Thread-0) - DocumentFilter will be re-used as per configuration
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.IndexerSupport (Thread-0) - DocumentFilter will be re-used as per configuration
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.Indexer (Thread-0) - BulkIndexer-0 Freeze requested.
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.Indexer (Thread-0) - IncrementalIndexer-0 Freeze requested.
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.Indexer (Thread-0) - Shutting down Indexer BulkIndexer-0 (will stop after current job)
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.Indexer (Thread-0) - Shutting down Indexer IncrementalIndexer-0 (will stop after current job)
[13 Feb 2013 20:36:51,461] INFO com.interwoven.hunter.index.Reaper (Thread-0) - Saving database
[13 Feb 2013 20:36:51,477] INFO com.interwoven.hunter.index.Reaper (Thread-0) - Index server shutdown complete
Any assistance on this would be greatly appreciated!
Thanks in advance,
Rob
(I wanted to also note there was a post about indexing service crashing but the error was different - which specifically refers to "EXCEPTION_ACCESS_VIOLATION"
-
http://devnet.interwoven.com/forums_vb/showthread.php?t=37889
