We found Tomcat crashing (once a week) occasionally and here is the excerpt of the log
==================================================================================
2015-08-27 22:36:50 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000001800057d0, pid=2252, tid=6056
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [tcnative-1.dll+0x57d0]
#
# Core dump written. Default location: D:\Tomcat 7.0\hs_err_pid2252.mdmp
#
# An error report file with more information is saved as:
# D:\Tomcat 7.0\hs_err_pid2252.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
ntialService.getCredential(CredentialService.java:611)
at com.documentum.web.env.CredentialService.getUsername(CredentialService.java:168)
at com.documentum.web.formext.session.SavedCredentialsAuthenticationScheme.authenticate(SavedCredentialsAuthenticationScheme.java:91)
at com.documentum.web.formext.session.AuthenticationService.authenticate(Unknown Source)
at com.documentum.wp.session.WpAuthenticationService.authenticate(WpAuthenticationService.java:28)
at com.documentum.web.formext.component.ComponentDispatcher.createComponentDef(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.mapRequestToComponent(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at com.documentum.web.formext.component.ComponentDispatcher.doService(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.serviceAsNonController(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:348)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.env.WDKController.processRequest(Unknown Source)
at com.documentum.web.env.WDKController.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
==================================================================================================
As you can see it is pointing us to tcnative-1.dll. This is the DLL we added to tomcat a while ago for improving performance.
http://tomcat.apache.org/native-doc/
Then we looked at the dump files — hs_err_pid2252.mdmp and here is what we found
========================================================
Stack: [0x000000001bdc0000,0x000000001bec0000], sp=0x000000001bebe330, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [tcnative-1.dll+0x57d0]
[error occurred during error reporting (printing native stack), id 0xc0000005]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.apache.tomcat.jni.Socket.sendbb(JII)I+0
j org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V+22
J 5791 C2 org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (514 bytes) @ 0x0000000002213e24 [0x0000000002213bc0+0x264]
J 2021 C2 org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (37 bytes) @ 0x000000000185a350 [0x000000000185a300+0x50]
j org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+101
j org.apache.catalina.connector.OutputBuffer.flush()V+2
j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
j com.documentum.ucf.server.transport.impl.ServerSession.sendAttachments(Ljava/io/OutputStream;)V+2258
j com.documentum.ucf.server.transport.impl.ServerSession.executeInternal(Z)Lcom/documentum/ucf/server/transport/IRequestResponseIterator;+1456
j com.documentum.ucf.server.transport.impl.ServerSession.execute(Z)Lcom/documentum/ucf/server/transport/IRequestResponseIterator;+59
j com.documentum.ucf.server.transport.impl.UCFCommManager.finishSession(Lcom/documentum/ucf/server/transport/IServerSession;)V+484
j com.documentum.web.contentxfer.ucf.UcfSessionManager.disconnectSession(Lcom/documentum/ucf/server/transport/IServerSession;)V+17
j com.documentum.web.contentxfer.ucf.UcfSessionManager.release(Lcom/documentum/ucf/server/transport/IServerSession;)V+90
j com.documentum.web.contentxfer.ucf.UcfContentTransport.releaseUcfSession()V+25
J 6258 C2 com.documentum.web.contentxfer.ContentTransferService.execute()V (440 bytes) @ 0x00000000023eba48 [0x00000000023eb120+0x928]
j com.documentum.web.contentxfer.JobAdapter.execute(Lcom/documentum/fc/client/IDfSessionManager;Ljava/lang/String;)Z+20
j com.documentum.job.async.AsyncJobManager$AsyncTimerJob.executeJob()V+40
j com.documentum.job.async.AsyncJobManager$AsyncTimerJob.run()V+37
J 3494 C2 java.util.TimerThread.mainLoop()V (232 bytes) @ 0x0000000001d8b410 [0x0000000001d8b3c0+0x50]
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub
================================================================================
I spent couple of hours looking through the Tomcat bugs/ Known issues that related to Tomcat Native Library component but had no luck.
Then, luckily I found following EMC note.
https://community.emc.com/docs/DOC-42889
Which seems to be the exact error I have in core dump. The Resolution was to
Add this properties org.apache.catalina.connector.RECYCLE_FACADES=true to tomcat/conf/catalina.properties file and restart the application server according to http://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues.
Your style is so unique compared to other people I have read stuff from.
Thank you for posting when you have the opportunity, Guess I will just bookmark
this blog.
Thanks , I have just been searching for info approximately this topic for
ages and yours is the best I’ve discovered till now.
However, what in regards to the bottom line? Are
you sure in regards to the supply?