We found Tomcat crashing (once a week) occasionally and here is the excerpt of the log

==================================================================================

2015-08-27 22:36:50 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000001800057d0, pid=2252, tid=6056
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [tcnative-1.dll+0x57d0]
#
# Core dump written. Default location: D:\Tomcat 7.0\hs_err_pid2252.mdmp
#
# An error report file with more information is saved as:
# D:\Tomcat 7.0\hs_err_pid2252.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
ntialService.getCredential(CredentialService.java:611)
at com.documentum.web.env.CredentialService.getUsername(CredentialService.java:168)
at com.documentum.web.formext.session.SavedCredentialsAuthenticationScheme.authenticate(SavedCredentialsAuthenticationScheme.java:91)
at com.documentum.web.formext.session.AuthenticationService.authenticate(Unknown Source)
at com.documentum.wp.session.WpAuthenticationService.authenticate(WpAuthenticationService.java:28)
at com.documentum.web.formext.component.ComponentDispatcher.createComponentDef(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.mapRequestToComponent(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at com.documentum.web.formext.component.ComponentDispatcher.doService(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.serviceAsNonController(Unknown Source)
at com.documentum.web.formext.component.ComponentDispatcher.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:348)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.documentum.web.env.WDKController.processRequest(Unknown Source)
at com.documentum.web.env.WDKController.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

==================================================================================================

As you can see it is pointing us to tcnative-1.dll. This is the DLL we added to tomcat a while ago for improving performance.

http://tomcat.apache.org/native-doc/

Then we looked at the dump files — hs_err_pid2252.mdmp and here is what we found

========================================================

Stack: [0x000000001bdc0000,0x000000001bec0000],  sp=0x000000001bebe330,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [tcnative-1.dll+0x57d0]

[error occurred during error reporting (printing native stack), id 0xc0000005]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.Socket.sendbb(JII)I+0
j  org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V+22
J 5791 C2 org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (514 bytes) @ 0x0000000002213e24 [0x0000000002213bc0+0x264]
J 2021 C2 org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (37 bytes) @ 0x000000000185a350 [0x000000000185a300+0x50]
j  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+101
j  org.apache.catalina.connector.OutputBuffer.flush()V+2
j  org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
j  com.documentum.ucf.server.transport.impl.ServerSession.sendAttachments(Ljava/io/OutputStream;)V+2258
j  com.documentum.ucf.server.transport.impl.ServerSession.executeInternal(Z)Lcom/documentum/ucf/server/transport/IRequestResponseIterator;+1456
j  com.documentum.ucf.server.transport.impl.ServerSession.execute(Z)Lcom/documentum/ucf/server/transport/IRequestResponseIterator;+59
j  com.documentum.ucf.server.transport.impl.UCFCommManager.finishSession(Lcom/documentum/ucf/server/transport/IServerSession;)V+484
j  com.documentum.web.contentxfer.ucf.UcfSessionManager.disconnectSession(Lcom/documentum/ucf/server/transport/IServerSession;)V+17
j  com.documentum.web.contentxfer.ucf.UcfSessionManager.release(Lcom/documentum/ucf/server/transport/IServerSession;)V+90
j  com.documentum.web.contentxfer.ucf.UcfContentTransport.releaseUcfSession()V+25
J 6258 C2 com.documentum.web.contentxfer.ContentTransferService.execute()V (440 bytes) @ 0x00000000023eba48 [0x00000000023eb120+0x928]
j  com.documentum.web.contentxfer.JobAdapter.execute(Lcom/documentum/fc/client/IDfSessionManager;Ljava/lang/String;)Z+20
j  com.documentum.job.async.AsyncJobManager$AsyncTimerJob.executeJob()V+40
j  com.documentum.job.async.AsyncJobManager$AsyncTimerJob.run()V+37
J 3494 C2 java.util.TimerThread.mainLoop()V (232 bytes) @ 0x0000000001d8b410 [0x0000000001d8b3c0+0x50]
j  java.util.TimerThread.run()V+1
v  ~StubRoutines::call_stub

================================================================================

I spent couple of hours looking through the Tomcat bugs/ Known issues that related to Tomcat Native Library component but had no luck.

Then, luckily I found following EMC note.

https://community.emc.com/docs/DOC-42889

Which seems to be the exact error I have in core dump. The Resolution was to

Add this properties org.apache.catalina.connector.RECYCLE_FACADES=true to tomcat/conf/catalina.properties file and restart the application server according to http://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues.