Quantcast
Channel: Adobe Community : Popular Discussions - ColdFusion
Viewing all articles
Browse latest Browse all 14291

Apache POST flex2gateway never closes or times out, reaches max child processes

$
0
0

We have been trying to pass an external PCI scan, and noticed some server lockups after starting a scan.  We are scanning a couple hundred IP addresses, which all resolve to the same servers.  The scans are actively looking for vulnerabilities on the box, and one of which is flash remoting.  When we look at the apache /server-status page, it shows a ton of long running flex2gateway processes.  For instance:

 

22-44466

0/3817/3817

W4.0716384000.057.7657.76x.x.x.101WebNode2.ambassador.intPOST /flex2gateway/http HTTP/1.1

 

As you can see, this POST request has been running for 163840 seconds, or nearly two days.  Since it seems these POST requests never complete, even though the client has long since disconnected, they simply stack up until the server's max number of child processes has been reached, effectively killing our webserver.

 

When I try to restart the clustered coldfusion instances one at a time, these POST requests do not die off.

If I stop both clustered CF instances, the requests complete (or get killed).

If I reload or restart apache, the requests are gone as well.

 

strace gives me nothing useful:

[root@WebNode1 ~]# strace -p 34025

Process 34025 attached - interrupt to quit

read(185,

 

pstack gives a little more, but nothing that looks obvious to me:

[root@WebNode1 ~]# pstack -p 34025     

Usage: pstack <process-id>

[root@WebNode1 ~]# pstack 34025  

#0  0x00007fdd40444740 in __read_nocancel () from /lib64/libpthread.so.0

#1  0x00007fdd33efe2e6 in jk_tcp_socket_recvfull () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so

#2  0x00007fdd33f1b68d in ajp_connection_tcp_get_message () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so

#3  0x00007fdd33f1ceea in ajp_get_reply () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so

#4  0x00007fdd33f20308 in ajp_service () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so

#5  0x00007fdd33ef8f5d in jk_handler () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so

#6  0x00007fdd41b92cd0 in ap_run_handler ()

#7  0x00007fdd41b9658e in ap_invoke_handler ()

#8  0x00007fdd41ba1c50 in ap_process_request ()

#9  0x00007fdd41b9eac8 in ?? ()

#10 0x00007fdd41b9a7d8 in ap_run_process_connection ()

#11 0x00007fdd41ba6ad7 in ?? ()

#12 0x00007fdd41ba6dea in ?? ()

#13 0x00007fdd41ba7a6c in ap_mpm_run ()

#14 0x00007fdd41b7e9b0 in main ()

 

I dont know what that tells us exactly, but I'm leaning toward the hangup between apache and tomcat. 

 

Any suggestions on where how to troubleshoot this issue?


Viewing all articles
Browse latest Browse all 14291

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>