Transcript ManageTomcatInProductionEnvironmentsx

Manage Tomcat in production
environments
INFN-CNAF
13/11/09
Michele Orrù
Network&Security Lead @ IntegratingWeb
1
•
Virtual Hosting
• Motivation
• Two-tiers with Apache
• Different JVMs
•
Performance
• Motivation
• Testing with Jmeter
• Connectors, Heap, GC and APR
•
Monitoring
• Motivation
• JMX and Hyperic
Outline
Outline
2
•
•
•
•
•
Name-based virtual hosting: multiple FQDN share the same IP address
Cheap solution
Hundreds of domains hosted on the same machine
IP-based virtual hosting can become a nightmare
Consolidate web application monitoring
Virtual Hosting: motivation
Virtual Hosting
3
New directory structure:
Virtual Hosting: Implementation
Virtual Hosting
4
In conf/server.xml:
<Server port="8005" shutdown="SHUTDOWN">
<Service name="Catalina">
<Connector port="8009" enableLookups="false" protocol="AJP/1.3" />
<Engine name="Catalina" defaultHost=“www.integratingweb.com”>
<Host name="www.integratingweb.com" debug="0"
appBase="/home/tomcat/sites/integratingweb.com/webapps"
unpackWARs="true”>
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="/home/tomcat/sites/integratingweb.com/logs"
prefix="integratingweb.com.access."
suffix=".log"
pattern="common"/>
</Host>
<Host name="N"> ... </Host>
Virtual Hosting: Implementation
Tomcat configuration
5
In conf/logging.properties:
# Set the root logger for Tomcat #
log4j.rootLogger=INFO, TomcatINFO, TomcatERROR
# Send all INFO messages and above to a file #
log4j.appender.TomcatINFO=org.apache.log4j.FileAppender
log4j.appender.TomcatINFO.File=/home/tomcat/logs/virtual_host_catalina_.log
# Use the simple layout
log4j.appender.TomcatINFO.layout=org.apache.log4j.SimpleLayout
# Send all ERROR messages and above to the console #
log4j.appender.TomcatERROR=org.apache.log4j.ConsoleAppender
log4j.appender.TomcatERROR.Target=System.out
log4j.appender.TomcatERROR.layout=org.apache.log4j.PatternLayout
log4j.appender.TomcatERROR.layout.ConversionPattern=%p: %m: %d{ABSOLUTE} %n
log4j.appender.TomcatERROR.Threshold=ERROR
# Define a log for the integratingweb.com host #
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[www.integratingweb.com]=INFO,integratingweb
# Log to a file
log4j.appender.integratingweb=org.apache.log4j.FileAppender
log4j.appender.integratingweb.File=/home/tomcat/sites/integratingweb.com/logs/integratingweb.com.log
# Use the simple layout
log4j.appender.integratingweb.layout=org.apache.log4j.SimpleLayout
# repeat last 4 lines for every virtual host
Virtual Hosting: Implementation
Tomcat configuration
6
Virtual Hosting: Two-tiers with Apache
Virtual Hosting
7
Be sure that the following module is loaded in httpd.conf:
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
<VirtualHost *:80>
ServerAdmin [email protected]
ServerName www.barbarabottini.com
ServerAlias www.barbarabottini.com
ProxyRequests Off
ProxyPreserveHost On
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / ajp://localhost:8009/
ErrorLog logs/www.barbarabottini.com-error_log
CustomLog logs/www.barbarabottini.com-access_log common
</VirtualHost>
Virtual Hosting: Two-tiers with Apache
Apache configuration
8
Why?
• Security: if one application crashes, consumes resources, is being hacked,
the problem is confined in his dedicated JVM without affecting others
• Large-scale Tomcat hosting: pay for JVM Heap size
• Per-virtual host JVMs must be supported by good hardware (multi-core,
lot of RAM)
Virtual Hosting: per-virtual host JVM
Per-Virtual Host JVM
9
• Tomcat depends on two environment variables:
• CATALINA_HOME: to find internal classes and libraries
• CATALINA_BASE: configuration-specific files, webapps, logs. If not defined,
default to CATALINA_HOME
• What we do here is to create N directories (one per virtual host) that will
be registered as CATALINA_BASE(s) for our virtual hosts.
• Every CATALINA_BASE will have her own server.xml, with specific
Connectors and Contexts
Virtual Hosting: per-virtual host JVM
Per-Virtual Host JVM
10
Virtual Hosting: motivation
Per-Virtual Host JVM
11
• Startup script for Tomcat mass-virtual hosting
#!/bin/bash
CATALINA_HOME="/home/tomcat"
SITE_ROOT="/home/jvms"
SITES=`ls ${SITE_ROOT}`
for host in ${SITES}
do
export JAVA_OPTS
CATALINA_BASE=${SITE_ROOT}/${host}/catalina
echo “Starting server: ${host} with CATALINA_BASE=${CATALINA_BASE}”
export CATALINA_BASE
${CATALINA_HOME}/bin/startup.sh
done
Virtual Hosting: motivation
Per-Virtual Host JVM
12
Performance
– Response time
– Throughput
– Number of users that the application should support
Performance: motivation
• User’s perspective: how fast the content is served to the client
• We want to measure:
13
Let see a demo…
Performance: Testing with Jmeter
Apache Jmeter
14
APR
•
•
Apache Portable Runtime can be used to improve scalability, stability and
performance
Uses Java Native Interface (JNI) wrappers for APR (libtcnative)
Powerful when using Tomcat as a webserver (OS syscalls such as sendfile(),
epoll(), native process handling, OpenSSL instead of Java SSL) without Apache
$tar xvzf bin/tomcat-native-1.1.16-src.tar.gz
$cd bin/tomcat-native-1.1.16-src/jni && ant
$cd bin/tomcat-native-1.1.16-src/jni/native
$./configure --with-apr=/usr/local/apache2/bin/apr-1-config
$make
#make install
$export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/apr/lib
Performance: APR
•
15
APR
Restarting Tomcat, having defined Connectors such as:
<Connector port="9080” maxThreads="150" minSpareThreads="25" maxSpareThreads="75” enableLookups="false"
redirectPort="8443" acceptCount="100”debug="0" connectionTimeout="20000” disableUploadTimeout="true" />
<Connector port="9099” enableLookups="false” protocol="AJP/1.3" />
Nov 12, 2009 1:40:04 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-9080
Nov 12, 2009 1:40:04 PM org.apache.coyote.ajp.AjpAprProtocol init
INFO: Initializing Coyote AJP/1.3 on ajp-9099
Nov 12, 2009 1:40:04 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 723 ms
Nov 12, 2009 1:40:04 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Nov 12, 2009 1:40:04 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.20
Nov 12, 2009 1:40:19 PM org.apache.coyote.http11.Http11AprProtocol start
INFO: Starting Coyote HTTP/1.1 on http-9080
Nov 12, 2009 1:40:19 PM org.apache.coyote.ajp.AjpAprProtocol start
INFO: Starting Coyote AJP/1.3 on ajp-9099
Nov 12, 2009 1:40:19 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 14982 ms
Performance: APR
The following output will be present in the logs:
16
Heap and Garbage Collection
• Some optimizations for the JVM:
– -Xms256m (minimum Heap size): note that really big values cause the GC to run with
bigger delay until the heap minimum value is full
– -Xmx512m (maximum Heap space): prevent java.lang.OutOfMemory
– -XX:MaxPermSize=256m (Permament Generation): if your application loads many classes
(hundreds, thousands) then increase the value in a way that Xms>permSize<Xmx
– -server: the JIT compiler is optimized for application overall performance instead of
small footprint and fast startup
Performance: Heap and GC
– -XX:+AggressiveOpts (compiler optimizations, default on Sun jdk>1.5u6)
17
Heap and Garbage Collection
– If you think your system will be in high load, set the Xms and Xmx heap sizes to the same
value: the Garbage Collection execution can became more predictable and run faster
– If you receive OutOfMemory errors such as “unable to create a new thread”, then you
can decrease the amount of memory of the stack per-thread (-Xss512k for instance)
– If you receive errors regarding “too many open files”, then increase the number of them
in the OS:
#cat /proc/sys/fs/file-max
75000
#sysctl -w fs.file-max=371304
– Use a 64-bit OS if you need more than 2Gb of Heap space for your JVM: this is the max
value in 32bit OS
– Use Sun JDK 1.6u17 (current): differences of performance from 1.5 to 1.6 are in terms of
20/25%
Performance: Heap and GC
• Some considerations
18
Monitoring
– the status and the health of your applications
– how many resources and bandwidth the application X is consuming (in a way to adjust
the payment to your customer  )
– the monthly availability of your services, to ensure SLA
– that en email just arrived to your mailbox at 04:00 AM saying that Tomcat is down 
Monitoring: motivation
• You need to know:
19
• Hyperic HQ is a SpringSource (now division of VMware) JEE application for
monitoring of web applications, common services and network devices
• Server-Agent logic: agents deployed to N machine where services that
need monitoring are running
• Sigar API (System Information Gatherer), engineered by the main
developer of mod_perl Doug MacEachern
• Auto-discovery of services
• Tens of pre-configured templates for common services
Monitoring: Hyperic HQ
Monitoring: Hyperic HQ
20
Monitoring: Hyperic HQ
Monitoring: Hyperic HQ
21
Monitoring: Hyperic HQ Sigar
java -jar ./agent-4.1.2/bundles/agent-4.1.2-1053/pdk/lib/sigar.jar
sigar> ps State.Name.eq=java,Args.*.ct=ofbiz.jar,CredName.User.eq=deadfish
4989 deadfish Nov8 948M 329M 3.9M S
26:28 java:ofbiz.jar
sigar> pargs 4989
pid=4989
exe=???
cwd=???
0=>/usr/java/jdk1.6.0_10/bin/java<=
1=>-Xms256M<=
2=>-Xmx512M<=
3=>-Dofbiz.admin.port=10700<=
4=>-Dofbiz.admin.key=so3du5kasd5ln<=
5=>-Denv.ofbiz.home=/home/deadfish<=
6=>-XX:MaxPermSize=256m<=
7=>-server<=
8=>-Dcom.sun.management.jmxremote<=
9=>-Dcom.sun.management.jmxremote.port=10701<=
10=>-Dcom.sun.management.jmxremote.ssl=false<=
11=>-Dcom.sun.management.jmxremote.authenticate=false<=
12=>-jar<=
13=>ofbiz.jar<=
Monitoring: Hyperic HQ Sigar
• Usage example (with PTQL - Process Table Query Language)
We want to list Java processes that runs ofbiz.jar, owned by the user deadfish.
22
• The JMX technology provides a simple, standard way of managing
resources such as applications, devices, and services (Sun JDK> 5.0)
• The resources inside the JVM are instrumented by Java objects called
Managed Beans (MBeans) that act as JMX agents
• With JMX is possible to monitor and control remotely (even via SSL) the
resources used by your applications
To enable JMx on Tomcat, add the following in bin/catalina.sh:
[ $1 != "stop" ] && JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=6969 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false $JAVA_OPTS"
export JAVA_OPTS
Monitoring: Hyperic and JMX
Monitoring: Hyperic and JMX
23
Let see a demo…
Monitoring: Hyperic and JMX
Monitoring: Hyperic and JMX
24
Thanks from IntegratingWeb
Questions?
25