Showing posts with label Tuning. Show all posts
Showing posts with label Tuning. Show all posts

Friday, October 03, 2014

Obtain a thread dump and heap dump for Java process

Thread dump shows locks used by different threads.
Ways to obtain a thread dump
1. get pid using ps or jps
    jstack pid xyz.log --> thread dump
    jstack -l to print additional information about locks
2. kill -QUIT pid --> thread dump
    kill -3 2>&1
3. login to jconsole  and request a thread dump
4. jrcmd print_threads
5. Login to AdminConsole—>Server —> Monitoring —> Threads - See more at: http://middlewaremagic.com/weblogic/?p=823#sthash.5X2b9Ntr.dpuf

Heap Dump
1. jmap -dump:file=xyz.log PID
Use JMAP to dump heap and analyze using visualVM
2. JConsole to get HeapDump
3. use $JAVA_HOME/bin/jvisualvm to take a heap dump and analyze it. For remote systems, use vnc viewer or xserver to redirect console windows to local env

Wednesday, October 01, 2014

Java Application Tuning

Tuning objectives

Tools used
Jprofiler
JRockit mission control
Wily introscope
Fluke
Splunk
Wireshark

testing - load runner, winruuner, junit, jmeter

Measurement
CPU (ideally should not cross more than 50-75%)
Memory
IO usage
Response time
Latency
Throughput

Application level tuning

Java language tuning
Using hashmaps vs hashtable, array vs vector,
Use StringBuffer to Concatenate Strings
Assign null to Variables That Are No Longer Needed
Declare Constants as static final
Avoid Finalizers - Adding finalizers to code makes the garbage collector more expensive and unpredictable
Synchronize Only When Necessary

EJB

JNDI

Application Server
Precompile jsps

OS
File Descriptors
set rlim_fd_max = 8192
Verify this hard limit by using the following command:
ulimit -a -H
Once the above hard limit is set, increase the value of this property explicitly (up to this limit)using the following command:
ulimit -n 8192
Verify this limit by using the following command:
ulimit –a

DISK IO settings

TCP/IP settings

Virtual memory settings

Run TOP command in Linux to check CPU usage.
Run VMSTAT, SAR, PRSTAT command to get more information on CPU, memory usage and possible blocking.
Enable the trace file before running your queries,then check the trace file using tkprof create output file.
According to explain plan check the elapsed time for each query,then tune them respectively.

What is the use of iostat/vmstat/netstat command in Linux?
Iostat – reports on terminal, disk and tape I/O activity.
Vmstat – reports on virtual memory statistics for processes, disk, tape and CPU activity.
Netstat – reports on the contents of network data structures.

Caching

JVM tuning/Garbage collection
best GC algorithm, especially for web server applications, is the Concurrent Mark Sweep GC
operation mode to server

-Xms, -Xmx
-XX:PermSize=256 m -XX:MaxPermSize=256m --> this option is not there in new HotSpot JVM
Log GC -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails
-XX:+PrintGCDateStamps
HeapDump on Out of memory - -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps

jvm - space ratios,
gc type – CMS (concurrent mark and sweep, parallel), G1 (this is new in HotSpot)

In the event CPU usage rate is high - If TPS is low but CPU usage rate is high, this is likely to result from inefficient implementation. In this case, you should find out the location of bottlenecks by using a profiler. You can analyze this by using jvisuavm, TPTP of Eclipse or JProbe.
     
JMS tuning
quota - space for jms messages at queue or jms server level. if quota limit is reached, exception is raised
send timeout - blocking producers when quota limits are reached
paging messages
message buffer size - amount of memory stored in memory before messages are paged out
message compression
expired messages

JDBC code tuning
Use prepared statements. Use parametrized SQL.
Tune the SQL to minimize the data returned (e.g. not 'SELECT *').
Minimize transaction conflicts (e.g. locked rows).
Use connection pooling.
Try to combine queries and batch updates.
Use stored procedures.
Cache data to avoid repeated queries.
Close resources (Connections, Statements, ResultSets) when finished with.
Select the fastest JDBC driver.

JDBC parameter tuning
Increasing Performance with the Statement Cache
Connection Testing Options for a Data Source - test frequency/
Minimized Connection Test Delay After Database Connectivity Loss
Minimized Connection Request Delay After Connection Test Failures
Minimized Connection Request Delays After Loss of DBMS Connectivity
Minimizing Connection Request Delay with Seconds to Trust an Idle Pool Connection
A leaked connection is a connection that was not properly returned to the connection pool in the data source. To automatically recover leaked connections, you can specify a value for Inactive Connection Timeout on the JDBC Data Source

Threadpool/work manager
Contention/locks
Minimize I/O
Parellelization/concurrency

Hibernate tuning
property name="show_sql"

Tip 1 - Reduce primary key generation overhead
Tip 2 - Use JDBC batch inserts/updates
Tip 3 - Periodically flush and clear the Hibernate session
entityManager.flush();
entityManager.clear();

Tip 4 - Reduce Hibernate dirty-checking overhead
@Transactional(readOnly=true)
public void someBusinessMethod() {
}

Tip 5 - Search for 'bad' query plans - full table scans and full cartesian joins
Tip 6 - Use the second-level and query caches
Define lazy loading as the preferred association loading strategy,
Set ReadOnly to "true" on Queries and Criteria, when objects returned will never be modified.

Cache – 1st level = Session, 2nd = Hibernate, 3rd = Query results cache

SQL Tuning
How to see execution plan for an SQL
SQL> set autotrace onSQL> select count(*) from table_name;
SQL> explain plan set statement_id = '111' for select * from tableName;SQL> select * from plan_table;

How to find out cpu time for an SQL?
SQL> TIMING START select_empSQL> SELECT * FROM employee ; SQL> TIMING SHOW select_emp timing for: select_emp real: 1760
use indexes, improve h/w, caching, tune it, hints,
use explain plan or sql analyzer to see the execution path
add hints
rewrite sql
replace sql with pl/sql

How would you improve a poor performing query?

How do you tune a sql statement?

understand row level lock and how it migrates to table level lock

soa performance tuning - dehydrate, db persistence
http://docs.oracle.com/cd/E25178_01/core.1111/e10108/bpel.htm

Oracle HTTP server tuning
http://docs.oracle.com/cd/E25178_01/core.1111/e10108/http.htm#ASPER99006

Java Memory leaks

Causes of memory leaks in Java
The four typical causes of memory leaks in a Java program are:
  1. Unknown or unwanted object references: These objects are no longer needed, but the garbage collector can not reclaim the memory because another object still refers to it.
  1. Long-living (static) objects: These objects stay in the memory for the application's full lifetime. Objects tagged to the session may also have the same lifetime as the session, which is created per user and remains until the user logs out of the application. Example: Listeners, Static Collection classes, Connections, JNI
  1. Failure to clean up or free native system resources: Native system resources are resources allocated by a function external to Java, typically native code written in C or C++. Java Native Interface (JNI) APIs are used to embed native libraries/code into Java code.
  1. Bugs in the JDK or third-party libraries: Bugs in various versions of the JDK or in the Abstract Window Toolkit and Swing packages can cause memory leaks.

Symptoms
  • Causes Out of memory after some time
  • GC keeps reclaiming lesser heap each time (memory used for long lived objects increase over time)
  • Causes frequent full GC
  • Available free heap decreases each time
  • Response time decreases

Detection of  Memory leaks
  • Add JVM args to collect GC metrics --> verbose:gc,-XX:+PrintGCTimeStamps,-XX:+PrintGCDetails
  • Monitor heap. If heap size increases after each full GC, this means a memory leak.
  • Use HeapDumpOnOutOfMemoryError to dump heal to a file
  • Use JMAP to dump heap and analyze using visualVM
    • jmap -dump:file=xyz.log PID
  • Use profilers

Other technique to detect memory leak
So Java Memory Leaks occur when objects still have a GC root reference, but are not actually used anymore. Those “Loitering Objects” stay around for the whole life of the JVM. If the application is creating those “dead objects” on and on, the memory will be filled up and eventually result in a java.lang.OutOfMemoryError. Typical causes are static collections, which are used as a kind of cache. Usually objects are added, but never removed (Let’s face it: How often have you used add() and put() and how often used remove() methods?). Because the objects are referenced by the static collection, they cannot be freed up anymore, as the collection has a GC root reference via the classloader.

Examples of memory leaks
Not closing resultset and statement objects explicitly…

JVM Performance Tuning

On a multi-processor machine with 8 CPUs, here are typical settings:

On a 8 processor
java 1.6 update 45

 Heap Size: -Xms2g -Xmx2g  
 Perm Size: -XX: PermSize=512m -XX: MaxPermSize=512m  
 Newgen Size(Fixed Size) : -Xmn1g  
 Garbage Collector(Parallel): -XX:+UseParallelOldGC -XX:ParallelGCThreads=8 -XX:LargePageSizeInBytes=4m -XX:+UseCompressedOops  
 GC Statistics: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps - Xloggc:/appserver/Weblogic/admin/360control_perf_domain/mserver/360control_perf_domain/logs/gc_360control.$(date +"%Y-%m-%d").log  


http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html

http://www.petefreitag.com/articles/gctuning/
http://java.sun.com/docs/hotspot/gc1.4.2/faq.html
http://java.sun.com/developer/technicalArticles/Programming/turbo/


The garbage collector first performs a task called marking. The garbage collector traverses the application graph, starting with the root objects; those are objects that are represented by all active stack frames and all the static variables loaded into the system. Each object the garbage collector meets is marked as being used, and will not be deleted in the sweeping stage.

The sweeping stage is where the deletion of objects take place. There are many ways to delete an object: The traditional C way was to mark the space as free, and let the allocator methods use complex data structures to search the memory for the required free space. This was later improved by providing a defragmenting system which compacted memory by moving objects closer to each other, removing any fragments of free space and therefore allowing allocation to be much faster:

For the last trick to be possible a new idea was introduced in garbage collected languages: even though objects are represented by references, much like in C, they don't really reference their real memory location. Instead, they refer to a location in a dictionary which keeps track of where the object is at any moment.

Fortunately for us – but unfortunately for these garbage collection algorithms – our servers and personal computers got faster (and multiple) processors and bigger memory capacities. Compacting memory areas this large often was very taxing on the application, especially considering that when doing that, the whole application had to freeze due to the changes in the virtual memory map. Fortunately for us though, some smart people improved those algorithms in three ways: concurrency, parallelization and generational collection.


Generational garbage collection

In any application, objects could be categorized according to their life-line. Some objects are short-lived, such as most local variables, and some are long-lived such as the backbone of the application. The thought about generational garbage collection was made possible with the understanding that in an application's lifetime, most instantiated objects are short-lived, and that there are few connections between long-lived objects to short-lived objects.

In order to take advantage of this information, the memory space is divided to two sections: young generation and old generation. In Java, the long-lived objects are further divided again to permanent objects and old generation objects. Permanent objects are usually objects the Java VM itself created for caching like code, reflection information etc. Old generation objects are objects that survived a few collections in the young generation area.

Since we know that objects in the young generation memory space become garbage early, we collect that area frequently while leaving the old generation's memory space to be collected in larger intervals. The young generation memory space is much smaller, thus having shorter collection times.

An additional advantage to the knowledge that objects die quickly in this area, we can also skip the compacting step and do something else called copying. This means that instead of seeking free areas (by seeking the areas marked as unused after the marking step), we copy the live objects from one young generation area to another young generation area. The originating area is called the From area, and the target area is called the To area, and after the copying is completed the roles switch: the From becomes the To, and the To becomes the From.

In addition, the Java VM splits the young generation to three areas, by adding an area called Eden which is where all objects are allocated into. To my understanding this is done to make allocation faster by always having the allocator reference to the beginning of Eden after a collection.

By using the copying method, garbage collection achieves defragmentation without seeking for dead memory blocks. However, this method proves itself to be more efficient in areas where most objects are garbage, so it is not a good approach to take on the old generation memory area. Indeed, that area is still collected using the compacting algorithm – but now, thanks to the separation of young and old generations, it is done in much larger intervals.


OutOfMemoryError (on HeapSpace, PermGen) and JVM Performance Tuning
HeapSpace OutOfMemoryError:


"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Most of you would have encountered this error before. This means JVM's usage of heap space exceeded what you specified in -Xmx vm argument or the default max heap size (if you did not specify one).

When you get this error, before you increase your -Xmx value, make sure to profile your application and see whether the usage is justified or if there is any memory leak. See below section on profiling tools.

PermGen OutOfMemoryError:

"java.lang.OutOfMemoryError: PermGen space"

Sometimes, you would also get a different kind of out of memory error. In JVM heap, there are different generations - Young, Tenured and PermGen. Young and tenured generations are where the regular objects from your application are stored. Size of young + tenured generation is controlled by -Xmx variable you saw above. PermGen is where JVM stores string pool, metadata about your application's classes etc.,

This error would normally happen only if your application loads a large number of classes, but is not unloading for some reason, which could be either legitimate or a memory leak. Also, make sure to check if your application heavily uses String.intern(), which could also contribute to this error.