WSO2 ESB calling a REST endpoint which expects a not chunked request

December 11, 2014, 6:24 pm

≫ Next: 12 Useful tcpdump commands you can use to troubleshoot network issues

WSO2 ESB provides the following property mediator to remove the "Chunked" encoding type when calling SOAP endpoints. You can use the below configuration to call a SOAP endpoint which expects a non-chunked request (Request with "Content-Length" header).

<property name=”DISABLE_CHUNKING” value=”true” scope=”axis2″/>

Here is a sample proxy service configuration.

<?xml version=”1.0″ encoding=”UTF-8″?>
<proxy xmlns=”http://ws.apache.org/ns/synapse”
name=”SampleProxy_v1″
transports=”https”
startOnLoad=”true”
trace=”enable”>
<description/>
<target>
<inSequence>
<property name=”DISABLE_CHUNKING” value=”true” scope=”axis2″/>
<send>
<endpoint>
<address uri="http://localhost:9764/services/HelloService"></address>
</endpoint>
</send>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</target>
<publishWSDL uri=”http://192.168.66.1:9764/services/HelloService?wsdl”/>
</proxy>

If you need to call a REST endpoint without the "chunked" encoding, sometimes the above property mediator would not work. In that kind of situation, if you need to send "Content-Length" header which means "non-chunked" request, you can use the following 2 properties.

<property name="FORCE_HTTP_CONTENT_LENGTH" value="true" scope="axis2"></property>
<property name="COPY_CONTENT_LENGTH_FROM_INCOMING" value="true" scope="axis2"></property>

Here is a sample API definition for WSO2 ESB.

<api xmlns="http://ws.apache.org/ns/synapse" name="SampleAPI" context="test">
<resource methods="POST" url-mapping="/*">
<inSequence>
<log level="custom">
<property name="msg" value="Executing IN sequence"></property>
</log>
<property name="FORCE_HTTP_CONTENT_LENGTH" value="true" scope="axis2"></property>
<property name="COPY_CONTENT_LENGTH_FROM_INCOMING" value="true" scope="axis2"></property>
<send>
<endpoint>
<address uri="https://www.google.com" format="rest"></address>
</endpoint>
</send>
</inSequence>
<outSequence>
<log level="custom">
<property name="msg" value="Sending response"></property>
</log>
<send></send>
</outSequence>
<faultSequence>
<log level="custom">
<property name="msg" value="Error occurred "></property>
</log>
<send></send>
</faultSequence>
</resource>
</api>

↧

12 Useful tcpdump commands you can use to troubleshoot network issues

December 11, 2014, 6:29 pm

≫ Next: Useful network level commands tutorials

≪ Previous: WSO2 ESB calling a REST endpoint which expects a not chunked request

tcpdump is a most powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface. It is available under most of the Linux/Unix based operating systems. tcpdump also gives us a option to save captured packets in a file for future analysis. It saves the file in a pcap format, that can be viewed by tcpdump command or a open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files.

How to Install tcpdump in Linux

Many of Linux distributions already shipped with tcpdump tool, if in case you don’t have it on systems, you can install it using following Yum command.

# yum install tcpdump

Once tcpdump tool is installed on systems, you can continue to browse following commands with their examples.

1. Capture Packets from Specific Interface

The command screen will scroll up until you interrupt and when we execute tcpdump command it will captures from all the interfaces, however with -i switch only capture from desire interface.

# tcpdump -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:33:31.976358 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 3500440357:3500440553, ack 3652628334, win 18760, length 196
11:33:31.976603 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 196, win 64487, length 0
11:33:31.977243 ARP, Request who-has tecmint.com tell 172.16.25.126, length 28
11:33:31.977359 ARP, Reply tecmint.com is-at 00:14:5e:67:26:1d (oui Unknown), length 46
11:33:31.977367 IP 172.16.25.126.54807 > tecmint.com: 4240+ PTR? 125.25.16.172.in-addr.arpa. (44)
11:33:31.977599 IP tecmint.com > 172.16.25.126.54807: 4240 NXDomain 0/1/0 (121)
11:33:31.977742 IP 172.16.25.126.44519 > tecmint.com: 40988+ PTR? 126.25.16.172.in-addr.arpa. (44)
11:33:32.028747 IP 172.16.20.33.netbios-ns > 172.16.31.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
11:33:32.112045 IP 172.16.21.153.netbios-ns > 172.16.31.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
11:33:32.115606 IP 172.16.21.144.netbios-ns > 172.16.31.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
11:33:32.156576 ARP, Request who-has 172.16.16.37 tell old-oraclehp1.midcorp.mid-day.com, length 46
11:33:32.348738 IP tecmint.com > 172.16.25.126.44519: 40988 NXDomain 0/1/0 (121)

2. Capture Only N Number of Packets

When you run tcpdump command it will capture all the packets for specified interface, until youHit cancel button. But using -c option, you can capture specified number of packets. The below example will only capture 6 packets.

# tcpdump -c 5 -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:40:20.281355 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 3500447285:3500447481, ack 3652629474, win 18760, length 196
11:40:20.281586 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 196, win 65235, length 0
11:40:20.282244 ARP, Request who-has tecmint.com tell 172.16.25.126, length 28
11:40:20.282360 ARP, Reply tecmint.com is-at 00:14:5e:67:26:1d (oui Unknown), length 46
11:40:20.282369 IP 172.16.25.126.53216 > tecmint.com.domain: 49504+ PTR? 125.25.16.172.in-addr.arpa. (44)
11:40:20.332494 IP tecmint.com.netbios-ssn > 172.16.26.17.nimaux: Flags [P.], seq 3058424861:3058424914, ack 693912021, win 64190, length 53 NBT Session Packet: Session Message
6 packets captured
23 packets received by filter
0 packets dropped by kernel

3. Print Captured Packets in ASCII

The below tcpdump command with option -A displays the package in ASCII format. It is a character-encoding scheme format.

# tcpdump -A -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:31:31.347508 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 3329372346:3329372542, ack 4193416789, win 17688, length 196
M.r0...vUP.E.X.......~.%..>N..oFk.........KQ..)Eq.d.,....r^l......m\.oyE....-....g~m..Xy.6..1.....c.O.@...o_..J....i.*.....2f.mQH...Q.c...6....9.v.gb........;..4.).UiCY]..9..x.)..Z.XF....'|..E......M..u.5.......ul
09:31:31.347760 IP 192.168.0.1.nokia-ann-ch1 > 192.168.0.2.ssh: Flags [.], ack 196, win 64351, length 0
M....vU.r1~P.._..........
^C09:31:31.349560 IP 192.168.0.2.46393 > b.resolvers.Level3.net.domain: 11148+ PTR? 1.0.168.192.in-addr.arpa. (42)
E..F..@.@............9.5.2.f+............1.0.168.192.in-addr.arpa.....

3 packets captured
11 packets received by filter
0 packets dropped by kernel

4. Display Available Interfaces

To list number of available interfaces on the system, run the following command with -D option.

# tcpdump -D

 1.eth0
2.eth1
3.usbmon1 (USB bus number 1)
4.usbmon2 (USB bus number 2)
5.usbmon3 (USB bus number 3)
6.usbmon4 (USB bus number 4)
7.usbmon5 (USB bus number 5)
8.any (Pseudo-device that captures on all interfaces)
9.lo

5. Display Captured Packets in HEX and ASCII

The following command with option -XX capture the data of each packet, including its link level header in HEX and ASCII format.

# tcpdump -XX -i eth0

11:51:18.974360 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 3509235537:3509235733, ack 3652638190, win 18760, length 196
        0x0000:  b8ac 6f2e 57b3 0001 6c99 1468 0800 4510  ..o.W...l..h..E.
        0x0010:  00ec 8783 4000 4006 275d ac10 197e ac10  ....@.@.']...~..
        0x0020:  197d 0016 1129 d12a af51 d9b6 d5ee 5018  .}...).*.Q....P.
        0x0030:  4948 8bfa 0000 0e12 ea4d 22d1 67c0 f123  IH.......M".g..#
        0x0040:  9013 8f68 aa70 29f3 2efc c512 5660 4fe8  ...h.p).....V`O.
        0x0050:  590a d631 f939 dd06 e36a 69ed cac2 95b6  Y..1.9...ji.....
        0x0060:  f8ba b42a 344b 8e56 a5c4 b3a2 ed82 c3a1  ...*4K.V........
        0x0070:  80c8 7980 11ac 9bd7 5b01 18d5 8180 4536  ..y.....[.....E6
        0x0080:  30fd 4f6d 4190 f66f 2e24 e877 ed23 8eb0  0.OmA..o.$.w.#..
        0x0090:  5a1d f3ec 4be4 e0fb 8553 7c85 17d9 866f  Z...K....S|....o
        0x00a0:  c279 0d9c 8f9d 445b 7b01 81eb 1b63 7f12  .y....D[{....c..
        0x00b0:  71b3 1357 52c7 cf00 95c6 c9f6 63b1 ca51  q..WR.......c..Q
        0x00c0:  0ac6 456e 0620 38e6 10cb 6139 fb2a a756  ..En..8...a9.*.V
        0x00d0:  37d6 c5f3 f5f3 d8e8 3316 d14f d7ab fd93  7.......3..O....
        0x00e0:  1137 61c1 6a5c b4d1 ddda 380a f782 d983  .7a.j\....8.....
        0x00f0:  62ff a5a9 bb39 4f80 668a                 b....9O.f.
11:51:18.974759 IP 172.16.25.126.60952 > mddc-01.midcorp.mid-day.com.domain: 14620+ PTR? 125.25.16.172.in-addr.arpa. (44)
        0x0000:  0014 5e67 261d 0001 6c99 1468 0800 4500  ..^g&...l..h..E.
        0x0010:  0048 5a83 4000 4011 5e25 ac10 197e ac10  .HZ.@.@.^%...~..
        0x0020:  105e ee18 0035 0034 8242 391c 0100 0001  .^...5.4.B9.....
        0x0030:  0000 0000 0000 0331 3235 0232 3502 3136  .......125.25.16
        0x0040:  0331 3732 0769 6e2d 6164 6472 0461 7270  .172.in-addr.arp
        0x0050:  6100 000c 0001                           a.....

6. Capture and Save Packets in a File

As we said, that tcpdump has a feature to capture and save the file in a .pcap format, to do this just execute command with -w option.

# tcpdump -w 0001.pcap -i eth0

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
4 packets captured
4 packets received by filter
0 packets dropped by kernel

7. Read Captured Packets File

To read and analyze captured packet 0001.pcap file use the command with -r option, as shown below.

# tcpdump -r 0001.pcap

reading from file 0001.pcap, link-type EN10MB (Ethernet)
09:59:34.839117 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 3353041614:3353041746, ack 4193563273, win 18760, length 132
09:59:34.963022 IP 192.168.0.1.nokia-ann-ch1 > 192.168.0.2.ssh: Flags [.], ack 132, win 65351, length 0
09:59:36.935309 IP 192.168.0.1.netbios-dgm > 192.168.0.255.netbios-dgm: NBT UDP PACKET(138)
09:59:37.528731 IP 192.168.0.1.nokia-ann-ch1 > 192.168.0.2.ssh: Flags [P.], seq 1:53, ack 132, win 65351, length 5

8. Capture IP address Packets

To capture packets for a specific interface, run the following command with option -n.

# tcpdump -n -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:07:03.952358 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 3509512873:3509513069, ack 3652639034, win 18760, length 196
12:07:03.952602 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 196, win 64171, length 0
12:07:03.953311 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 196:504, ack 1, win 18760, length 308
12:07:03.954288 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 504:668, ack 1, win 18760, length 164
12:07:03.954502 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 668, win 65535, length 0
12:07:03.955298 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 668:944, ack 1, win 18760, length 276
12:07:03.955425 IP 172.16.23.16.netbios-ns > 172.16.31.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
12:07:03.956299 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 944:1236, ack 1, win 18760, length 292
12:07:03.956535 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 1236, win 64967, length 0

9. Capture only TCP Packets.

To capture packets based on TCP port, run the following command with option tcp.

# tcpdump -i eth0 tcp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:10:36.216358 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 3509646029:3509646225, ack 3652640142, win 18760, length 196
12:10:36.216592 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 196, win 64687, length 0
12:10:36.219069 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 196:504, ack 1, win 18760, length 308
12:10:36.220039 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 504:668, ack 1, win 18760, length 164
12:10:36.220260 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 668, win 64215, length 0
12:10:36.222045 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 668:944, ack 1, win 18760, length 276
12:10:36.223036 IP 172.16.25.126.ssh > 172.16.25.125.apwi-rxspooler: Flags [P.], seq 944:1108, ack 1, win 18760, length 164
12:10:36.223252 IP 172.16.25.125.apwi-rxspooler > 172.16.25.126.ssh: Flags [.], ack 1108, win 65535, length 0
^C12:10:36.223461 IP mid-pay.midcorp.mid-day.com.netbios-ssn > 172.16.22.183.recipe: Flags [.], seq 283256512:283256513, ack 550465221, win 65531, length 1[|SMB]

10. Capture Packet from Specific Port

Let’s say you want to capture packets for specific port 22, execute the below command by specifying port number 22 as shown below.

# tcpdump -i eth0 port 22

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:37:49.056927 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 3364204694:3364204890, ack 4193655445, win 20904, length 196
10:37:49.196436 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 4294967244:196, ack 1, win 20904, length 248
10:37:49.196615 IP 192.168.0.1.nokia-ann-ch1 > 192.168.0.2.ssh: Flags [.], ack 196, win 64491, length 0
10:37:49.379298 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 196:616, ack 1, win 20904, length 420
10:37:49.381080 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 616:780, ack 1, win 20904, length 164
10:37:49.381322 IP 192.168.0.1.nokia-ann-ch1 > 192.168.0.2.ssh: Flags [.], ack 780, win 65535, length 0

11. Capture Packets from source IP

To capture packets from source IP, say you want to capture packets for 192.168.0.2, use the command as follows.

# tcpdump -i eth0 src 192.168.0.2

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:49:15.746474 IP 192.168.0.2.ssh > 192.168.0.1.nokia-ann-ch1: Flags [P.], seq 3364578842:3364579038, ack 4193668445, win 20904, length 196
10:49:15.748554 IP 192.168.0.2.56200 > b.resolvers.Level3.net.domain: 11289+ PTR? 1.0.168.192.in-addr.arpa. (42)
10:49:15.912165 IP 192.168.0.2.56234 > b.resolvers.Level3.net.domain: 53106+ PTR? 2.0.168.192.in-addr.arpa. (42)
10:49:16.074720 IP 192.168.0.2.33961 > b.resolvers.Level3.net.domain: 38447+ PTR? 2.2.2.4.in-addr.arpa. (38)

12. Capture Packets from destination IP

To capture packets from destination IP, say you want to capture packets for 50.116.66.139, use the command as follows.

# tcpdump -i eth0 dst 50.116.66.139

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:55:01.798591 IP 192.168.0.2.59896 > 50.116.66.139.http: Flags [.], ack 2480401451, win 318, options [nop,nop,TS val 7955710 ecr 804759402], length 0
10:55:05.527476 IP 192.168.0.2.59894 > 50.116.66.139.http: Flags [F.], seq 2521556029, ack 2164168606, win 245, options [nop,nop,TS val 7959439 ecr 804759284], length 0
10:55:05.626027 IP 192.168.0.2.59894 > 50.116.66.139.http: Flags [.], ack 2, win 245, options [nop,nop,TS val 7959537 ecr 804759787], length 0

This article may help you to explore tcpdump command in depth and also to capture and analysis packets in future. There are number of options available, you can use the options as per your requirement. Please share if you find this article useful through our comment box.

This blog post is an extract from the following site.

http://www.tecmint.com/12-tcpdump-commands-a-network-sniffer-tool/

↧

Useful network level commands tutorials

December 11, 2014, 6:33 pm

≫ Next: Implementing Rule based systems with WSO2 products - Learning tutorials and resources

≪ Previous: 12 Useful tcpdump commands you can use to troubleshoot network issues

Hers is a great web site which provides lot of tutorials on network level commands and use cases.

http://www.tecmint.com/20-netstat-commands-for-linux-network-management/

http://www.tecmint.com/35-practical-examples-of-linux-find-command/

http://www.tecmint.com/12-tcpdump-commands-a-network-sniffer-tool/

↧

Implementing Rule based systems with WSO2 products - Learning tutorials and resources

December 13, 2014, 11:01 pm

≫ Next: Learning GIT from level zero

≪ Previous: Useful network level commands tutorials

Here are some resources which you can use to learn about implementing rule based solutions using WSO2 products stack.

Writing Business rules with WSO2 Carbon platform

http://www.dimuthu.org/blog/2010/01/07/writing-business-rules-in-wso2-carbon-platform/

Samples available at WSO2 BRS documentation

https://docs.wso2.com/display/BRS210/Samples

Integrating business rules and the architecture of rules component

http://wso2.com/library/articles/2012/12/bringing-business-rules-soa/

Integrate business rules with WSO2 ESB and WSO2 BRS

http://wso2.com/library/articles/2011/04/integrate-rules-soa/

Integrate business rules with WSO2 BPS and WSO2BRS

http://wso2.com/library/articles/2011/05/integrate-business-rules-bpel/

Complex event processing and business rules integrations with SOA

http://wso2.com/library/articles/2011/07/complex-event-processing-business-rule-management-soa/

Developing business rule services with WSO2 Developer Studio

http://wso2.com/library/articles/2013/05/eclipse-plugin-wso2-business-rules-server/

↧

Learning GIT from level zero

February 7, 2015, 3:18 am

≫ Next: GIT Cheat Sheet for beginners

≪ Previous: Implementing Rule based systems with WSO2 products - Learning tutorials and resources

GIT is one of the heavily used version control systems in the software industry. In this blog post, I am covering the fundamental concepts of GIT vcs for a beginner level user.

create a new repository

create a new directory, open it and perform a
git init
to create a new git repository.

checkout a repository

create a working copy of a local repository by running the command
git clone /path/to/repository

when using a remote server, your command will be
git clone username@host:/path/to/repository

workflow

your local repository consists of three "trees" maintained by git. the first one is your Working Directory which holds the actual files. the second one is the Index which acts as a staging area and finally the HEAD which points to the last commit you've made.

add & commit

You can propose changes (add it to the Index) using
git add <filename>
git add *

This is the first step in the basic git workflow. To actually commit these changes use
git commit -m "Commit message"
Now the file is committed to the HEAD, but not in your remote repository yet.

pushing changes

Your changes are now in the HEAD of your local working copy. To send those changes to your remote repository, execute
git push origin master
Change master to whatever branch you want to push your changes to.

If you have not cloned an existing repository and want to connect your repository to a remote server, you need to add it with
git remote add origin <server>
Now you are able to push your changes to the selected remote server

branching

Branches are used to develop features isolated from each other. The master branch is the "default" branch when you create a repository. Use other branches for development and merge them back to the master branch upon completion.

create a new branch named "feature_x" and switch to it using
git checkout -b feature_x

switch back to master
git checkout master

and delete the branch again
git branch -d feature_x

a branch is not available to others unless you push the branch to your remote repository
git push origin <branch>

update & merge

to update your local repository to the newest commit, execute
git pull
in your working directory to fetch and merge remote changes.

to merge another branch into your active branch (e.g. master), use
git merge <branch>

in both cases git tries to auto-merge changes. Unfortunately, this is not always possible and results in conflicts. You are responsible to merge those conflicts manually by editing the files shown by git. After changing, you need to mark them as merged with
git add <filename>

before merging changes, you can also preview them by using
git diff <source_branch> <target_branch>

tagging

it's recommended to create tags for software releases. this is a known concept, which also exists in SVN. You can create a new tag named 1.0.0 by executing
git tag 1.0.0 1b2e1d63ff

the 1b2e1d63ff stands for the first 10 characters of the commit id you want to reference with your tag. You can get the commit id by looking at the...

log

in its simplest form, you can study repository history using.. git log
You can add a lot of parameters to make the log look like what you want. To see only the commits of a certain author:
git log --author=bob

To see a very compressed log where each commit is one line:
git log --pretty=oneline

Or maybe you want to see an ASCII art tree of all the branches, decorated with the names of tags and branches:
git log --graph --oneline --decorate --all

See only which files have changed:
git log --name-status

These are just a few of the possible parameters you can use. For more, see git log --help

replace local changes

In case you did something wrong (which for sure never happens ;) you can replace local changes using the command
git checkout -- <filename>

this replaces the changes in your working tree with the last content in HEAD. Changes already added to the index, as well as new files, will be kept.

If you instead want to drop all your local changes and commits, fetch the latest history from the server and point your local master branch at it like this
git fetch origin
git reset --hard origin/master

useful hints

built-in git GUI
gitk

use colorful git output
git config color.ui true

show log on just one line per commit
git config format.pretty oneline

use interactive adding
git add -i

This blog post is prepared with the reference of this content.

http://rogerdudler.github.io/git-guide/

↧

GIT Cheat Sheet for beginners

February 7, 2015, 6:16 am

≫ Next: Simple tutorial to work with WSO2 GIT source code for developers

≪ Previous: Learning GIT from level zero

CREATE
Clone an existing repository
$ git clone ssh://user@domain.com/repo.git
Create a new local repository
$ git init

LOCAL CHANGES
Changed files in your working directory
$ git status
Changes to tracked files
$ git diff
Add all current changes to the next commit
$ git add .
Add some changes in <file> to the next commit
$ git add -p <file>
Commit all local changes in tracked files
$ git commit -a
Commit previously staged changes
$ git commit
Change the last commit
Don‘t amend published commits!
$ git commit --amend

COMMIT HISTORY
Show all commits, starting with newest
$ git log
Show changes over time for a specific file
$ git log -p <file>
Who changed what and when in <file>
$ git blame <file>

BRANCHES & TAGS
List all existing branches
$ git branch
Switch HEAD branch
$ git checkout <branch>
Create a new branch based on your current HEAD
$ git branch <new-branch>
Create a new tracking branch based on a remote branch
$ git checkout --track <remote/branch>
Delete a local branch
$ git branch -d <branch>
Mark the current commit with a tag
$ git tag <tag-name>

UPDATE & PUBLISH
List all currently configured remotes
$ git remote -v
Show information about a remote
$ git remote show <remote>
Add new remote repository, named <remote>
$ git remote add <remote> <url>
Download all changes from <remote>,
but don‘t integrate into HEAD
$ git fetch <remote>
Download changes and directly merge/ integrate into HEAD
$ git pull <remote> <branch>
Publish local changes on a remote
$ git push <remote> <branch>
Delete a branch on the remote
$ git branch -dr <remote/branch>
Publish your tags
$ git push --tags

MERGE & REBASE
Merge <branch> into your current HEAD
$ git merge <branch>
Rebase your current HEAD onto <branch>
Don‘t rebase published commits!
$ git rebase <branch>
Abort a rebase
$ git rebase --abort
Continue a rebase after resolving conflicts $ git rebase --continue
Use your configured merge tool to solve conflicts
$ git mergetool
Use your editor to manually solve conflicts and (after resolving) mark file as resolved
$ git add <resolved-file>
$ git rm <resolved-file>

UNDO
Discard all local changes in your working directory
$ git reset --hard HEAD
Discard local changes in a specific file
$ git checkout HEAD <file>
Revert a commit (by producing a new commit with contrary changes)
$ git revert <commit>
Reset your HEAD pointer to a previous commit
...and discard all changes since then
$ git reset --hard <commit>
...and preserve all changes as unstaged changes
$ git reset <commit>
...and preserve uncommitted local changes
$ git reset --keep <commit>

↧

Simple tutorial to work with WSO2 GIT source code for developers

February 11, 2015, 7:53 pm

≫ Next: Enabling audit logs for WSO2 carbon based servers

≪ Previous: GIT Cheat Sheet for beginners

Some important Git commands for branching with remotes and upstream projects

First create a fork from an existing repository on wso2 repo

https://github.com/wso2

Then create a cloned repository in your local machine from that forked repo. Replace the URL with your own repo URL.

git clone https://github.com/chanakaudaya/carbon-mediation.git

Now add upstream repositories for your local clones repository.This should be the upstream project which you have forked from.

git remote add upstream https://github.com/wso2/carbon-mediation.git

You can remove an upstream if you needed.

git remote rm upstream

Now you can see all the available branches in your git metadata repository. Here you need to keep in mind that only metadata available for all the branches except the master (which is the local branch created by default)

git branch -a

* master

remotes/origin/HEAD -> origin/master

remotes/origin/master

remotes/origin/release-2.1.3-wso2v1

remotes/origin/release-2.1.3-wso2v2

remotes/upstream/master

remotes/upstream/release-2.1.3-wso2v1

remotes/upstream/release-2.1.3-wso2v2

If you cannot see the upstream branches list from the above command, just fetch them with the following command.

git fetch upstream

Now you should see the branched list as above.

This will list down all the available branches. Then decide on which branch you are going to work on. Let’s say you are going to work on release-2.1.3-wso2v2 branch. Now you should create a tracking branch in your local repository for that branch.

git checkout --track origin/release-2.1.3-wso2v2

Here we are creating a tracking local branch to track the origin (your forked repository, not the upstream). Now if you look at the branch list, you can see the newly created branch.

git branch -a

master

* release-2.1.3-wso2v2

remotes/origin/HEAD -> origin/master

remotes/origin/master

remotes/origin/release-2.1.3-wso2v1

remotes/origin/release-2.1.3-wso2v2

remotes/upstream/master

remotes/upstream/release-2.1.3-wso2v1

remotes/upstream/release-2.1.3-wso2v2

Now you need to see what is the link between your remote branches and local branches. You can do that with the following command.

git remote show origin

This will give you the information about the connection between your local and remote branches.

* remote origin

Fetch URL: https://github.com/chanakaudaya/wso2-synapse.git

Push URL: https://github.com/chanakaudaya/wso2-synapse.git

HEAD branch: master

Remote branches:

master tracked

release-2.1.3-wso2v1 tracked

release-2.1.3-wso2v2 tracked

Local branches configured for 'git pull':

master merges with remote master

release-2.1.3-wso2v2 merges with remote release-2.1.3-wso2v2

Local refs configured for 'git push':

master pushes to master (up to date)

release-2.1.3-wso2v2 pushes to release-2.1.3-wso2v2 (up to date)

↧

Enabling audit logs for WSO2 carbon based servers

March 12, 2015, 11:59 am

≫ Next: Understanding WSO2 ESB Pass-Through Transport concepts

≪ Previous: Simple tutorial to work with WSO2 GIT source code for developers

Audit logs provide very useful information related to the users who has tried to access the server. By default, most of the WSO2 carbon based products (ESB, APIM, DSS) have not enabled this logging. In production environments, it is always better to enable audit logs due to various reasons.

All you need to do is add the following section to log4j.properties file which resides in <CARBON_HOME>/repository/conf directory.

# Configure audit log for auditing purposes
log4j.logger.AUDIT_LOG=INFO, AUDIT_LOGFILE
log4j.appender.AUDIT_LOGFILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.AUDIT_LOGFILE.File=${carbon.home}/repository/logs/audit.log
log4j.appender.AUDIT_LOGFILE.Append=true
log4j.appender.AUDIT_LOGFILE.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout
log4j.appender.AUDIT_LOGFILE.layout.ConversionPattern=[%d] %P%5p - %x %m %n
log4j.appender.AUDIT_LOGFILE.layout.TenantPattern=%U%@%D [%T] [%S]
log4j.appender.AUDIT_LOGFILE.threshold=INFO
log4j.additivity.AUDIT_LOG=false

Once you enable this, you can see the audit log file is created under <CARBON_HOME>/repository/logs directory. It will contain information similar to below mentioned lines.

[2015-03-12 10:44:01,565] INFO - 'admin@carbon.super [-1234]' logged in at [2015-03-12 10:44:01,565-0500]
[2015-03-12 10:44:45,825] INFO - User admin successfully authenticated to perform JMX operations.
[2015-03-12 10:44:45,826] INFO - User : admin successfully authorized to perform JMX operations.
[2015-03-12 10:44:45,851] WARN - Unauthorized access attempt to JMX operation.
java.lang.SecurityException: Login failed for user : jmx_user. Invalid username or password.
at org.wso2.carbon.core.security.CarbonJMXAuthenticator.authenticate(CarbonJMXAuthenticator.java:124)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

↧

Understanding WSO2 ESB Pass-Through Transport concepts

May 2, 2015, 10:49 pm

≫ Next: WSO2 ESB Development best practices

≪ Previous: Enabling audit logs for WSO2 carbon based servers

WSO2 ESB is arguably the highest performing open source ESB available in the industry. It has been tested and deployed in various enterprise environments with serving zillions of transactions. The ultimate reason behind this resounding success is it's high performance HTTP transport implementation known as Pass-Through Transport which provides efficient mechanism to process client requests in a highly concurrent scenarios. In this blog post/s we are going to give you some introduction in to PTT by covering some of the theories behind this implementation. The actual implementation is far more complex than described in this tutorial and I have added some useful resources whenever possible to provide you more information.

Most of the content present in this tutorial are extracted from following blog posts/tutorials on the web. I would like to give credit to those authors for writing such great content.

http://tutorials.jenkov.com/java-nio/index.html

http://jeewanthad.blogspot.com/2013/02/reactor-pattern-explained-part-1.html

http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf

http://wso2.com/library/articles/2014/04/demystifying-wso2-esb-pass-through-transport-part-iii/

http://techfeast-hiranya.blogspot.com/2013/02/how-worlds-fastest-esb-was-made.html

Pass-Through transport uses the httpcore-nio library as the underlying implementation library which uses the java NIO api for the implementation. First we will go through the concepts of java NIO and then go through httpcore-nio concepts and finally the PTT implementation.

Java Non-blocking IO (NIO)

In the standard IO API you work with byte streams and character streams. In NIO you work with channels and buffers. Data is always read from a channel into a buffer, or written from a buffer to a channel. Java NIO enables you to do non-blocking input output. As an example, a thread can ask a channel to read data into a buffer. While the channel reads data in to the buffer, the thread can do something else. Once the data is read into the buffer, the thread can continue processing it. In standard IO, java uses byte streams and character streams. But in NIO, java uses channels and buffers. Data is always read from channel to buffer or write from a buffer to channel.

Channels

Java NIO Channels are similar to streams with a few differences:

You can both read and write to a Channels. Streams are typically one-way (read or write).
Channels can be read and written asynchronously.
Channels always read to, or write from, a Buffer.

As mentioned above, you read data from a channel into a buffer, and write data from a buffer into a channel. Here are the most important Channel implementations in Java NIO:

FileChannel - The FileChannel reads data from and to files.

DatagramChannel - The DatagramChannel can read and write data over the network via UDP.

SocketChannel - The SocketChannel can read and write data over the network via TCP.

ServerSocketChannel - The ServerSocketChannel allows you to listen for incoming TCP connections, like a web server does. For each incoming connection a SocketChannel is created.

Buffers

Buffers are used when interacting with NIO channels. Buffer is a block of memory into which data can be written and later can read from. Buffer provides set of methods which makes it easier to work with the memory block.

Using a Buffer to read and write data typically follows this little 4-step process:

1Write data into the Buffer

2Call buffer.flip()

3Read data out of the Buffer

4Call buffer.clear() or buffer.compact()

When you write data into a buffer, the buffer keeps track of how much data you have written. Once you need to read the data, you need to switch the buffer from writing mode into reading mode using the flip() method call. In reading mode the buffer lets you read all the data written into the buffer.

Selectors

A selector is an object that can monitor multiple channels for events. Therefore, a single thread can monitor multiple channels for data. A Selector allows a single thread to handle multiple Channels. This is a very useful concept in cases where your application has many open connections but has low traffic on each connection. To use a Selector, you register the Channels with it. Then you call it’s select() method. This method will block until there is an event ready for one of the registered channels. Once the method returns, the thread can process the events.

Now we have the basic understanding about the concepts related to Java NIO and let's take a look at the actual problem we are going to solve with this technology.

Network services and Reactor pattern

Web services, Distributed Objects, etc Most have same basic structure:

Read request
Decode request
Process service
Encode reply
Send reply

But differ in nature and cost of each step.

XML parsing, File transfer, Web page generation, computational services

In a classical server design, there will be new handler thread for each client connection.

This approach makes following scalability challenges.

Graceful degradation under increasing load (more clients)
Continuous improvement with increasing resources (CPU, memory, disk, bandwidth)
Also meet availability and performance goals
Short latencies
Meeting peak demand Tunable quality of service

Divide processing into small tasks makes it more effective when processing high and variable loads.

Each task performs an action without blocking
Execute each task when it is enabled Here, an IO event usually serves as trigger
Basic mechanisms supported in java.nio Non-blocking reads and writes
Dispatch tasks associated with sensed IO events
Usually more efficient than alternatives Fewer resources
Don't usually need a thread per client Less overhead
Less context switching, often less locking But dispatching can be slower
Must manually bind actions to events Usually harder to program
Must break up into simple non-blocking actions
Similar to GUI event-driven actions
Cannot eliminate all blocking: GC, page faults, etc
Must keep track of logical state of service

According to the above comparison, we can clearly see that dividing the task in to small non-blocking small operations make it more efficient. But programming this model is more complex than the first approach. Reactor pattern is used to implement this behavior.

Reactor pattern

Basic reactor pattern can be depicted with the below diagram.

A Reactor runs in a separate thread and its job is to react to IO events by dispatching the work to the appropriate handler. Its like a telephone operator in a company who answers the calls from clients and transfers the communication line to the appropriate receiver.

A Handler performs the actual work to be done with an IO event similar to the actual officer in the company the client who called wants to speak to.

Selection Keys maintain IO event status and bindings. Its a representation of the relationship between a Selector and a Channel. By looking at the Selection Key given by the Selector, the Reactor can decide what to do with the IO event which occurs on the Channel.

Here, there is a single ServerSocketChannel which is registered with a Selector. The SelectionKey 0 for this registration has information on what to do with the ServerSocketChannel if it gets an event. Obviously the ServerSocketChannel should receive events from incoming connection requests from clients. When a client requests for a connection and wants to have a dedicated SocketChannel, the ServerSocketChannel should get triggered with an IO event. What does the Reactor have to do with this event? It simply has to Accept it to make a SocketChannel. Therefore SelectionKey 0 will be bound to an Acceptor which is a special handler made to accept connections so that the Reactor can figure out that the event should be dispatched to the Acceptor by looking at SelectionKey 0. Notice that ServerSocketChannel, SelectionKey 0 and Acceptor are all in same colour ( light green )

The Selector is made to keep looking for IO events. When the Reactor calls Selector.select() method, the Selector will provide a set of SelectionKeys for the channels which have pending events. When SelectionKey 0 is selected, it means that an event has occurred on ServerSocketChannel. So the Reactor will dispatch the event to the Acceptor.

When the Acceptor accepts the connection from Client 1, it will create a dedicated SocketChannel1 for the client. This SocketChannel will be registered with the same Selector with SelectionKey 1. What would the client do with this SocketChannel? It will simply read from and write to the server. The server does not need to accept connections from client 1 any more since it already accepted the connection. Now what the server needs is to Read and Write data to the channel. So SelectionKey 1 will be bound to Handler 1 object which handles reading and writing. Notice that SocketChannel 1, SelectionKey 1 and Handler 1 are all in Green.

The next time the Reactor calles Selector.select(), if the returned SelectionKey Set has SelectionKey 1 in it, it means that SocketChannel 1 is triggered with an event. Now by looking at SelectionKey 1, the Reactor knows that it has to dispatch the event to Handler 1 since Hander 1 is bound to SelectionKey 1. If the returned SelectionKey Set has SelectionKey 0 in it, it means that ServerSocketChannel has received an event from another client and by looking at the SelectionKey 0 the Reactor knows that it has to dispatch the event to the Acceptor again. When the event is dispatched to the Acceptor it will make SocketChannel 2 for client 2 and register the socket channel with the Selector with SelectionKey 2.

So in this scenario we are interested in 3 types of events.

1Connection request events which get triggered on the ServerSocketChannel which we need to Accept.

2Read events which get triggerd on SocketChannels when they have data to be read, from which we need to Read.

3Write events which get triggered on SocketChannels when they are ready to be written with data, to which we need to Write.

This is the theory behind the reactor pattern and this pattern is implemented in the apache httpcore-nio library. WSO2 ESB PTT uses this library as the underlying realization of the reactor pattern.

This gives a basic understanding about the reactor pattern and Java NIO framework. Let’s map this knowledge to Passthrough transport implementation of WSO2 ESB.

You need to download following items before we continue debugging the code.

WSO2 ESB 4.8.1 - http://wso2.com/products/enterprise-service-bus/

Apache-Synapse Source - https://svn.wso2.org/repos/wso2/carbon/platform/tags/turing-chunk07/dependencies/synapse/2.1.2-wso2v4/

Http-Core-NIO Source - https://svn.wso2.org/repos/wso2/carbon/platform/tags/turing-chunk07/dependencies/httpcore-nio/4.2.3-wso2v3/

Apache axis2 core source - https://svn.wso2.org/repos/wso2/carbon/kernel/tags/4.2.0/dependencies/axis2/1.6.1-wso2v10/

Apache axis2 transports source - https://svn.wso2.org/repos/wso2/carbon/kernel/tags/4.2.0/dependencies/transports/1.1.0-wso2v9/

Once you download all the components, extract the ESB 4.8.1 distribution to a location (ESB_HOME) and then import the maven projects to your favorite IDE (Intellij Idea or Eclipse) and create a remote debugging configuration (with port 5005) to debug in to the source code.

Then start the ESB with the following command

sh ESB_HOME/bin/wso2server.sh -debug 5005

This will start the ESB in the debug mode and now you can start your remote debugging session from your IDE. Once it is connected with the ESB, it will startup the server. During the server startup, you can observe the following INFO logs printed in the console.

[2015-04-04 13:54:48,996] INFO - PassThroughHttpSSLSender Pass-through HTTPS Sender started...

[2015-04-04 13:54:48,996] INFO - PassThroughHttpSender Initializing Pass-through HTTP/S Sender...

[2015-04-04 13:54:48,998] INFO - PassThroughHttpSender Pass-through HTTP Sender started...

[2015-04-04 13:54:54,370] INFO - PassThroughHttpSSLListener Initializing Pass-through HTTP/S Listener...

[2015-04-04 13:54:56,114] INFO - PassThroughHttpListener Initializing Pass-through HTTP/S Listener…

The above log lines will confirm that 4 main components of the ESB message flow has been started with the server startup. These transport listener and sender classes are configured in the axis2.xml file.

PassThroughHttpSSLSender - ( HTTPS transport for sending messages from ESB to back end )

PassThroughHttpSender - ( HTTP transport for sending messages from ESB to back end )

PassThroughHttpSSLListener - ( HTTPS transport for receiving messages to ESB from clients )

PassThroughHttpListener - ( HTTP transport for receiving messages to ESB from clients )

During the server startup, these components will be started from the CarbonServerManager class.

Let’s add some debug point in to PassThroughHttpListener class (#init method) and see what is happening inside this class initialization.

Within the init() method of this class, it creates the following 3 major objects.

ServerConnFactory - Factory class for creating connections

SourceHandler - This is the class where transport interacts with the client. This class receives events for a particular connection. These events give information about the message and its various states.

ServerIODispatch - This class receives various events from http core and dispatch them to PTT level code (SourceHandler)

connFactory = connFactoryBuilder.build(sourceConfiguration.getHttpParams());

handler = new SourceHandler(sourceConfiguration);

ioEventDispatch = new ServerIODispatch(handler, connFactory);

Within the start() method, it creates Reactor object with a thread group and calls the execute() method within a separate thread.

This will call the execute method of the AbstractMultiworkerIOReactor(httpcore-nio) class in which it will start N number of Worker threads (N equals to number of cores in the processor) with the prefix HTTP-Listener I/O dispatcher. After starting these worker threads, this class will go in to infinite loop to process the events received by the selector. Within this loop, it will process all the connection requests from the clients. This class acts as the Acceptor of the reactor pattern. It will create a new socketChannel and add that to the channel list of the dispatcher object.

These worker threads will execute the execute() method of the BaseIOReactor class which eventually calls the AbstractIOReactor class’s execute method. This will execute infinite for loop for processing the IO events. Within this infinite loop, it will first process the existing events which can be processed (ex: events of already registered channels which can accept events). After processing, it will check for new channels added and create sessions for newly added channels for future processing.

Now we have an understanding about how the requests are processed at the http core level. Once the client sends an http request to the ESB, it will trigger a series of events at the IO level within the ESB server side. This series of events is modeled in to a state machine within the http core level.

Converting of incoming events to this state machine is done within the http core level.

client sends a request to the ESB. This will be picked by the AbstractMultiworkerIOReactor thread and create a new socketChannel for this request and add this channel to reactor threads channel list and notify the selector (wakeup()).
Once this notification is received by the worker thread, it will execute the processNewChannells() method within the AbstractIOReactor and During this process it will create a new HTTP session and call the Connected method on the SourceHandler (State Connected)
Then it will go into the processEvents() method of the AbstractIOReactor class and process the remaining IO events for this channel. During this process it will consume the incoming message and change the state and call the requestReceived method from DefaultNHttpServerConnection class. This will also call the inputReady method.

When sending a request from ESB to back end server, message flow is mentioned below.

Message flow should contain either send or call mediator to send a message to a back end server. From either of this mediators, it calls the

Axis2SynapseEnvironment.send()

Axis2Sender.sendon()

Axis2FlexibleMEPClient.send()

OperationClient.execute()

OutInOperationClient.executeImpl()

AxisEngine.send()

PassThroughHttpSender.invoke()

DeliveryAgent.submit()

TargetConnections.getConnection()

DefaultConnectingIOReactor.connect()

requestQueue.add()

selector.wakeup()

DefaultConnectingIOReactor.processEvents()

DefaultConnectingIOReactor.processSessionRequests()

DefaultConnectingIOReactor.processEvent()

AbstractIOReactor.processNewChannels()

BaseIOReactor.sessionCreated()

AbstractIODispatch.connected()

ClientIODispatch.onConnected()

TargetHandler.Connected()

Detailed description about the internal state transition could be found in the following article.

http://wso2.com/library/articles/2014/04/demystifying-wso2-esb-pass-through-transport-part-iii/

This will give you some understanding about the PTT implementation. Idea of this blog post is to give you a starting point to study about the complex implementation of PTT. Here are some important links to study more about the PTT.

Java NIO -

http://www.ibm.com/developerworks/java/tutorials/j-nio/j-nio.html

http://rox-xmlrpc.sourceforge.net/niotut/

http://tutorials.jenkov.com/java-nio/index.html

Reactor pattern -

http://jeewanthad.blogspot.com/2013/02/reactor-pattern-explained-part-1.html

http://jeewanthad.blogspot.com/2013/03/reacter-pattern-explained-part-2.html

http://jeewanthad.blogspot.com/2013/03/reacter-pattern-explained-part-3.html

http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf

Pass-Through transport

http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-i/

http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-thru-transport-part-ii/

http://wso2.com/library/articles/2014/04/demystifying-wso2-esb-pass-through-transport-part-iii/

http://techfeast-hiranya.blogspot.com/2013/02/how-worlds-fastest-esb-was-made.html

http://shafreenanfar.blogspot.com/2014/02/wso2-esb-passthrough-transport-basics.html

↧

WSO2 ESB Development best practices

May 2, 2015, 10:56 pm

≫ Next: Performance Tuning WSO2 ESB with a practical example

≪ Previous: Understanding WSO2 ESB Pass-Through Transport concepts

Naming Conventions

When developing artifacts for WSO2 servers, follow the guidelines mentioned below.

Project names - Create project names which reflects the intention of the project. It is better to end the project name with the server type which this project is deployed in to. This name should be unique within your source code repository

Artifact names - When you create artifacts, always better to start the name with the project name and end with the artifact type. This would make the artifact name unique within the source repository.

Custom Java Classes - When you create custom java code (custom mediators, custom formatters), adhere to the standard java packaging names with project names to distinguish your code

Project Structures

When developing artifacts for WSO2 servers, follow the guidelines mentioned below

Start with a “Maven multi module” project as your parent project. This will be the top most level project in your source code hierarchy.

Create projects according to the artifact types which you are going to develop.

Ex: ESBConfigProject to develop ESB artifacts, RegistryResourceProject for governance artifacts

Create Composite ARchive (CAR) projects to bundle the relevant artifacts into an archive.

If you need to create a common (core) project to include different types of projects (ESB, Registry), create a maven multi module project and add them as sub projects.

Do not bundle custom mediators with the ESB artifacts which referenced that specific mediators in a single CAR file. This may cause deployment time exceptions during the due to dependencies conflicts. If you have a custom java code, always deploy that before deploying the ESB artifacts, which refers this custom code

General Best Practices

When you have environment specific artifacts such as endpoints, data source configurations, always better to use governance registry to store them.

When sharing the implemented projects among team members of the development team, use WSO2 Project Import Wizard since WSO2 Project Import Wizard is knowledgeable on handling the nested project structures supported by Developer Studio.

When exporting Developer Studio Projects you can either export individual projects and generate individual deployable artifacts or you can package them into a CAR file and export thar CAR file from IDE itself.

If you need to integrate your solution with a Continuous Integration server(Ex:Bamboo, Jenkins) and automate the build and deploy process, you need to use Maven support provided by Developer Studio for that. You should not include the Carbon Server extensions in CAR files and deploy to Carbon Servers,

Developer Studio - Deployment Best Practices

When you want to deploy artifacts to a Carbon Server, first you need to package all the artifacts to a CAR file. There are 2 possible methods to deploy to a Carbon Server.

Via Developer Studio IDE

Via Maven-Car-Deploy-Plugin

You can use any of the above 2 methods to deploy the CAR files to a Carbon Server. When you have modified your projects in Developer Studio, you can simply use Maven or IDE to publish latest version of the CAR file to the Carbon Server.

Design and Development Best Practices

1. Use mediators like XSLT and XQuery in complex transformation. Payload Factory and Enrich mediator perform better in simple transformation scenarios.

2. When a message is sent out from the inSequence , the reply is received by the outSequence by default. To receive the reply to a named sequence, use send mediator in the inSequence .

3. Do not use two consecutive send mediators without cloning the message using Clone mediator.

4. Always use REST APIs to expose RESTful interfaces over invoking Proxy Services.

5. Aggregation mediator works in tandem with Iterate or Clone.

6. Use templates to reuse the ESB configuration whenever possible.

7. Limit the use of in-line endpoints and use references to existing endpoints from sequence or proxy services.

Maintenance is easier when you isolate endpoint definitions from sequences and proxy services.

Security Guidelines and Best Practices

This section provides some security guidelines and best practices useful in service development.

Use DMZ-based deployments. Also see

http://wso2.com/blogs/architecture/2011/04/public-services-gateway-and-internal-services-gateway-patterns.

Change default admin username and password.
Change the default key stores.
Secure plain text passwords and configure user password policies.
In REST services deployments, secure your services with OAuth (Bearer and HMAC) and Basic authentication with HTTPS. This is recommended because:
It adds authentication, authorization and validation in to the services.
It enables communication to be done in a confidential manner while achieving non-repudiation and integrity of the messages.
It helps you avoid attacks such as replay, DoS and to define rules for throttling the service calls.
It helps you to control access to services in a fine-grained manner using role-based, attribute-based and policy-based access control.

↧

Performance Tuning WSO2 ESB with a practical example

May 3, 2015, 1:25 am

≫ Next: Understanding Threads created in WSO2 ESB

≪ Previous: WSO2 ESB Development best practices

WSO2 ESB is arguably the highest performing open source ESB available in the industry. With the default settings, it will provide a really good performance OOTB. You can find the official performance tuning guide in the below link.

https://docs.wso2.com/display/ESB481/Performance+Tuning

Above document includes lot of parameters and how to change them for best performance. Even though it provides some recommended values for the performance tuning, that is not straightforward. The values you put in the configuration files will highly reliant on your use case. Idea of this blog post is to provide a practical example of tuning these parameters with a sample use case.

For the performance testing, I am using a simple proxy service which iterate through set of xml elements and send request to a sample server and aggregate the responses back to the client.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="SplitAggregateProxy"
startOnLoad="true">
<target>
<inSequence>
<iterate xmlns:m0="http://services.samples"
preservePayload="true"
attachPath="//m0:getQuote"
expression="//m0:getQuote/m0:request">
<target>
<sequence>
<send>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
</endpoint>
</send>
</sequence>
</target>
</iterate>
</inSequence>
<outSequence>
<aggregate>
<completeCondition>
<messageCount/>
</completeCondition>
<onComplete xmlns:m0="http://services.samples" expression="//m0:getQuoteResponse">
<send/>
</onComplete>
</aggregate>
</outSequence>
</target>
</proxy>

The reason for selecting this type of proxy is that it will allow us to test different thread groups within the WSO2 ESB. Request for this proxy is given below.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">
<soapenv:Header/>
<soapenv:Body>
<ser:getQuote>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>

<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>

<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
<ser:request>

<xsd:symbol>IBM</xsd:symbol>
</ser:request>
</ser:getQuote>
</soapenv:Body>
</soapenv:Envelope>

We have used the apache jmeter as the test client. There we have created a Thread group with following configurations.

Number of Threads (Users) - 50
Ramp up period (seconds) - 10
Loop count - 200
Total requests - 10000

With this information in hand, we have used WSO2 ESB 4.8.1 within a ubuntu linux machine with 8GB memory and 4 core cpu.

First, let's talk about the parameters which we have used to carry out this performance test.

(ESB_HOME/bin/wso2server.sh)
Memory (Xmx) - Maximum heap memory allocated for JVM. We kept both Xmx and Xms the same.

(ESB_HOME/repository/conf/synapse.properties)
synapse.threads.core - Core number of threads allocated for ThreadPoolExecutor used for creating threads for iterate mediator executions

synapse.threads.qlen - Task queue length used for ThreadPoolExecutor used for creating threads for iterate mediator executions

(ESB_HOME/repository/conf/passthru-http.properties)
worker_pool_size_core - Core number of threads allocated for ThreadPoolExecutor used for processing incoming requests to the ESB

io_buffer_size - Size of the memory buffer used for reading/writing data from NIO channels

Performance tests were carried out while monitoring the server using Java Mission Control tool and the server load is kept at a perfect value around 60-70 % CPU usage and Load average value around 3-4 (4 core). There were no GC related issues observed during this testing.

You can read the following article to get an understanding about the performance of a server and the methods for tuning servers.

http://www.infoq.com/articles/Tuning-Java-Servers

We have captured both Latency and the TPS value for monitoring the performance of the server

Performance variation with Memory allocation

Theoretically, the performance of the server should be increased with the allocated memory. By performance, we consider both Latency and the TPS value of the server. According to the below results, we can see that TPS is increased with the memory and the Latency has reduced (increased performance).

Performance variation with number of Server Worker/Client Worker Threads

WSO2 ESB uses the ThreadPoolExecutor to create threads when there is data to be processed from the client requests. worker_pool_size_core parameter controls the number of core threads for this executor pool. By increasing the thread pool, we would expect that we see a performance improvement. According to the below graphs, we can see that latency is reduced and the TPS is slightly improved with this parameter (performance increased with number of threads)

Performance variation with Synapse Worker Threads count

When using iterate or clone mediator within the ESB, it will use a separate thread pool to create new threads when there is data to be iterated and processed. The size of this thread pool is configured with the parameter synapse.threads.core. By increasing the value here, we would expect a better performance when iterate mediator is used. According to the test results, we can see that performance is increased when the value is changed from 20 to 100. But after that, we can see some performance degradation since when the number of threads in the system increases, that will degrade the performance due to the high load in the thread scheduler in the operating system.

Performance variation with Synapse Worker Queue length

When using iterate or clone mediator within the ESB, it will use a separate thread pool to create new threads when there is data to be iterated and processed. We can configure the task queue length of this thread pool with the synapse.threads.qlen parameter. By giving a finite value to this task queue, it will make the system to create new threads when all the core number of threads are created and task queue is full. This is the only time that max value of thread pool is used. If the queue length is infinite (-1), max value is never used and there will only be core number of threads at any given time. According to the results, we can see that, when there is a finite value for queue length, we can see increased performance. One thing to note here is that, when we have a limited value for this queue length, there can be situations that requests will be rejected when the task queue is full and all the threads are occupied. Therefore, you need to make a decision according to your actual load. One additional thing with this parameter is that if there is some thread blocking happens when the queue length is infinite, server can go in to OOM situation.

Performance variation with IO Buffer size

IO buffer size parameter(io_buffer_size) decides the value of the memory buffer allocated for reading data in to the memory from the underlying socket/file channels. According to the average value of the payloads, we can configure this parameter. According to the results we observed from the testing, we cannot clearly come to a conclusion for this scenario since the request/response size is <4k during this testing.

According to the above results, we can see that when tuning the WSO2 ESB is not straightforward and you need to have a proper idea about your use cases.

↧

Understanding Threads created in WSO2 ESB

May 8, 2015, 8:33 pm

≫ Next: What happens to a message going through WSO2 ESB synapse mediation engine ( Looking up the source code)

≪ Previous: Performance Tuning WSO2 ESB with a practical example

WSO2 ESB is an asynchronous high performing messaging engine which uses Java NIO technology for its internal implementations. You can find more information about the implementation details about the WSO2 ESB’s high performing http transport known as Pass-Through Transport (PTT) from the links given below.

[1] http://soatutorials.blogspot.com/2015/05/understanding-wso2-esb-pass-through.html

[2] http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-i/

From this tutorial, I am going to discuss about various threads created when you start the ESB and start processing requests with that. This would help you to troubleshoot critical ESB server issues with the usage of a thread dump. You can monitor the threads created by using a monitoring tool like Jconsole or java mission control (java 1.7.40 upwards). Given below is a list of important threads and their stack traces from an active ESB server.

PassThroughHTTPSSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to https endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to and https endpoint, it will eventually executes this thread. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSSender

State: RUNNABLE

Total blocked: 1 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@2bef6f1

- locked java.util.Collections$UnmodifiableSet@578b8a47

- locked sun.nio.ch.KQueueSelectorImpl@1a8f39bb

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)

org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)

java.lang.Thread.run(Thread.java:744)

HTTPS-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to https endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).

Name: HTTPS-Sender I/O dispatcher-1

State: RUNNABLE

Total blocked: 1 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@25484cc9

- locked java.util.Collections$UnmodifiableSet@3f6342dd

- locked sun.nio.ch.KQueueSelectorImpl@66ac838c

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)

org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)

java.lang.Thread.run(Thread.java:744)

PassThroughHTTPSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to http endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to http endpoint, it will eventually executes this thread and create a new socket channel. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPSender

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@c15e5c9

- locked java.util.Collections$UnmodifiableSet@7e49d31e

- locked sun.nio.ch.KQueueSelectorImpl@9052336

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)

org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)

java.lang.Thread.run(Thread.java:744)

HTTP-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to http endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Sender I/O dispatcher-1

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@327d74e9

- locked java.util.Collections$UnmodifiableSet@460208f5

- locked sun.nio.ch.KQueueSelectorImpl@33f55d67

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)

org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)

java.lang.Thread.run(Thread.java:744)

PassThroughHTTPSListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from https requests from clients. When there is a https request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSListener

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@50cff7e

- locked java.util.Collections$UnmodifiableSet@271d259c

- locked sun.nio.ch.KQueueSelectorImpl@2282db4d

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)

org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)

java.lang.Thread.run(Thread.java:744)

HTTPS-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from https requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).

Name: HTTPS-Listener I/O dispatcher-1

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@48c97759

- locked java.util.Collections$UnmodifiableSet@8ac860c

- locked sun.nio.ch.KQueueSelectorImpl@ff4fe7c

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)

org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)

java.lang.Thread.run(Thread.java:744)

PassThroughHTTPListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from http requests from clients. When there is a http request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPListener

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@1f6438be

- locked java.util.Collections$UnmodifiableSet@152987f9

- locked sun.nio.ch.KQueueSelectorImpl@11d60796

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)

org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)

java.lang.Thread.run(Thread.java:744)

HTTP-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from http requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Listener I/O dispatcher-1

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@34095061

- locked java.util.Collections$UnmodifiableSet@60e8e6e5

- locked sun.nio.ch.KQueueSelectorImpl@5fbe8e73

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)

org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)

org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)

java.lang.Thread.run(Thread.java:744)

PassThroughMessageProcessor-1(400)

This is the worker thread which is responsible for processing requests coming in to the ESB when the request message is received. Passthrough transport will start these worker threads once the HTTP headers are received from the clients. Then these threads will run independently from PTT threads which are described earlier. There will be n number of threads created where n is the worker_pool_size_core parameter defined in the passthrough-http.properties file.

Name: PassThroughMessageProcessor-1

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7050afc9

Total blocked: 0 Total waited: 1

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

java.lang.Thread.run(Thread.java:744)

SynapseWorker-1(20)

This is the worker thread which is responsible for handling iterate/clone mediator executions. There will be a separate thread pool to handle such executions. Number of threads in this thread pool can be configured in synapse.properties file with the parameter synapse.threads.core.

Name: SynapseWorker-1

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d3b7d9

Total blocked: 1 Total waited: 2

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

java.lang.Thread.run(Thread.java:744)

Following are threads related to embedded tomcat instance run within the ESB. It will be used to serve requests coming in to ESB management console, admin services. You can configure the thread pool and other parameters from ESB_HOME/repository/conf/tomcat/catalina-server.xml file.

NioBlockingSelector.BlockPoller-1(2)

Name: NioBlockingSelector.BlockPoller-1

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@5d630f69

- locked java.util.Collections$UnmodifiableSet@59cdfa64

- locked sun.nio.ch.KQueueSelectorImpl@510b6d29

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:327)

http-nio-9763-ClientPoller-0(2)

Name: http-nio-9763-ClientPoller-0

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@5f185036

- locked java.util.Collections$UnmodifiableSet@1b5f14d

- locked sun.nio.ch.KQueueSelectorImpl@5b8b369f

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)

java.lang.Thread.run(Thread.java:744)

http-nio-9763-Acceptor-0(2)

Name: http-nio-9763-Acceptor-0

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)

- locked java.lang.Object@525f8df5

org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)

java.lang.Thread.run(Thread.java:744)

http-nio-9443-ClientPoller-0(2)

Name: http-nio-9443-ClientPoller-0

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)

sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)

sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked sun.nio.ch.Util$2@3d8e1f2f

- locked java.util.Collections$UnmodifiableSet@2f3ecb19

- locked sun.nio.ch.KQueueSelectorImpl@113dc8a9

sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)

java.lang.Thread.run(Thread.java:744)

http-nio-9443-Acceptor-0(2)

Name: http-nio-9443-Acceptor-0

State: RUNNABLE

Total blocked: 0 Total waited: 0

Stack trace:

sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)

- locked java.lang.Object@38fcf802

org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)

java.lang.Thread.run(Thread.java:744)

http-nio-9443-exec-1(50)

Name: http-nio-9443-exec-1

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a4e4fb2

Total blocked: 1 Total waited: 6

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)

org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

java.lang.Thread.run(Thread.java:744)

↧

What happens to a message going through WSO2 ESB synapse mediation engine ( Looking up the source code)

May 9, 2015, 9:21 pm

≫ Next: WSO2 ESB tuning performance with threads

≪ Previous: Understanding Threads created in WSO2 ESB

In the previous blog post [1], I have discussed about the PassThrough Transport and how it works. If you follow this [2] article, you can learn about what is happening inside PTT when a message is received. The below diagram depicts the relationship of PTT, Axis2 and Synaps Mediation engine within WSO2 ESB.

[1] http://soatutorials.blogspot.com/2015/05/understanding-wso2-esb-pass-through.html

[2] http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-iii/

As depicted in the above diagram, When a message comes to the ESB, it will be received by the reactor thread of Pass Through Listener and the message will be read in to an internal buffer. Then it will be processed through a separate worker thread and will flows through the Axis2 In message flow. Then Axis2 engine will call the respective message receiver (SynapseMessageReceiver, ProxyMessageReceiver or SynapseCallbackReceiver) class. This would be the main entry point to the synapse mediation engine. Then it will go through the synapse mediation flow and handed over to Axis2FlexibleMEPClient and and it will hand over the message to Axis2 engine for the out flow. Then the message will be send to the backend through PassThroughSender thread.

This is a high level description of what happens inside WSO2 ESB. The following section will describe what happens within the synapse mediation engine when it receives by the MessageReceiver interface class.

I am taking the following sample ESB configuration for this debugging session. It contains all the basic elements of a proxy service definition which are

InSequence

OutSequence

FaultSequence

Endpoint

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

name="DebugProxy"

transports="https,http"

statistics="disable"

trace="disable"

startOnLoad="true">

<inSequence>

</log>

</inSequence>

<outSequence>

</log>

<send/>

</outSequence>

<faultSequence>

</log>

</faultSequence>

<endpoint>

</endpoint>

</target>

</proxy>

Here I have used the log mediator with level “full” such that message will be converted in to canonical format (building the message) within the ESB.

The below description provides information on the classes and methods executed when a message received from a client to the ESB. This will be really helpful when you are debugging a particular issue related to ESB and you can start debugging at a desired level without starting from the beginning.

Request Path

AxisEngine.receive()

ProxyServiceMessageReceiver.receive()

————————————

Get the Fault Sequence and push in to fault handler stack

————————————

Execute the inSequence

SequenceMediator.mediate()

AbstractListMediator.mediate()

RelayUtils.buildMessage()

DeferredMessageBuilder.getDocument()

SOAPBuilder.processDocument()

LogMediator.mediate()

————————————

If the inSequence execution returns true, execute the endpoint

AddressEndpoint.send()

AbstractEndpoint.send()

If message needs to build at this level, call the RelayUtils.build() method

Axis2SynapseEnvironment.send()

AxisSender.sendOn()

Axis2FlexibleMEPClient.send()

Create OperationClient and register the callback to process the response (if any)

OperationClient.execute()

OutInAxisOperation.executeImpl()

Add the registered callback to the callback store with the messageID

AxisEngine.send()

PassThroughHttpSender.invoke()

————————————

When a response is received from the back end, it will be captured by the TargetHandler class and spawn a new ClientWorker thread to process the response and hand it over to the axis engine to process. AxisEngine will call the respective message receiver (in this case CallBackReceiver).

Response Path

AxisEngine.receive()

SynapseCallbackReceiver.receive()

SynapseCallbackReceiver.handleMessage()

If there is an error, pop the fault handler and set the fault parameters and execute the fault handler

If there are no errors or special cases, hand over the message to synapse environment

Axis2SynapseEnvironment.injectMessage()

Check if this response is for a proxy service or not and proceed

If this is for a proxy service, then check the fault handler and add that to fault stack

————————————

Execute outSequence

SequenceMediator.mediate()

AbstractListMediator.mediate()

RelayUtils.buildMessage()

DeferredMessageBuilder.getDocument()

SOAPBuilder.processDocument()

LogMediator.mediate()

SendMediator.mediate()

Axis2SynapseEnvironment.send()

Axis2Sender.sendBack()

AxisEngine.send()

PassThroughHttpSender.invoke()

After going through this article, it would be better if you can do some debugging on the synapse code.

↧

WSO2 ESB tuning performance with threads

May 21, 2015, 8:27 am

≫ Next: Understanding Java Garbage Collection for beginners

≪ Previous: What happens to a message going through WSO2 ESB synapse mediation engine ( Looking up the source code)

I have written several blog posts explaining the internal behavior of the ESB and the threads created inside ESB. With this post, I am talking about the effect of threads in the WSO2 ESB and how to tune up threads for optimal performance. You can refer [1] and [2] to understand the threads created within the ESB.

[1] http://soatutorials.blogspot.com/2015/05/understanding-threads-created-in-wso2.html

[2] http://wso2.com/library/articles/2012/03/importance-performance-wso2-esb-handles-nonobvious/

Within this blog post, I am discussing about the "worker threads" which are used for processing the data within the WSO2 ESB. There are 2 types of worker threads created when you start sending the requests to the server

1) Server Worker/Client Worker Threads
2) Mediator Worker (Synapse-Worker) Threads

Server Worker/Client Worker Threads

These set of threads will be used to process all the requests/responses coming to the ESB server. ServerWorker Threads will be used to process the request path and Client Worker threads will be used to process the responses.

Mediator Worker (Synapse-Worker) Threads

These threads will only be started if you have iterate/clone mediators in your ESB mediation flow. These threads will be used for processing iterate/clone operations in separate threads for parallel processing of a single request.

WSO2 ESB uses the java ThreadPoolExecutor implementation for spawning new threads for processing requests. Both the above mentioned thread categories will be using this implementation underneath.

The java.util.concurrent.ThreadPoolExecutor is an implementation of the ExecutorService interface. The ThreadPoolExecutor executes the given task (Callable or Runnable) using one of its internally pooled threads.

The thread pool contained inside the ThreadPoolExecutor can contain a varying amount of threads. The number of threads in the pool is determined by these variables:

corePoolSize
maximumPoolSize

If less than corePoolSize threads are created in the the thread pool when a task is delegated to the thread pool, then a new thread is created, even if idle threads exist in the pool.

If the internal queue of tasks is full, and corePoolSize threads or more are running, but less than maximumPoolSize threads are running, then a new thread is created to execute the task.

These parameter of the thread pools can be configured in the following configuration files in the WSO2 ESB

ServerWorker/ClientWorker Thread pool (ESB_HOME/repository/conf/passthru-http.properties)

worker_pool_size_core=400

worker_pool_size_max=500

#worker_thread_keepalive_sec=60

#worker_pool_queue_length=-1

The default values given in the standalone ESB pack would be enough for most of the scenarios. But you need to do some performance testing with a similar load and tune these values accordingly. In the above configuration, there are 2 commented out parameters.

worker_thread_keepalive_sec - If the pool currently has more than corePoolSize threads, excess threads will be terminated if they have been idle for more than the keepAliveTime. This provides a means of reducing resource consumption when the pool is not being actively used. If the pool becomes more active later, new threads will be constructed.

worker_pool_queue_length - This is the task queue length to which new tasks will be delegated by the server when there are new data to be processed. The length of this queue is -1 (infinite) by default. This is one of the most important parameter when you are tuning the server for capacity. When you have infinite length queue, it will never reject any request. But the drawback with this value is that, if there are less number of processing threads and you have a peak load, the server can easily go into OOM status since the task queue will hold all the requests coming in to the server. You need to decide on a considerable value for this queue length rather than keeping this value as -1. If you have a limited value for this queue length, it will reject some requests in a high load scenario. But the server will not crash (OOM). This would be better rather than loosing all the requests. Another disadvantage of having -1 as the queue length would be that server will never create the max number of threads but only create core number of threads in any kind of load.

MediatorWorker (SynapseWorker) Threads (ESB_HOME/repository/conf/synapse.properties)

synapse.threads.core = 20

synapse.threads.max = 100

#synapse.threads.keepalive = 5

#synapse.threads.qlen = 10

The same theory which I have described above can be applied when tuning this thread pool. Apart from that, It is always better to have a matching core value with the ServerWorker threads if you have used iterate/clone mediators heavily in your mediation flow. Considerable value for these parameters would like below.

synapse.threads.core = 100

synapse.threads.max = 200

I hope this would help you when tuning WSO2 ESB server for your production deployments.

↧

Understanding Java Garbage Collection for beginners

June 13, 2015, 11:55 pm

≫ Next: Garbage Collection and Application Performance

≪ Previous: WSO2 ESB tuning performance with threads

Java is one of the heavily used languages in enterprise application development. One of the key features of the Java language is it's capability to clear out memory automatically. This gives application developer more freedom to think about his business logic rather than worrying about memory management of the application. This may be the utmost reason for the selection of java language for complex business application development.

Java uses a technology called Automatic Garbage Collection (GC) for clearing out any unused memory from your application. During this blog post, I will be discussing about Java memory model and how GC works within the Java virtual machine (JVM). In fact every java application runs on top of its own JVM.

Java Memory model

Each thread running in the Java virtual machine has its own thread stack. The thread stack contains information about what methods the thread has called to reach the current point of execution. I will refer to this as the "call stack". As the thread executes its code, the call stack changes. The thread stack also contains all local variables for each method being executed (all methods on the call stack). A thread can only access it's own thread stack. Local variables created by a thread are invisible to all other threads than the thread who created it. Even if two threads are executing the exact same code, the two threads will still create the local variables of that code in each their own thread stack. Thus, each thread has its own version of each local variable.

All local variables of primitive types ( boolean, byte, short, char, int, long, float, double) are fully stored on the thread stack and are thus not visible to other threads. One thread may pass a copy of a primitive variable to another thread, but it cannot share the primitive local variable itself. The heap contains all objects created in your Java application, regardless of what thread created the object. This includes the object versions of the primitive types (e.g. Byte, Integer, Long etc.). It does not matter if an object was created and assigned to a local variable, or created as a member variable of another object, the object is still stored on the heap.

A local variable may be of a primitive type, in which case it is totally kept on the thread stack.

A local variable may also be a reference to an object. In that case the reference (the local variable) is stored on the thread stack, but the object itself if stored on the heap.

An object may contain methods and these methods may contain local variables. These local variables are also stored on the thread stack, even if the object the method belongs to is stored on the heap.

An object's member variables are stored on the heap along with the object itself. That is true both when the member variable is of a primitive type, and if it is a reference to an object.

Static class variables are also stored on the heap along with the class definition.

Objects on the heap can be accessed by all threads that have a reference to the object. When a thread has access to an object, it can also get access to that object's member variables. If two threads call a method on the same object at the same time, they will both have access to the object's member variables, but each thread will have its own copy of the local variables.

Java Heap Memory

As discussed in the previous section, Java heap memory is responsible for storing all the objects created during the runtime of a program and all the member variables and the static variables with its class definitions. This is the area of memory which needs to be carefully controlled. Below diagram depicts the memory model of the JVM heap.

Young Generation

Most of the newly created objects are located in the Eden memory space.

When Eden space is filled with objects, Minor GC is performed and all the survivor objects are moved to one of the survivor spaces.

Minor GC also checks the survivor objects and move them to the other survivor space. So at a time, one of the survivor space is always empty.

Objects that are survived after many cycles of GC, are moved to the Old generation memory space. Usually it’s done by setting a threshold for the age of the young generation objects before they become eligible to promote to Old generation.

Old Generation

Old Generation memory contains the objects that are long lived and survived after many rounds of Minor GC. Usually garbage collection is performed in Old Generation memory when it’s full. Old Generation Garbage Collection is called Major GC and usually takes longer time.

Stop the World Event

All the Garbage Collections are “Stop the World” events because all application threads are stopped until the operation completes.

Since Young generation keeps short-lived objects, Minor GC is very fast and the application doesn’t get affected by this.

However Major GC takes longer time because it checks all the live objects. Major GC should be minimized because it will make your application unresponsive for the garbage collection duration. So if you have a responsive application and there are a lot of Major Garbage Collection happening, you will notice timeout errors.

The duration taken by garbage collector depends on the strategy used for garbage collection. That’s why it’s necessary to monitor and tune the garbage collector to avoid timeouts in the highly responsive applications.

Permanent Generation

Permanent Generation or “Perm Gen” contains the application metadata required by the JVM to describe the classes and methods used in the application. Note that Perm Gen is not part of Java Heap memory.

Perm Gen is populated by JVM at runtime based on the classes used by the application. Perm Gen also contains Java SE library classes and methods. Perm Gen objects are garbage collected in a full garbage collection.

Garbage Collection

As already mentioned in the beginning of this post, Garbage Collection is one of the prime features of the java programming language. Many people think garbage collection collects and discards dead objects. In reality, Java garbage collection is doing the opposite! Live objects are tracked and everything else designated garbage. As you'll see, this fundamental misunderstanding can lead to many performance problems.

Let's start with the heap, which is the area of memory used for dynamic allocation. In most configurations the operating system allocates the heap in advance to be managed by the JVM while the program is running. This has a couple of important ramifications:

Object creation is faster because global synchronization with the operating system is not needed for every single object. An allocation simply claims some portion of a memory array and moves the offset pointer forward. The next allocation starts at this offset and claims the next portion of the array.
When an object is no longer used, the garbage collector reclaims the underlying memory and reuses it for future object allocation. This means there is no explicit deletion and no memory is given back to the operating system.

Garbage Collection Roots

Every object tree must have one or more root objects. As long as the application can reach those roots, the whole tree is reachable. But when are those root objects considered reachable? Special objects called garbage-collection roots (GC roots; see below figure) are always reachable and so is any object that has a garbage-collection root at its own root.

There are four kinds of GC roots in Java:

Local variables are kept alive by the stack of a thread. This is not a real object virtual reference and thus is not visible. For all intents and purposes, local variables are GC roots.

Active Java threads are always considered live objects and are therefore GC roots. This is especially important for thread local variables.

Static variables are referenced by their classes. This fact makes them de facto GC roots. Classes themselves can be garbage-collected, which would remove all referenced static variables. This is of special importance when we use application servers, OSGi containers or class loaders in general.

JNI References are Java objects that the native code has created as part of a JNI call. Objects thus created are treated specially because the JVM does not know if it is being referenced by the native code or not. Such objects represent a very special form of GC root, which we will examine in more detail in the Problem Patterns section below.

Marking and Sweeping Away Garbage

To determine which objects are no longer in use, the JVM intermittently runs what is very aptly called a mark-and-sweep algorithm . As you might intuit, it's a straightforward, two-step process:

The algorithm traverses all object references, starting with the GC roots, and marks every object found as alive.
All of the heap memory that is not occupied by marked objects is reclaimed. It is simply marked as free, essentially swept free of unused objects.

Garbage collection is intended to remove the cause for classic memory leaks: unreachable-but-not-deleted objects in memory. However, this works only for memory leaks in the original sense. It's possible to have unused objects that are still reachable by an application because the developer simply forgot to dereference them. Such objects cannot be garbage-collected. Even worse, such a logical memory leak cannot be detected by any software. Even the best analysis software can only highlight suspicious objects.

Garbage Collection Algorithms and Performance

As I have already mentioned previously, GC event is a "Stop the World" operation where application stop its execution during this time period. Therefore, it is very important to choose a proper GC algorithm for your performance critical enterprise application. There are 5 GC algorithms available with the Oracle JVM.

Serial GC (-XX:+UseSerialGC): Serial GC uses the simple mark-sweep-compact approach for young and old generations garbage collection i.e Minor and Major GC. Serial GC is useful in client-machines such as our simple stand alone applications and machines with smaller CPU. It is good for small applications with low memory footprint.

Parallel GC (-XX:+UseParallelGC): Parallel GC is same as Serial GC except that is spawns N threads for young generation garbage collection where N is the number of CPU cores in the system. We can control the number of threads using -XX:ParallelGCThreads=n JVM option. Parallel Garbage Collector is also called throughput collector because it uses multiple CPUs to speed up the GC performance. Parallel GC uses single thread for Old Generation garbage collection.

Parallel Old GC (-XX:+UseParallelOldGC): This is same as Parallel GC except that it uses multiple threads for both Young Generation and Old Generation garbage collection.

Concurrent Mark Sweep (CMS) Collector (-XX:+UseConcMarkSweepGC): CMS Collector is also referred as concurrent low pause collector. It does the garbage collection for Old generation. CMS collector tries to minimize the pauses due to garbage collection by doing most of the garbage collection work concurrently with the application threads. CMS collector on young generation uses the same algorithm as that of the parallel collector. This garbage collector is suitable for responsive applications where we can’t afford longer pause times. We can limit the number of threads in CMS collector using -XX:ParallelCMSThreads=n JVM option.

G1 Garbage Collector (-XX:+UseG1GC): The Garbage First or G1 garbage collector is available from Java 7 and it’s long term goal is to replace the CMS collector. The G1 collector is a parallel, concurrent, and incrementally compacting low-pause garbage collector. Garbage First Collector doesn’t work like other collectors and there is no concept of Young and Old generation space. It divides the heap space into multiple equal-sized heap regions. When a garbage collection is invoked, it first collects the region with lesser live data, hence “Garbage First”. You can find more details about it at Garbage-First Collector Oracle Documentation.

That is enough for this lengthy blog post. I am planning to write several blog posts on GC tuning in the future.

Happy GC !!!

References

http://www.journaldev.com/2856/java-jvm-memory-model-and-garbage-collection-monitoring-tuning

http://www.dynatrace.com/en/javabook/how-garbage-collection-works.html

http://tutorials.jenkov.com/java-concurrency/java-memory-model.html

↧

Garbage Collection and Application Performance

June 14, 2015, 3:22 am

≫ Next: Monitoring Garbage Collection of WSO2 ESB

≪ Previous: Understanding Java Garbage Collection for beginners

Automatic Garbage Collection is one of the finest features of the Java programming language. You can find more information about Garbage Collection concepts from the below link.

http://soatutorials.blogspot.com/2015/06/understanding-java-garbage-collection.html

Even though GC is a cool feature in the JVM, it comes at a cost. Your application will stop working (Stop the World) when GC happens in the JVM level. Which means that, GC events will affect the performance of your java application. Due to this, you should have a proper understanding about the impact of GC for your application.

There are two general ways to reduce garbage-collection pause time and the impact it has on application performance:

The garbage collection itself can leverage the existence of multiple CPUs and be executed in parallel. Although the application threads remain fully suspended during this time, the garbage collection can be done in a fraction of the time, effectively reducing the suspension time.

The second approach is leave the application running, and execute garbage collection concurrently with the application execution.

These two logical solutions have led to the development of serial, parallel, and concurrent garbage-collection strategies , which represent the foundation of all existing Java garbage-collection implementations.

In the above diagram, serial collector suspends the application threads and executes the mark-and-sweep algorithm in a single thread. It is the simplest and oldest form of garbage collection in Java and is still the default in the Oracle HotSpot JVM.

The parallel collector uses multiple threads to do its work. It can therefore decrease the GC pause time by leveraging multiple CPUs. It is often the best choice for throughput applications.

The concurrent collector does the majority of its work concurrent with the application execution. It has to suspend the application for only very short amounts of time. This has a big benefit for response-time for sensitive applications, but is not without drawbacks.

Concurrent Mark and Sweep algorithm

Concurrent garbage-collection strategies complicate the relatively simple mark-and-sweep algorithm a bit. The mark phase is usually sub-divided into some variant of the following:

In the initial marking, the GC root objects are marked as alive. During this phase, all threads of the application are suspended.

During concurrent marking, the marked root objects are traversed and all reachable objects are marked. This phase is fully concurrent with application execution, so all application threads are active and can even allocate new objects. For this reason there might be another phase that marks objects that have been allocated during the concurrent marking. This is sometimes referred to as pre-cleaning and is still done concurrent to the application execution.

In the final marking, all threads are suspended and all remaining newly allocated objects are marked as alive. This is indicated in Figure 2.6 by the re-mark label.

The concurrent mark works mostly, but not completely, without pausing the application. The tradeoff is a more complex algorithm and an additional phase that is not necessary in a normal stop-the-world GC: the final marking.

The Oracle JRockit JVM improves this algorithm with the help of a keep area, which, if you're interested, is described in detail in the JRockit documentation . New objects are kept separately and not considered garbage during the first GC. This eliminates the need for a final marking or re-mark.

In the sweep phase of the CMS, all memory areas not occupied by marked objects are found and added to the free list. In other words, the objects are swept by the GC. This phase can run at least partially concurrent to the application. For instance, JRockit divides the heap into two areas of equal size and sweeps one then the other. During this phase, no threads are stopped, but allocations take place only in the area that is not actively being swept.

There is only one way to make garbage collection faster: ensure that as few objects as possible are reachable during the garbage collection. The fewer objects that are alive, the less there is to be marked. This is the rationale behind the generational heap.

Young Generation vs Old Generation

In a typical application most objects are very short-lived. On the other hand, some objects last for a very long time and even until the application is terminated. When using generational garbage collection, the heap area is divided into two areas—a young generation and an old generation—that are garbage-collected via separate strategies.

Objects are ussually created in the young area. Once an object has survived a couple of GC cycles it is tenured to the old generation. After the application has completed its initial startup phase (most applications allocate caches, pools, and other permanent objects during startup), most allocated objects will not survive their first or second GC cycle. The number of live objects that need to be considered in each cycle should be stable and relatively small.

Allocations in the old generation should be infrequent, and in an ideal world would not happen at all after the initial startup phase. If the old generation is not growing and therefore not running out of space, it requires no garbage-collection at all. There will be unreachable objects in the old generation, but as long as the memory is not needed, there is no reason to reclaim it.

To make this generational approach work, the young generation must be big enough to ensure that all temporary objects die there. Since the number of temporary objects in most applications depends on the current application load, the optimal young generation size is load-related. Therefore, sizing the young generation, known as generation-sizing, is the key to achieving peak load.

Unfortunately, it is often not possible to reach an optimal state where all objects die in the young generation, and so the old generation will often often require a concurrent garbage collector. Concurrent garbage collection together with a minimally growing old generation ensures that the unavoidable, stop-the-world events will at least be very short and predictable.

↧

Monitoring Garbage Collection of WSO2 ESB

June 20, 2015, 8:03 am

≫ Next: Extending WSO2 ESB with a Custom Transport Implementation - Part I

≪ Previous: Garbage Collection and Application Performance

Garbage Collection has been one of the most important features of Java programming language which made it the automatic choice for developing enterprise applications. WSO2 ESB has been written entirely in Java. Garbage Collection is pretty much related to the performance of a Java program. WSO2 ESB is a java program and it needs to provide the maximum performance to the users who use that for their enterprise integrations. From this blog post, I will be discussing about different tools which we can use to monitor the GC performance of WSO2 ESB.

1) Monitoring GC activity using jstat command

We can use the jstat command line tool which comes with the JDK to monitor the GC activity on a java program. Let's start the WSO2 ESB server by executing the wso2server.sh file located under ESB_HOME/bin directory.

sh wso2server.sh start

Then we need to find the process ID of this java process using the following command

ps -ef | grep wso2esb | grep java

501 13352 13345 0 7:25PM ttys000 1:18.41

We can execute the jstat command with the process ID

jstat -gc 13352 1000

In the above command, last argument is the time gap in which it prints the statistics. For the above command, it will print statistics every 1 second.

S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT

49664.0 50688.0 0.0 0.0 246272.0 135276.4 175104.0 91437.3 114688.0 61223.4 24 0.954 1 0.864 1.818

49664.0 50688.0 0.0 0.0 246272.0 135276.7 175104.0 91437.3 114688.0 61223.4 24 0.954 1 0.864 1.818

49664.0 50688.0 0.0 0.0 246272.0 135281.1 175104.0 91437.3 114688.0 61223.4 24 0.954 1 0.864 1.818

49664.0 50688.0 0.0 0.0 246272.0 135281.2 175104.0 91437.3 114688.0 61223.4 24 0.954 1 0.864 1.818

49664.0 50688.0 0.0 0.0 246272.0 135285.7 175104.0 91437.3 114688.0 61223.4 24 0.954 1 0.864 1.818

The above output provides a detailed information on the GC activity going on with the java program.

S0C and S1C: This column shows the current size of the Survivor0 and Survivor1 areas in KB.
S0U and S1U: This column shows the current usage of the Survivor0 and Survivor1 areas in KB. Notice that one of the survivor areas are empty all the time.
EC and EU: These columns show the current size and usage of Eden space in KB. Note that EU size is increasing and as soon as it crosses the EC, Minor GC is called and EU size is decreased.
OC and OU: These columns show the current size and current usage of Old generation in KB.
PC and PU: These columns show the current size and current usage of Perm Gen in KB.
YGC and YGCT: YGC column displays the number of GC event occurred in young generation. YGCT column displays the accumulated time for GC operations for Young generation. Notice that both of them are increased in the same row where EU value is dropped because of minor GC.
FGC and FGCT: FGC column displays the number of Full GC event occurred. FGCT column displays the accumulated time for Full GC operations. Notice that Full GC time is too high when compared to young generation GC timings.
GCT: This column displays the total accumulated time for GC operations. Notice that it’s sum of YGCT and FGCT column values.

2) Monitoring GC activity using JVisualVM an VisualGC

If you need to monitor the GC activity in a graphical manner, you can use the jvisualvm tool which comes with the JDK by installing the Visual GC plugin.

Just run jvisualvm command in the terminal to launch the Java VisualVM application.

Once launched, you need to install Visual GC plugin from Tools -> Plugins->Available Plugins (Tab) option

After installing Visual GC, just open the application(by double clicking) from the left side column and head over to Visual GC section

As depicted in the above diagram, you can visually monitor the GC activities of the WSO2 ESB using the jvisualvm tool.

3) Monitoring GC activity using GC log file

In most of the production use cases, we don't need to interact with the running process through different programs. Instead, we would like to have the GC logging as an internal part of the program itself. We can enable GC logging for the JVM such that it will log all the GC activities into a separate log file such that monitoring tools can interpret this file separately without interacting with the application directly.

You can enable GC logging in to an external log file by adding the following flags to startup script of the WSO2 ESB (wso2server.sh)

-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps \

-Xloggc:/Users/chanaka-mac/Documents/wso2/wso2esb-4.8.1/repository/logs/gc.log \

When you start the server with these flags included in the wso2server.sh file, you can observe that gc.log file is populating with the relevant GC activities as depicted below.

Chanakas-MacBook-Air:logs chanaka-mac$ tail -f gc.log

2015-06-20T20:04:22.222-0530: 20.347: [GC [PSYoungGen: 253355K->31555K(285184K)] 354302K->132534K(460288K), 0.0179690 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]

2015-06-20T20:04:23.031-0530: 21.156: [GC [PSYoungGen: 262979K->33422K(288256K)] 363958K->134431K(463360K), 0.0183790 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]

2015-06-20T20:04:23.797-0530: 21.922: [GC [PSYoungGen: 264846K->35384K(292352K)] 365855K->136426K(467456K), 0.0192760 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]

2015-06-20T20:04:24.468-0530: 22.593: [GC [PSYoungGen: 271928K->33834K(292864K)] 372970K->134994K(467968K), 0.0195170 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]

2015-06-20T20:04:25.162-0530: 23.287: [GC [PSYoungGen: 270378K->29641K(288768K)] 371538K->130840K(463872K), 0.0186680 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]

2015-06-20T20:04:26.547-0530: 24.672: [GC [PSYoungGen: 267721K->2845K(290816K)] 368920K->104320K(465920K), 0.0069150 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]

2015-06-20T20:04:29.429-0530: 27.554: [GC [PSYoungGen: 240925K->9509K(294400K)] 342400K->111496K(469504K), 0.0123910 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]

2015-06-20T20:04:32.290-0530: 30.415: [GC [PSYoungGen: 248613K->28794K(268288K)] 350600K->134734K(443392K), 0.0373390 secs] [Times: user=0.13 sys=0.01, real=0.03 secs]

2015-06-20T20:04:37.493-0530: 35.618: [GC [PSYoungGen: 249742K->21673K(287744K)] 355682K->152515K(462848K), 0.0903050 secs] [Times: user=0.16 sys=0.02, real=0.09 secs]

2015-06-20T20:04:37.584-0530: 35.709: [Full GC [PSYoungGen: 21673K->0K(287744K)] [ParOldGen: 130841K->80598K(175104K)] 152515K->80598K(462848K) [PSPermGen: 57507K->57484K(115200K)], 0.8345630 secs] [Times: user=1.68 sys=0.14, real=0.83 secs]

From the above log, we can extract information related to GC activities going on within the WSO2 ESB server.

2015-06-20T20:04:22.222-0530: – Time when the GC event started.
20.347 – Time when the GC event started, relative to the JVM startup time. Measured in seconds.
GC – Flag to distinguish between Minor & Full GC. This time it is indicating that this was a Minor GC.
PSYoungGen – Collection type.
253355K->31555K – Usage of Young generation before and after collection.
(285184K) – Total size of the Young generation.
354302K->132534K – Total used heap before and after collection.
(460288K) – Total available heap.
0.0179690 secs – Duration of the GC event in seconds.
[Times: user=0.02 sys=0.00, real=0.02 secs] – Duration of the GC event, measured in different categories:

user – Total CPU time that was consumed by Garbage Collector threads during this collection
sys – Time spent in OS calls or waiting for system event
real – Clock time for which your application was stopped. As Serial Garbage Collector always uses just a single thread, real time is thus equal to the sum of user and system times.

I hope this blog post have provided you a comprehensive guide on monitoring GC activities with WSO2 ESB for tuning the performance. I will discuss about GC performance of different algorithms with WSO2 ESB in a future blog post.

Happy GC Monitoring !!!

↧

Extending WSO2 ESB with a Custom Transport Implementation - Part I

June 21, 2015, 12:21 am

≫ Next: Extending WSO2 ESB with a Custom Transport Implementation - Part II

≪ Previous: Monitoring Garbage Collection of WSO2 ESB

WSO2 ESB is considered as one of the best and highest performing open source integration solutions available in the market. One of the astonishing features of the WSO2 ESB is the extensibility of the solution to meet your custom requirements. This means a LOT, if you have dealt with proprietary solutions provided big players (you name it). With this blog post, I will be discussing about one of the not so frequently used but THE BEST extension point in WSO2 ESB which is implementing a custom transport.

Given that WSO2 ESB is an extensible solution, that does not mean that it is lacking OOTB features. In fact it provides the most complete feature set provided by any open source integration solution in the market. But as you know, it is not a silver bullet (In fact we can't make silver bullets). Therefore, you may encounter some scenarios where you need to write a custom transport implementation to connect with one of your systems.

I will be taking ISO8583 messaging standard to write this custom transport implementation. It is used heavily in the financial transactions domain for credit card transaction processing. One of the reasons to select this as my custom transport implementation is that there is an already written code for this transport by Manoj Fernando in this blog post. Since my focus of this blog post is to describe about writing a custom transport implementation, I think I am not re-writing what Manoj has written.

Enough talking. Let's do some real work. WSO2 ESB mediation engine can be depicted in the following diagram.

As depicted in the above diagram, requests/responses coming from clients/servers (Inbound) will be hitting the ESB through the transport layer. It will select the proper transport receiver implementation by looking at the request URI (eg: HTTP, TCP, JMS, etc.)
Transport will hand over this message to the appropriate message builder according to the content-type (if specified) specified in the message.
Then the message will be handed over to the Axis engine(In Flow) where it does the QOS related operations.
After that, message will be handed over to the mediation engine for executing the mediation logic configured with mediators.
Then again, message will be going through the Axis engine (Out Flow) for any QOS related operations.
Message formatter will be selected according to the content-type provided in the message.
Then the message will be passed back to the relevant transport sender implementation to send the message from ESB to client/server (Outbound)

Alright.. Alright .. Now we know what happens to a message coming towards the WSO2 ESB and what happens when message is going out of the same. I have highlighted 4 terms in the previous section. Those 4 terms are

Transport Receiver
Message Builder
Message Formatter
Transport Sender

These would be the classes we need to implement for our custom transport implementation. WSO2 ESB has provided the interfaces for these implementations such that you need to focus only on the business logic rather than knowing the internals of WSO2 ESB. We will be using following interfaces to write our custom implementation.

org.apache.axis2.transport.base.AbstractTransportListener
org.apache.axis2.builder.Builder
org.apache.axis2.transport.MessageFormatter
org.apache.axis2.transport.base.AbstractTransportSender

Now we have the ground covered for our custom transport implementation. Let's do some coding. I will be transferring this discussion to my next blog post since this is getting too long here.

http://soatutorials.blogspot.com/2015/06/extending-wso2-esb-with-custom_21.html

↧

Extending WSO2 ESB with a Custom Transport Implementation - Part II

June 21, 2015, 3:24 am

≫ Next: WSO2 ESB Error Handling Tutorial - Part I (Client side error handling)

≪ Previous: Extending WSO2 ESB with a Custom Transport Implementation - Part I

This blog post is a continuation to my previous blog post where I have described the concepts of WSO2 ESB transports mechanism. Since we have covered the basics, let's start writing some real code. I will be using the ISO8583 standard as my subject to this custom implementation. I will be grabbing some content from this blog post for my reference to ISO8583 java implementation (Business logic). Thanks Manoj Fernando for writing such an informative post.

http://manoj-fernando.blogspot.com/2013/08/iso8583-with-wso2-esb.html

Idea of the custom transport implementation is to provide a mechanism to write your business logic which can plug in to the WSO2 ESB runtime. I am not going to tell more about ISO8583 or it's internal implementations. I will be using already implemented java library jPos for this purpose. It has the functionality to cover the basic use cases of ISO8583 implementations.

Sample use case

Let’s take the scenario of a certain financial application needing to make a credit transaction by sending an XML message that needs to be converted to an ISO8583 byte stream before passed on to the wire through a TCP channel.

Implementation

First, we need to define our ISO8583 field definition. This might be a bit confusing to some. If we are dealing with a specification, why do we need a field definition? This is because that ISO8583 specification is not hard-binding any data elements and/or field ordering. It is entirely up to the application designer to define which field types/IDs need to be placed for their specific transactional requirements.

At a glance, the field definition file looks like the following.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE isopackager SYSTEM "genericpackager.dtd">
<isopackager>
<isofield
id="0"
length="4"
name="Message Type Indicator"
class="org.jpos.iso.IFA_NUMERIC"/>
<isofield
id="1"
length="16"
name="Bitmap"
class="org.jpos.iso.IFA_BITMAP"/>
<isofield
id="2"
length="19"
name="Primary Account number"
class="org.jpos.iso.IFA_LLNUM"/>
<isofield
id="3"
length="6"
name="Processing Code"
class="org.jpos.iso.IFA_NUMERIC"/>
…
</isopackager>

Please refer to [1] & [2] for a complete reference of ISO8583. As per now, let me just say that each field should have an ID, a length and type specified in its definition. I have only listed a snippet of the XML config here, and you may find the full definition jposdef.xml inside the codebase.
I have created a simple maven project to implement this transport. Make sure that you have included the jPOS dependencies on pom.xml as follows.
<dependency>
<groupId>org.jpos</groupId>
<artifactId>jpos</artifactId>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>com.sleepycat</groupId>
<artifactId>je</artifactId>
<version>4.0.92</version>
</dependency>

To implement the transport sender, you need to subclass the AbstractTransportSender and implement its sendMessage method as follows.

public class ISO8583TransportSender extends AbstractTransportSender {

@Override
public void sendMessage(MessageContext msgCtx, String targetEPR,
OutTransportInfo outTransportInfo) throws AxisFault {

try {
URI isoURL = new URI(targetEPR);
ISOPackager packager = new GenericPackager(this.getClass()
.getResourceAsStream("jposdef.xml"));

ASCIIChannel chl = new ASCIIChannel(isoURL.getHost(),
isoURL.getPort(), packager);

writeMessageOut(msgCtx, chl);

} catch (Exception e) {
throw new AxisFault(
"An exception occured in sending the ISO message");
}

}

/**
* Writting the message to the output channel after applying correct message formatter
* @param msgContext
* @param chl
* @throws org.apache.axis2.AxisFault
* @throws java.io.IOException
*/
private void writeMessageOut(MessageContext msgContext,
ASCIIChannel chl) throws AxisFault, IOException {
ISO8583MessageFormatter messageFormatter = (ISO8583MessageFormatter)BaseUtils.getMessageFormatter(msgContext);
OMOutputFormat format = BaseUtils.getOMOutputFormat(msgContext);
messageFormatter.setAsciiChannel(chl);
messageFormatter.writeTo(msgContext, format, null, true);
}

Within the TransportSender, we are extracting the URL and then create the relevant entities for the Message Formatter and pass the control to the MessageFormatter. Within the MessageFormatter, we can send the actual message to the back end server.

public class ISO8583MessageFormatter implements MessageFormatter {

private ASCIIChannel asciiChannel;

@Override

public byte[] getBytes(MessageContext messageContext, OMOutputFormat omOutputFormat) throws AxisFault {

return new byte[0];

}

@Override

public void writeTo(MessageContext messageContext, OMOutputFormat omOutputFormat, OutputStream outputStream, boolean b) throws AxisFault {

ISOMsg isoMsg = toISO8583(messageContext);

ASCIIChannel chl = this.asciiChannel;

try {

chl.connect();

chl.send(isoMsg);

chl.disconnect();

} catch (Exception ex) {

throw new AxisFault(

"An exception occured in sending the ISO message");

}

@Override

public String getContentType(MessageContext messageContext, OMOutputFormat omOutputFormat, String s) {

return null;

}

@Override

public URL getTargetAddress(MessageContext messageContext, OMOutputFormat omOutputFormat, URL url) throws AxisFault {

return null;

}

@Override

public String formatSOAPAction(MessageContext messageContext, OMOutputFormat omOutputFormat, String s) {

return null;

}

public ISOMsg toISO8583(MessageContext messageContext) throws AxisFault {

SOAPEnvelope soapEnvelope = messageContext.getEnvelope();

OMElement isoElements = soapEnvelope.getBody().getFirstElement();

ISOMsg isoMsg = new ISOMsg();

@SuppressWarnings("unchecked")

Iterator<OMElement> fieldItr = isoElements.getFirstChildWithName(

new QName(ISO8583Constant.TAG_DATA)).getChildrenWithLocalName(

ISO8583Constant.TAG_FIELD);

String mtiVal = isoElements

.getFirstChildWithName(new QName(ISO8583Constant.TAG_CONFIG))

.getFirstChildWithName(new QName(ISO8583Constant.TAG_MTI))

.getText();

try {

isoMsg.setMTI(mtiVal);

while (fieldItr.hasNext()) {

OMElement isoElement = (OMElement) fieldItr.next();

String isoValue = isoElement.getText();

int isoTypeID = Integer.parseInt(isoElement.getAttribute(

new QName("id")).getAttributeValue());

isoMsg.set(isoTypeID, isoValue);

}

return isoMsg;

} catch (ISOException ex) {

throw new AxisFault("Error parsing the ISO8583 payload");

} catch (Exception e) {

throw new AxisFault("Error processing stream");

}

public ASCIIChannel getAsciiChannel() {

return asciiChannel;

}

public void setAsciiChannel(ASCIIChannel asciiChannel) {

this.asciiChannel = asciiChannel;

}

Here within the formatter, we are transforming the XML message into ISO8583 binary message and send to the back end server.

This is only an example of dividing your message sending logic to message sender and message formatter. You can design your implementation according to your requirement. Sometimes, you may not need specific formatter but you can do the formatting part also in the sender itself. But I have delegated a part of the message handling to message formatter for demonstration purpose.

Likewise, you can write a message receiver and message builder for receiving the messages via iso8583 protocol. I will leave that as an exercise for the reader.

Once we have the message sender and formatter implemented, we need to register them in the axis2.xml file. Let's go to the axis2.xml file and add following 2 entries there.

<messageFormatter contentType="application/iso8583"

class="org.wso2.transport.iso8583.message.ISO8583MessageFormatter"/>

<transportSender name="iso8583" class="org.wso2.transport.iso8583.ISO8583TransportSender"/>

Once you create the jar file from your custom transport implementation code, place it is ESB_HOME/repository/components/lib directory.

If you are done with the above steps, you can start the ESB server.

Let's create a sample API to interact with this custom transport implementation. Here is the API definition.

<api xmlns="http://ws.apache.org/ns/synapse" name="iso8583" context="/iso8583">

<resource methods="POST GET">

<inSequence>

<log level="full"></log>

<property name="OUT_ONLY" value="true"></property>

<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"></property>

<property name="messageType" value="application/iso8583" scope="axis2"></property>

<send>

<endpoint name="isoserver">

<address uri="iso8583://localhost:5000"></address>

</endpoint>

</send>

</inSequence>

</resource>

</api>

In the above configuration, I have specified the messageType as application/iso8583 such that it will engage the correct message formatter within the mediation flow.

Now we need to create a sample TestServer to test the functionality of the ISO8583 back end server. We can create a MockServer using the jpos library itself. Here is the code for the TestServer.

public class TestServer implements ISORequestListener {

static final String hostname = "localhost";

static final int portNumber = 5000;

public static void main(String[] args) throws ISOException {

ISOPackager packager = new GenericPackager("jposdef.xml");

ServerChannel channel = new ASCIIChannel(hostname, portNumber, packager);

ISOServer server = new ISOServer(portNumber, channel, null);

server.addISORequestListener(new TestServer());

System.out.println("ISO8583 server started...");

new Thread(server).start();

}

public boolean process(ISOSource isoSrc, ISOMsg isoMsg) {

try {

System.out.println("ISO8583 incoming message on host ["

+ ((BaseChannel) isoSrc).getSocket().getInetAddress()

.getHostAddress() + "]");

if (isoMsg.getMTI().equalsIgnoreCase("1800")) {

receiveMessage(isoSrc, isoMsg);

logISOMsg(isoMsg);

}

} catch (Exception ex) {

ex.printStackTrace();

}

return true;

}

private void receiveMessage(ISOSource isoSrc, ISOMsg isoMsg)

throws ISOException, IOException {

System.out.println("ISO8583 Message received...");

ISOMsg reply = (ISOMsg) isoMsg.clone();

reply.setMTI("1810");

reply.set(39, "00");

isoSrc.send(reply);

}

private static void logISOMsg(ISOMsg msg) {

System.out.println("----ISO MESSAGE-----");

try {

System.out.println(" MTI : " + msg.getMTI());

for (int i = 1; i <= msg.getMaxField(); i++) {

if (msg.hasField(i)) {

System.out.println(" Field-" + i + " : "

+ msg.getString(i));

}

} catch (ISOException e) {

e.printStackTrace();

} finally {

System.out.println("--------------------");

}

You can run the above program to mimic the ISO8583 server and then we can send a message from a client like Advanced REST Client plugin in the chrome browser. Our message payload should be like below.

<iso8583message>

<config>

<mti>1800</mti>

</config>

<data>

<field id="3">110</field>

<field id="5">4200.00</field>

<field id="48">Simple Credit Transaction</field>

<field id="6">645.23</field>

<field id="88">66377125</field>

</data>

</iso8583message>

When we send this message from client, ESB will accept the message and execute the message sender we have written and then selects the message formatter and send to the mock back end server. You can see the following log printed in the TestServer side if you have done all the things right.

ISO8583 server started...

ISO8583 incoming message on host [127.0.0.1]

ISO8583 Message received...

----ISO MESSAGE-----

MTI : 1800

Field-3 : 000110

Field-5 : 000004200.00

Field-6 : 000000645.23

Field-48 : Simple Credit Transaction

Field-88 : 0000000066377125

--------------------

When you are have configured the ESB, you will get exceptions if you do not copy following jar files to the lib directory alongside with custom transport jar file.

jpos-1.9.0.jar
jdom-1.1.3.jar
commons-cli-1.3.1.jar

Now we have written our message sender and message formatter implementations. Likewise, you can implement the message receiver and message builder code also. I have created an archive with all the relevant artifacts which I have developed for this blog post and uploaded them to github. You can download all the projects and relevant jar files from following location.

https://github.com/chanakaudaya/WSO2ESB-CustomTransport

[1] ISO8583 Specification - http://www.iso.org/iso/catalogue_detail.htm?csnumber=31628

[2] ISO8583 Wiki - http://en.wikipedia.org/wiki/ISO_8583

↧

WSO2 ESB Error Handling Tutorial - Part I (Client side error handling)

June 28, 2015, 7:48 am

≫ Next: WSO2 ESB 4.9.0 Released !!!

≪ Previous: Extending WSO2 ESB with a Custom Transport Implementation - Part II

Recently, I found a nice video on facebook which was shared by Sanjiva Weerawarana (CEO @ WSO2), which was a narration by Matt Damon. The original paragraph was taken from a speech by Howard Zinn's 1970 speech.

https://vimeo.com/48834336

According to that, world is topsy turvy (upside down). Wrong people are in power, Wrong people are out of power. But there is one thing missing in that speech. Which is that wrong people are using software, wrong people are not using software :).

Sorry about going out of the topic. But this speech has really moved me. Anyway, let's start talking about the subject. WSO2 ESB is the central hub of your SOA architecture. It will communicate with all kinds of heterogenous systems. These systems can go mad sometimes. In such a scenarios, WSO2 ESB should not go mad. If you haven't done proper error handling at WSO2 ESB, even though it does not go mad, people will feel that it has gone mad by looking at lengthy error logs and exceptions. So why do you let people to think in that way. Rather we can implement a proper error handling mechanism and make the WSO2 ESB looks solid at all time.

Here is a typical message flow in your enterprise system which involves WSO2 ESB.

In the above message flow, things can go wrong in all 3 components.

Client Error
ESB Error
Server Error

In all 3 scenarios, we need to have a proper handling mechanism to identify the error scenarios as soon as they occur. Otherwise, it will cost your organization's business. This can be ranged from hundreds of dollars to millions of dollars. Let's discuss about error handling at each and every component depicted above.

Handling Client errors

Anything can go wrong at any time. That is a fact in this world. So is the Clients who are using the services/APIs exposed through WSO2 ESB. Here are some example scenarios where clients go mad during the message execution.

Sending wrong messages (non-allowed content)
Closing connections early
Sending requests to wrong URLs

Let's discuss these scenarios one by one and learn about the error handling mechanisms which we can take to get over these.

Sending wrong messages

Let's say your client is sending XML messages to the ESB. Due to a mistake in the client code, let's say client sends the following message.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">

<soapenv:Header/>

<soapenv:Body>

<ser:getQuote>

<ser:request>

<xsd:symbol>WSO2&&&&</xsd:symbol>

</ser:request>

</ser:getQuote>

</soapenv:Body>

</soapenv:Envelope>

In the above message, client is sending '&' character within the XML message (This is only an example). Let's say we have a very simple PassThrough Proxy service defined in the ESB.

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

name="PassThroughProxy"

transports="https,http"

statistics="disable"

trace="disable"

startOnLoad="true">

<send/>

</outSequence>

</endpoint>

</target>

</proxy>

In this case, ESB will simply pass-through the message to the backend without worrying about the content. According to the capabilities of the back end server, it will respond with some message and ESB will pass the response to the client. No worries at this point.

All good with the above proxy service. Let's add a log mediator with log level "full". Once we have this content-aware mediator in the message flow, ESB tries to convert the incoming data stream into a canonical XML message. Here comes the exception. Since this message contains wrong XML characters, ESB will fail during the message building process. Now the proxy looks like below.

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

name="PassThroughProxy"

transports="https,http"

statistics="disable"

trace="disable"

startOnLoad="true">

</inSequence>

<send/>

</outSequence>

</endpoint>

</target>

</proxy>

Once we have this content-aware mediator in place, ESB will try to build the message and it will fail with an exception similar to this.

ERROR - NativeWorkerPool Uncaught exception

org.apache.axiom.om.OMException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[8,29]

Message: The entity name must immediately follow the '&' in the entity reference.

at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)

at org.apache.axiom.om.impl.llom.OMElementImpl.buildNext(OMElementImpl.java:653)

at org.apache.axiom.om.impl.llom.OMElementImpl.getFirstOMChild(OMElementImpl.java:670)

This is fine, since the message is wrong. But what is wrong here is that client did not get any message from ESB related to this error scenario. Then client will timeout the connection after waiting for the configured time duration and you can see the following log in the ESB log console.

[2015-06-27 17:26:08,885] WARN - SourceHandler Connection time out after request is read: http-incoming-2

We need to handle this situation with a proper error handler. We can define a fault sequence at the proxy service level. Then the message will go through this fault handler sequence and we can send a fault message to the client if we encounter this kind of message. Let's add the fault sequence to the proxy service.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="PassThroughProxy"
transports="https,http"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<log level="full"/>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence>
<makefault version="soap11">
<code xmlns:soap11Env="http://schemas.xmlsoap.org/soap/envelope/"
value="soap11Env:Client"/>
<reason expression="$ctx:ERROR_MESSAGE"/>
<role/>
</makefault>
<send/>
</faultSequence>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
</endpoint>
</target>
<description/>
</proxy>

Once we have the fault sequence, client will get a fault message as the response for this message.

Closing connections early

Another common use case is that client closes the connection before the server respond back with the message. When this happens, you can observe the following error message in the server log file.

[2015-06-28 19:28:37,524] WARN - SourceHandler Connection time out after request is read: http-incoming-1

This slowness can be occurred due to back end slowness or due to ESB server contention. In both scenarios, you need to increase the client timeout to get rid of this error. You can configure the client timeout to a considerable value which is greater than the maximum response time of the server. But configuring the client timeout only will not make this scenario work for you. The reason is that, even though the client has increased the timeout, ESB will close the connection after 60 seconds (default value). Therefore, you need to configure the client side HTTP connection timeout in the ESB_HOME/repository/conf/passthru-http.properties file with the following parameter. Add this parameter if it is not already there.

http.socket.timeout=120000

Sending Requests to wrong URL

Sometimes, client may send requests to non existing URLs. For example, let's say you have an API defined in the ESB like below.

<api xmlns="http://ws.apache.org/ns/synapse" name="test" context="/test">
<resource methods="POST GET" url-mapping="/echo">
<inSequence>
<log level="full"></log>
<send>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"></address>
</endpoint>
</send>
</inSequence>
</resource>
</api>

According to the definition, you need to send the request to following URL.

http://localhost:8280/test/echo

But due to a mistake by the client, it sends a request to the following URL

http://localhost:8280/test/echo2

Here what happens is, ESB will respond with 202 accepted message to the client. That is not the correct message ESB should send to the client since ESB is not processing this message correctly. What it should do is that, it needs to respond to the client with some error message such that client can go though error response and identify the root cause.

We need to define a special sequence for handling this kind of failed requests. You can define this sequence as given below.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="_resource_mismatch_handler_">
<payloadFactory media-type="xml">
<format>
<tp:fault xmlns:tp="http://test.com">
<tp:code>404</tp:code>
<tp:type>Status report</tp:type>
<tp:message>Not Found</tp:message>
<tp:description>The requested resource (/$1) is not available.</tp:description>
</tp:fault>
</format>
<args>
<arg xmlns:ns="http://org.apache.synapse/xsd" xmlns:ns3="http://org.apache.synapse/xsd" expression="$axis2:REST_URL_POSTFIX" evaluator="xml"></arg>
</args>
</payloadFactory>
<property name="RESPONSE" value="true" scope="default"></property>
<property name="NO_ENTITY_BODY" action="remove" scope="axis2"></property>
<property name="HTTP_SC" value="404" scope="axis2"></property>
<header name="To" action="remove"></header>
<send></send>
<drop></drop>
</sequence>

In the above sequence, you can change the internal mediators as per your wish. But the name of the sequence should be as it is (_resource_mismatch_handler_). One you have this sequence in place, clients will get the following error message if they send requests to non-existing API resources.

<tp:fault>
<tp:code>404</tp:code>
<tp:type>Status report</tp:type>
<tp:message>Not Found</tp:message>
<tp:description>The requested resource (//echo-test) is not available.</tp:description>
</tp:fault>

I will be discussing about the rest of the 2 scenarios in a future blog post.

Handling back end Server errors

Handling ESB errors

↧