Important steps of clustering WSO2 API Manager in hybrid mode or worker manager seperated mode

In this post I'm discussing some important facts about additional transports and port mappings in WSO2 Elastic Load Balancer and WSO2 API Manager product clustering scenarios. To find out a complete guide you have to follow WSO2 Product clustering documentations.

WSO2 API Manager is a descendant of WSO2 ESB. So it uses a few more than one transports which is Servlet and Passthrough

I'm using 101 as ELB and APIM's port offset, so the port mapping will be as following

ELB	Port offset 101	APIM	Port offset 101
http	8381	http (pt)	8381
https	8344	https (pt)	8344
http2	8382	http2 (servlet)	9864
https2	8345	https2 (servlet)	9544

When fronting an API-M with ELB, you have to open two more transports in ELB to map to the APIM's extra transports. For that you have to add additional transportReceiver and transportSender configurations.  That is done as following.

ELB - axis2.xml

transportRecceiver

   <transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">  
    <parameter name="port">8280</parameter>  
    <parameter name="non-blocking"> true</parameter>  
    <parameter name="httpGetProcessor" locked="false">org.wso2.carbon.transport.nhttp.api.PassThroughNHttpGetProcessor</parameter>  
   </transportReceiver>  
   <transportReceiver name="http2" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">  
    <parameter name="port">8281</parameter>  
    <parameter name="non-blocking"> true</parameter>  
    <parameter name="httpGetProcessor" locked="false">org.wso2.carbon.transport.nhttp.api.PassThroughNHttpGetProcessor</parameter>  
   </transportReceiver>  
   <transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">  
     <parameter name="port" locked="false">8243</parameter>  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="httpGetProcessor" locked="false">org.wso2.carbon.transport.nhttp.api.PassThroughNHttpGetProcessor</parameter>  
     <!--parameter name="bind-address" locked="false">hostname or IP address</parameter-->  
     <!--parameter name="WSDLEPRPrefix" locked="false">https://apachehost:port/somepath</parameter-->  
     <parameter name="keystore" locked="false">  
       <KeyStore>  
         <Location>repository/resources/security/wso2carbon.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
         <KeyPassword>wso2carbon</KeyPassword>  
       </KeyStore>  
     </parameter>  
     <parameter name="truststore" locked="false">  
       <TrustStore>  
         <Location>repository/resources/security/client-truststore.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
       </TrustStore>  
     </parameter>  
   </transportReceiver>  
   <transportReceiver name="https2" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">  
     <parameter name="port" locked="false">8244</parameter>  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="httpGetProcessor" locked="false">org.wso2.carbon.transport.nhttp.api.PassThroughNHttpGetProcessor</parameter>  
     <!--parameter name="bind-address" locked="false">hostname or IP address</parameter-->  
     <!--parameter name="WSDLEPRPrefix" locked="false">https://apachehost:port/somepath</parameter-->  
     <parameter name="keystore" locked="false">  
       <KeyStore>  
         <Location>repository/resources/security/wso2carbon.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
         <KeyPassword>wso2carbon</KeyPassword>  
       </KeyStore>  
     </parameter>  
     <parameter name="truststore" locked="false">  
       <TrustStore>  
         <Location>repository/resources/security/client-truststore.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
       </TrustStore>  
     </parameter>  
   </transportReceiver>  

transportSender

   <transportSender name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpSender">  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="warnOnHTTP500" locked="false">*</parameter>  
     <!--parameter name="http.proxyHost" locked="false">localhost</parameter>  
     <parameter name="http.proxyPort" locked="false">3128</parameter>  
     <parameter name="http.nonProxyHosts" locked="false">localhost|moon|sun</parameter-->  
   </transportSender>  
   <transportSender name="http2" class="org.apache.synapse.transport.passthru.PassThroughHttpSender">  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="warnOnHTTP500" locked="false">*</parameter>  
     <!--parameter name="http.proxyHost" locked="false">localhost</parameter>  
     <parameter name="http.proxyPort" locked="false">3128</parameter>  
     <parameter name="http.nonProxyHosts" locked="false">localhost|moon|sun</parameter-->  
   </transportSender>  
   <transportSender name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLSender">  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="keystore" locked="false">  
       <KeyStore>  
         <Location>repository/resources/security/wso2carbon.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
         <KeyPassword>wso2carbon</KeyPassword>  
       </KeyStore>  
     </parameter>  
     <parameter name="truststore" locked="false">  
       <TrustStore>  
         <Location>repository/resources/security/client-truststore.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
       </TrustStore>  
     </parameter>  
     <parameter name="HostnameVerifier">AllowAll</parameter>  
    </transportSender>  
   <transportSender name="https2" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLSender">  
     <parameter name="non-blocking" locked="false">true</parameter>  
     <parameter name="keystore" locked="false">  
       <KeyStore>  
         <Location>repository/resources/security/wso2carbon.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
         <KeyPassword>wso2carbon</KeyPassword>  
       </KeyStore>  
     </parameter>  
     <parameter name="truststore" locked="false">  
       <TrustStore>  
         <Location>repository/resources/security/client-truststore.jks</Location>  
         <Type>JKS</Type>  
         <Password>wso2carbon</Password>  
       </TrustStore>  
     </parameter>  
     <parameter name="HostnameVerifier">AllowAll</parameter>  
       <!--supports Strict|AllowAll|DefaultAndLocalhost or the default if none specified -->  
    </transportSender>  

The loadbalancer.conf of ELB need to have the information of the APIM cluster  such as domains, and hosts. Normally the hosts are separated as manager and worker. But here since, I'm going to use the APIM instance as a hybrid instance the separation of host names as management and worker will be a wrong configuration.

If an APIM node joined the ELB as a management node, even if it's Passthrough transports are mapped into ELB's one transport, the requests will not be served, since ELB doesn't know if APIM node is capable of serving pt requests.

So the loadbalancer.conf configuration specific to the APIM is as following.

   apimanager {  
     # multiple hosts should be separated by a comma.  
     #url_suffix       esb.wso2.com;  
     domains  {  
       wso2.am160.domain {  
         tenant_range  *;  
         group_mgt_port 4050;  
           hosts mgt.am160.gw.com,am160.gw.com;  
       }  
     }  
   }  

if, however, someone need to use seperate management and worker nodes for gateway, then the elb configuraion should be as following. Note the difference.

   apimanager {  
     # multiple hosts should be separated by a comma.  
     #url_suffix       esb.wso2.com;  
     domains  {  
       wso2.am160.domain {  
         tenant_range  *;  
         group_mgt_port 4050;  
         mgt {  
           hosts mgt.am160.gw.com;  
         }  
         worker {  
           hosts am160.gw.com;  
         }  
     }  
   }  

In the APIM side, other than the passthrough transports, I have opened the Servlet transports naming their transport names to match with those in the ELB. This is not mandatory since it is by default open in any wso2 server. But I've done this specifically so that the transport Receiver names matches with the ELB's ones.

   <transportReceiver name="http2"  
             class="org.wso2.carbon.core.transports.http.HttpTransportListener">  
     <!--  
       Uncomment the following if you are deploying this within an application server. You  
       need to specify the HTTP port of the application server  
     -->  
     <parameter name="port">9763</parameter>  
     <!--  
     Uncomment the following to enable Apache2 mod_proxy. The port on the Apache server is 80  
     in this case.  
     -->  
     <!--<parameter name="proxyPort">80</parameter>-->  
   </transportReceiver>  
   <transportReceiver name="https2"  
             class="org.wso2.carbon.core.transports.http.HttpsTransportListener">  
     <!--  
       Uncomment the following if you are deploying this within an application server. You  
       need to specify the HTTPS port of the application server  
     -->  
     <parameter name="port">9443</parameter>  
     <!--  
     Uncomment the following to enable Apache2 mod_proxy. The port on the Apache server is 443  
     in this case.  
     -->  
     <!--<parameter name="proxyPort">443</parameter>-->  
   </transportReceiver>  

The other most important thing is the port mapping.

Those transports we have defined as http and https in ELB and in APIM will be mapped together when the APIM joins the ELB, since those are the default names that is used as transportSender and transportReceiver names.

The other transports in APIM which are the Servlet transports need to be mapped to ELBs, additionally opened transports, so that ELB knows how to server Management requests of the APIM node.  The configuration is as follows.

     <parameter name="properties">  
       <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
       <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
       <property name="port.mapping.8382" value="9864"/>  
       <property name="port.mapping.8345" value="9544"/>  
       <!--<property name="subDomain" value="mgt"/>-->  
     </parameter>  

See that I have commented out the subDomain property so that the APIM does not specifically say ELB, that it is a management or worker node. So that ELB can serve Passthrough and Servlet requests together.

Oracle Database Commands

In Oracle unlike mysql, the database is represented by a schema owner.

A database schema is a logical container for data structures, called schema objects. Examples of schema objects are tables and indexes.

A database user has a password and particular database privileges assigned by the database admin.

In Oracle each user owns a single schema, which has the same name as the user. The schema contains the data for the user owning the schema. For example, the user "chamara" owns the schema "chamara" which contains schema objects such as tables in the database. In a production database, the schema owner usually represents a database application rather than a person.

Using the following commands u can create a schema in oracle database, which is a database you can used within your application.

Connect to the database as the db admin;

 sqlplus sys/admin as sysdba  

Drop user/schema if exists;

 drop user chamaraa cascade;

Create user/schema;

 create user chamaraa identified by chamaraa account unlock;

Grant connect permission;

 grant connect to chamaraa;

Grant other permisssions;

 grant create session,dba to chamaraa;

finally;

 commit;
 exit;

Connect to the created schema;

 sqlplus chamaraa/chamaraa@orcl

Running a script;

 SQL> @C:\path\oracle.sql;

Simple HTTP GET request processor with Perl

Perl script;

 #!/usr/bin/perl  
 use HTTP::Daemon;  
 use HTTP::Status;  
 use Net::HTTP;  
 my $server = HTTP::Daemon->new(  
   LocalAddr => 'chamaradell',   
   LocalPort => '3113') || die;  
 system(clear);    
 print ("HTTP Server is up at: <URL: ", $server->url, "\n");  
 while(my $client = $server->accept) {  
   while(my $request = $client->get_request) {  
      if($request->method eq 'GET') {  
        print "GET request accepted\n";        
        $client->send_status_line(200, "OK");  
       $client->send_header("Content-Type", "text/html");  
       $client->send_file_response("index.html");  
       $client->close;  
      }  
   }   
   $client->close;  
   undef($client);  
 }  

A simple html page;

 <!DOCTYPE html>  
 <html>  
   <body>  
    <h1>Index</h1>  
    <p>This is a test HTTP Server<p>  
   </body>  
 </html>  

The HTTP server will be started after running the perl script. Now if the browser is pointed to the correct url;

 http://chamaradell:3113/  

This will give the html page in the browser.

If the content is requested using curl (as following);

 $ curl -v -X GET http://chamaradell:3113  

The content will be returned as;

 * Rebuilt URL to: http://chamaradell:3113/  
 * Adding handle: conn: 0x19ecbd0  
 * Adding handle: send: 0  
 * Adding handle: recv: 0  
 * Curl_addHandleToPipeline: length: 1  
 * - Conn 0 (0x19ecbd0) send_pipe: 1, recv_pipe: 0  
 * About to connect() to chamaradell port 3113 (#0)  
 *  Trying 127.0.1.1...  
 * Connected to chamaradell (127.0.1.1) port 3113 (#0)  
 > GET / HTTP/1.1  
 > User-Agent: curl/7.32.0  
 > Host: chamaradell:3113  
 > Accept: */*  
 >   
 < HTTP/1.1 200 OK  
 < Content-Type: text/html  
 < HTTP/1.1 200 OK  
 < Date: Sat, 08 Feb 2014 17:32:49 GMT  
 * Server libwww-perl-daemon/6.01 is not blacklisted  
 < Server: libwww-perl-daemon/6.01  
 < Content-Type: text/html  
 < Content-Length: 112  
 < Last-Modified: Sat, 08 Feb 2014 17:25:01 GMT  
 <   
 <!DOCTYPE html>  
 <html>  
   <body>  
    <h1>Index</h1>  
    <p>This is a test HTTP Server<p>  
   </body>  
 </html>  
 * Connection #0 to host chamaradell left intact  

The HTTP Status 200 is returned by the server by default. You can return any status using that method and test various http clients with this server, for various conditions.

CQL 3 commands for Cassandra 1.2

I used apache-cassandra-1.2.4 cqlsh client with WSO2 Storage Server 1.1.0

Connection:

 $ ./cqlsh -3 node3 9160 -u admin -p admin  
 Connected to SSCluster at node3:9160.  
 [cqlsh 2.3.0 | Cassandra 1.2.10 | CQL spec 3.0.0 | Thrift protocol 19.36.0]  
 Use HELP for help.  
 cqlsh>

Create Keyspace:

 cqlsh> CREATE KEYSPACE ks215 WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 3};  

Use Keyspace:

cqlsh> use ks215;

Alter Keyspace

cqlsh:ks215> ALTER KEYSPACE ks215 WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 2};

Create Table (Column Family)

 cqlsh:ks215> CREATE TABLE company ( empID int, nationalID int, first_name varchar, last_name varchar, possition varchar, PRIMARY KEY (empID, nationalID));  

Insert into table:

 cqlsh:ks215> INSERT INTO company (empID, nationalID, first_name, last_name, possition) VALUES (101, 900810543, 'Chamara', 'Ariyarathne', 'SSE');  

Select:

 cqlsh:ks215> SELECT * FROM company;  

How to create a patch

1. Checkout the code

I checked out the WSO2 JAX-RS Starbucks service from here;

 http://svn.wso2.org/repos/wso2/carbon/platform/branches/4.2.0/products/as/5.2.0/modules/samples/product/Jaxws-Jaxrs/jaxrs_starbucks_service  

2. Make the change

I wanted this service to work as Non-SOAP for HTTP POST requests; So I changed the code as;

From here;

 @Consumes(MediaType.TEXT_XML)  // consumes text/xml  

To this;

 @Consumes(MediaType.APPLICATION_XML)  // consumes application/xml  

3. Create the diff

Now use the following svn command to create the patch diff

 $ svn diff > patch.diff  

4. Patch diff will look like this

 Index: src/main/java/demo/jaxrs/server/StarbucksOutletService.java  
 ===================================================================  
 --- src/main/java/demo/jaxrs/server/StarbucksOutletService.java (revision 184754)  
 +++ src/main/java/demo/jaxrs/server/StarbucksOutletService.java (working copy)  
 @@ -21,7 +21,7 @@  
    @Path("/orders/")  
  //  @Produces(MediaType.TEXT_PLAIN)  
    @Produces(MediaType.APPLICATION_JSON)  // produces application/json  
 -  @Consumes(MediaType.TEXT_XML)  // consumes text/xml  
 +  @Consumes(MediaType.APPLICATION_XML)  // consumes application/xml  
    public Response addOrder (Order orderBean);  
    /**  

Clustering WSO2 Identity Server and Load balancing it with WSO2 ELB

The WSO2 Identity server 4.5.0 and WSO2 Enterprise Load Balancer 2.1.0 is now released. So with this post I'm going to guide you with each and every and complete set of steps on how to cluster Identity server with two nodes and loadbalance it with ELB. And this setup is going to be in one Ubuntu machine.

Prerequisites:
1. Ubuntu 12.04 or a Linux similar platform
2. Java 1.6 or above (I'm using 1.7.0_25 64-bit)
3. Mysql (I'm using Ver 14.14 Distrib 5.5.32)

Let's go through the steps on how to cluster IS

1. Pointing to same User store and User Management db.

Being a cluster of IS, every node need to share the same User Store and User Management databases. So that every request come through load balancer will be served by any node similarly.

For User Store you can use any LDAP as Active Directory, Apache DS, OpenLDAP etc. Here I'm using the JDBC userstore which is supported by Identity Server to use a relational database as the user store.

I'm using mysql database for this.

Create a new database

 $ mysql -uroot -proot  

 mysql> create database is_user_store;  

Populate the tables needed. For that you can use the db scripts in the dbscripts directory in Identity Server home.

 ${IS_HOME}/dbscripts/mysql.sql  
 ${IS_HOME}/dbscripts/identity/mysql.sql  
 ${IS_HOME}/dbscripts/service-provider/mysql.sql  

 mysql> source mysql.sql;  
 mysql> source identity/mysql.sql;  
 mysql> source service-provider/mysql.sql;  

Now you have to use this database in the configurations. For the User Store you have to add the following configurations;

In the;

 ${IS_HOME}/repository/conf/datasources/master-datasources.xml  

Have the configuration

     <datasource>  
       <name>WSO2_USERSTORE_DB</name>  
       <description>The datasource used for registry and user manager</description>  
       <jndiConfig>  
         <name>jdbc/WSO2UserStoreDB</name>  
       </jndiConfig>  
       <definition type="RDBMS">  
         <configuration>  
           <url>jdbc:mysql://localhost:3306/is_user_store</url>  
           <username>root</username>  
           <password>root</password>  
           <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
           <maxActive>50</maxActive>  
           <maxWait>60000</maxWait>  
           <testOnBorrow>true</testOnBorrow>  
           <validationQuery>SELECT 1</validationQuery>  
           <validationInterval>30000</validationInterval>  
         </configuration>  
       </definition>  
     </datasource>  

And that jndi database reference have to be used in the user-mgt.xml as to use it as the userstore

${IS_HOME}/repository/conf/user-mgt.xml

     <UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">  
       <Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>  
       <Property name="ReadOnly">false</Property>  
       <Property name="MaxUserNameListLength">100</Property>  
       <Property name="IsEmailUserName">false</Property>  
       <Property name="DomainCalculation">default</Property>  
       <Property name="PasswordDigest">SHA-256</Property>  
       <Property name="StoreSaltedPassword">true</Property>  
       <Property name="ReadGroups">true</Property>  
       <Property name="WriteGroups">true</Property>  
       <Property name="UserNameUniqueAcrossTenants">false</Property>  
       <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>  
       <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>  
       <Property name="UsernameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>  
       <Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>  
       <Property name="RolenameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>  
       <Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>  
       <Property name="UserRolesCacheEnabled">true</Property>  
       <Property name="MaxRoleNameListLength">100</Property>  
       <Property name="MaxUserNameListLength">100</Property>  
       <Property name="SharedGroupEnabled">false</Property>  
       <Property name="SCIMEnabled">false</Property>  
       <Property name="dataSource">jdbc/WSO2UserStoreDB</Property>  
     </UserStoreManager>  

And for that to be the Primary userstore of the IS that need to be on top in the user-mgt.xml as the first mentioned userstore configuration. Every other userstore configuration you specify will be secondary userstores with a domain name.

and this has to be done in both of the IS nodes.

Now lets create another db and set it as the User Management db. Run the scripts to populate tables and refer it in the master-datasources.xml as following.

     <datasource>  
       <name>WSO2_UM_DB</name>  
       <description>The datasource used for registry and user manager</description>  
       <jndiConfig>  
         <name>jdbc/WSO2UserMgtDB</name>  
       </jndiConfig>  
       <definition type="RDBMS">  
         <configuration>  
           <url>jdbc:mysql://localhost:3306/is_user_mgt</url>  
           <username>root</username>  
           <password>root</password>  
           <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
           <maxActive>50</maxActive>  
           <maxWait>60000</maxWait>  
           <testOnBorrow>true</testOnBorrow>  
           <validationQuery>SELECT 1</validationQuery>  
           <validationInterval>30000</validationInterval>  
         </configuration>  
       </definition>  
     </datasource>  

You have to refer this in the user-mgt.xml as in following.

     <Configuration>  
         <AddAdmin>true</AddAdmin>  
         <AdminRole>admin</AdminRole>  
         <AdminUser>  
            <UserName>admin</UserName>  
            <Password>admin</Password>  
         </AdminUser>  
       <EveryOneRoleName>everyone</EveryOneRoleName> <!-- By default users in this role sees the registry root -->  
       <Property name="dataSource">jdbc/WSO2UserMgtDB</Property>  
     </Configuration>  

Now the User management related configurations are done. The next step is sharing the config and governance registries among each IS nodes. That is called Registry Mounting.

2. Registry Mounting

Let's create a database to be used in the mount configurations so that it will be the same repository for both IS nodes to share as the Config and Governance collections. That will be called the Mounted db. However for each IS node, the local registry will still be pointed to each and every node's H2 database. That configuration is already there in the master-datasources.xml. That need not to be changed.

You'll have to create a database in mysql and populate the tables. Let's see how it will be refered in the configurations.

First in the;

${IS_HOME}/repository/conf/datasources/master-datasources.xml  

     <datasource>  
       <name>WSO2_MOUNT_DB</name>  
       <description>The datasource used for registry and user manager</description>  
       <jndiConfig>  
         <name>jdbc/WSO2ISMountDB</name>  
       </jndiConfig>  
       <definition type="RDBMS">  
         <configuration>  
           <url>jdbc:mysql://localhost:3306/is_mount_db</url>  
           <username>root</username>  
           <password>root</password>  
           <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
           <maxActive>50</maxActive>  
           <maxWait>60000</maxWait>  
           <testOnBorrow>true</testOnBorrow>  
           <validationQuery>SELECT 1</validationQuery>  
           <validationInterval>30000</validationInterval>  
         </configuration>  
       </definition>  
     </datasource>  

And now this jndi database reference will be used in both nodes as in;

${IS_HOME}/repository/conf/registry.xml

   <currentDBConfig>wso2registry</currentDBConfig>  
   <readOnly>false</readOnly>  
   <enableCache>true</enableCache>  
   <registryRoot>/</registryRoot>  
   <dbConfig name="wso2registry">  
     <dataSource>jdbc/WSO2CarbonDB</dataSource>  
   </dbConfig>  
   <dbConfig name="mounted_registry">  
     <dataSource>jdbc/WSO2ISMountDB</dataSource>  
   </dbConfig>  
   <handler class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyHandler">  
     <filter class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyMediaTypeMatcher">  
       <property name="mediaType">application/xacml-policy+xml</property>  
     </filter>  
   </handler>  

   <remoteInstance url="https://localhost:9443/registry">  
     <id>instanceid</id>  
     <dbConfig>mounted_registry</dbConfig>  
     <readOnly>false</readOnly>  
     <enableCache>true</enableCache>  
     <registryRoot>/</registryRoot>   
   </remoteInstance>  
   <mount path="/_system/config" overwrite="true">  
     <instanceId>instanceid</instanceId>  
     <targetPath>/_system/nodes</targetPath>  
   </mount>  
   <mount path="/_system/governance" overwrite="true">  
     <instanceId>instanceid</instanceId>  
     <targetPath>/_system/governance</targetPath>  
   </mount>  

Now let's look at the clustering configurations.

3. Clustering of IS nodes

We need to cluster two IS nodes as management nodes, and load balance it with WSO2 ELB.

First let's look at Identity Server configurations.

This is for the IS node 1.

${IS_HOME}/repository/conf/axis2/axis2.xml

There is a section as "clustering". You have to find the place and edit as following.

 <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"  
         enable="true">  

 <parameter name="membershipScheme">wka</parameter>  

We have to specify a domain for Identity Server cluster. And this domain name need to be unique from other WSO2 product clusters. Even from ELB domain.

 <parameter name="domain">wso2.is.chamara.domain</parameter>  

 <parameter name="localMemberPort">4001</parameter>  

 <parameter name="properties">  
       <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
       <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
       <property name="subDomain" value="mgt"/>  
 </parameter>  

The next step is to define the well known member for IS nodes. (this will be kept as default)

         <members>
            <member>
                <hostName>127.0.0.1</hostName>
                <port>4000</port>
            </member>
        </members>

Here the well known member is WSO2 ELB. In the ELB in the file

${ELB_HOME}/repository/conf/axis2/axis2.xml

the property of localMemberHost will be kept as default.

<parameter name="localMemberHost">127.0.0.1</parameter>

and in the file

${ELB_HOME}/repository/conf/loadbalancer.conf

I will define group management port as 4000. This Hostname and the Port are the socket which each IS node will search for, as the well known member. 



In the carbon.xml

${IS_HOME}/repository/conf/carbon.xml

 <Offset>1</Offset>

 <HostName>wso2.is.chamara.com</HostName>  

 <MgtHostName>wso2.is.chamara.com</MgtHostName>

This hostname will be the one that is used by IS cluster and the ELB. It need to be specified in the;

 /etc/hosts  

as;

 127.0.0.1 wso2.is.chamara.com  

In the carbon.xml we are specifying the deployment synchronizer configuration as well. The deployment synchronizer will be enabled for, each IS nodes to share the secondary user store configurations.

Here I'm using svn based deployment synchronizer. So you'll have to find your own svn server and specify a location. Or else you can use Registry Based Deployment Synchronizer

   <DeploymentSynchronizer>  
     <Enabled>true</Enabled>  
     <AutoCommit>true</AutoCommit>  
     <AutoCheckout>true</AutoCheckout>  
     <RepositoryType>svn</RepositoryType>  
     <SvnUrl>http://svnexample.wso2.com/svn/chamara450</SvnUrl>  
     <SvnUser>wso2</SvnUser>  
     <SvnPassword>wso2123</SvnPassword>  
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>  
   </DeploymentSynchronizer>  

I'm going to run ELB on default HTTP/HTTPS ports which are 80/443.
So we have to specify those as proxy ports in the

${IS_HOME}/repository/conf/tomcat/catalina-server.xml

 <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9763"  
         proxyPort="80"  

 <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"  
         port="9443"  
         proxyPort="443"  

In the IS node 2 the configurations are almost same as for node1. The only differences that need to be changed are; in;

${IS_HOME}/repository/conf/axis2/axis2.xml

 <parameter name="localMemberPort">4002</parameter>  

and in;

${IS_HOME}/repository/conf/carbon.xml

 <Offset>2</Offset>

4. Sharing of same keystores in IS cluster and ELB

In this step I'm going to explain how to use the same keystore in three servers and having certificates in the truststores. We can always use the default keystore wso2carbon.jks and it's public certificates. But in this case since we are using ELB in default ports, the servers need to have same keystore with the Common Name as the domain name (CN=wso2.is.chamara.com). So here are the steps to generate your own keystore.

 $ keytool -genkey -alias iscarbon -keyalg RSA -keysize 1024 -keypass iscarbon -keystore iscarbon.jks -storepass iscarbon  
 What is your first and last name?  
  [Unknown]: wso2.is.chamara.com  
 What is the name of your organizational unit?  
  [Unknown]: users.wso2  
 What is the name of your organization?  
  [Unknown]: wso2  
 What is the name of your City or Locality?  
  [Unknown]: Colombo  
 What is the name of your State or Province?  
  [Unknown]: Western  
 What is the two-letter country code for this unit?  
  [Unknown]: 94  
 Is CN=wso2.is.chamara.com, OU=users.wso2, O=wso2, L=Colombo, ST=Western, C=94 correct?  
  [no]: yes  

Here for local testing purposes it is ok to use self signed certificate. If this is going to be used in production the public certificate need to be signed by Certificate Authority.

Now export the public certificate and import it to the server trust store. Here it need to be imported in two IS nodes and the ELB.

 $ keytool -export -alias iscarbon -keystore iscarbon.jks -storepass iscarbon -file iscarbon.pem  

 $ keytool -import -alias iscarbon -file iscarbon.pem -keystore client-truststore.jks -storepass wso2carbon  

Now we need to change the configuration files to use this new keystore as the server keystore. For that following locations needs to be updated accordingly.

 identity.xml: <Location>${carbon.home}/repository/resources/security/iscarbon.jks</Location>  
 identity.xml: <Password>iscarbon</Password>  
 security/cipher-text.properties: Carbon.Security.KeyStore.Password=[iscarbon]  
 security/cipher-text.properties: Carbon.Security.KeyStore.KeyPassword=[iscarbon]  
 carbon.xml: <Location>${carbon.home}/repository/resources/security/iscarbon.jks</Location>  
 carbon.xml: <Password>iscarbon</Password>  
 carbon.xml: <KeyAlias>iscarbon</KeyAlias>  
 carbon.xml: <KeyPassword>iscarbon</KeyPassword>  
 axis2/axis2.xml: <Location>repository/resources/security/iscarbon.jks</Location>  
 axis2/axis2.xml: <Password>iscarbon</Password>  
 axis2/axis2.xml: <KeyPassword>iscarbon</KeyPassword>  
 axis2/axis2.xml: <Password>iscarbon</Password>  
 axis2/axis2.xml: <Location>repository/resources/security/iscarbon.jks</Location>  
 axis2/axis2.xml: <Password>iscarbon</Password>  
 axis2/axis2.xml: <KeyPassword>iscarbon</KeyPassword>  
 axis2/axis2.xml: <Password>iscarbon</Password>  

5. Configuring the Elastic Load Balancer

We are done with configuring the IS nodes. Lets look at how to configure the ELB

${ELB_HOME}/repository/conf/axis2/axis2.xml

 <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">  

 <parameter name="domain">wso2.chamara.lb.domain</parameter>  

this is a individual port unique from is nodes' localMemberPort

 <parameter name="localMemberPort">4100</parameter>  

For changing Transport ports (Proxy ports in IS nodes)

 <transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">  
    <parameter name="port">80</parameter>  

 <transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">  
     <parameter name="port" locked="false">443</parameter>  

In the file

${ELB_HOME}/repository/conf/loadbalancer.conf

Remove all the services entries and add the following.

   identity {  
     domains  {  
       wso2.is.chamara.domain {  
         tenant_range  *;  
         group_mgt_port 4000;  
         mgt {  
           hosts wso2.is.chamara.com;  
         }  
       }  
     }  
   }  

In the ELB also you need to specify keystore configurations as above, in order to use the iscarbon.jks. Here you don't have add configure or add any identity.xml.

6. Running the cluster

Now we have come to the last part. First run the ELB. For run that you need to be the superuser of that computer since we are running ELB on default ports.

 {ELB_HOME}# ./bin/wso2server.sh  

Then run two IS servers.

 {IS_HOME}$ ./bin/wso2server.sh  

You will see the following logs in ELB while IS nodes joining ELB.

 [2013-09-04 08:21:15,211] INFO - HazelcastGroupManagementAgent Member joined [ad2e310c-bf27-4bf3-9e03-afbe2117b6dd]: /127.0.0.1:4001  
 [2013-09-04 08:21:18,275] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:21:18,276] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster  

 [2013-09-04 08:24:59,428] INFO - HazelcastGroupManagementAgent Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:02,493] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:02,494] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster  

And also while IS node joins each other following log will be appeared.

Node 1

 [2013-09-04 08:22:15,083] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:22:23,314] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8228ms  
 [2013-09-04 08:22:23,365] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [bb0a0482-aea0-4895-a2f1-68ee721a38a5] - Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:22:23,374] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

 [2013-09-04 08:24:59,430] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002  
 [2013-09-04 08:25:01,494] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

Node 2

 [2013-09-04 08:24:53,226] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true  
 [2013-09-04 08:25:01,451] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8223ms  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [3db96e0f-d4e9-4c45-bbdf-6331682a61cd] - Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  
 [2013-09-04 08:25:01,492] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true  

Now you can access the IS cluster Management Consoles using following url;

 https://wso2.is.chamara.com/carbon/  

Login using the default admin user {admin:admin}

WSO2 Appserver 5.2.0 JAX-RS Starbucks Sample

Checkout the sample from

 http://svn.wso2.org/repos/wso2/carbon/platform/branches/4.2.0/products/as/5.2.0/modules/samples/product/Jaxws-Jaxrs/jaxrs_starbucks_service  

and build with

 mvn clean install  

Or download the pre built app from here

Deploy the sample app in WSO2 Appserver 5.2.0.

Now let's look at the RESTful Invocations using curl.

1. Retrieve

 curl -v -X GET http://localhost:9763/jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders/123

 * About to connect() to localhost port 9764 (#0)  
 *  Trying 127.0.0.1... connected  
 > GET /jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders/123 HTTP/1.1  
 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3  
 > Host: localhost:9764  
 > Accept: */*  
 >   
 < HTTP/1.1 200 OK  
 < Date: Sat, 31 Aug 2013 19:07:42 GMT  
 < Content-Type: application/json  
 < Transfer-Encoding: chunked  
 < Server: WSO2 Carbon Server  
 <   
 * Connection #0 to host localhost left intact  
 * Closing connection #0  
 {"Order":{"additions":"Milk","drinkName":"Vanilla Flavored Coffee","locked":false,"orderId":123}}  

2. Create

 <Order> 
   <drinkName>Mocha Coffee</drinkName> 
   <additions>Caramel</additions> 
 </Order> 

 curl -v -X POST -H "Content-Type: text/xml" -d @starbucks_order.xml http://localhost:9763/jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders

 * About to connect() to localhost port 9764 (#0)  
 *  Trying 127.0.0.1... connected  
 > POST /jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders HTTP/1.1  
 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3  
 > Host: localhost:9764  
 > Accept: */*  
 > Content-Type: text/xml  
 > Content-Length: 86  
 >   
 * upload completely sent off: 86out of 86 bytes  
 < HTTP/1.1 200 OK  
 < Date: Sat, 31 Aug 2013 19:10:09 GMT  
 < Content-Type: application/json  
 < Transfer-Encoding: chunked  
 < Server: WSO2 Carbon Server  
 <   
 * Connection #0 to host localhost left intact  
 * Closing connection #0  
 {"Order":{"additions":"Caramel","drinkName":"Mocha Coffee","locked":false,"orderId":"55a7a380-e852-44f6-b2d6-c40bd9d70be0"}}  

3. Update

 {  
  "Order": {  
   "orderId": "55a7a380-e852-44f6-b2d6-c40bd9d70be0",  
   "additions": "Chocolate Chip Cookies"  
  }  
 }  

 curl -v -X PUT -H "Content-Type: application/json" -d @starbucks_update.json http://localhost:9763/jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders  

 * About to connect() to localhost port 9764 (#0)  
 *  Trying 127.0.0.1... connected  
 > PUT /jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders HTTP/1.1  
 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3  
 > Host: localhost:9764  
 > Accept: */*  
 > Content-Type: application/json  
 > Content-Length: 112  
 >   
 * upload completely sent off: 112out of 112 bytes  
 < HTTP/1.1 200 OK  
 < Date: Sat, 31 Aug 2013 19:14:11 GMT  
 < Content-Type: application/json  
 < Transfer-Encoding: chunked  
 < Server: WSO2 Carbon Server  
 <   
 * Connection #0 to host localhost left intact  
 * Closing connection #0  
 {"Order":{"additions":"Chocolate Chip Cookies","drinkName":"Mocha Coffee","locked":false,"orderId":"55a7a380-e852-44f6-b2d6-c40bd9d70be0"}}  

4. Delete

 curl -v -X DELETE http://localhost:9763/jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders/55a7a380-e852-44f6-b2d6-c40bd9d70be0  

 * About to connect() to localhost port 9764 (#0)  
 *  Trying 127.0.0.1... connected  
 > DELETE /jaxrs_starbucks_service/services/Starbucks_Outlet_Service/orders/55a7a380-e852-44f6-b2d6-c40bd9d70be0 HTTP/1.1  
 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3  
 > Host: localhost:9764  
 > Accept: */*  
 >   
 < HTTP/1.1 200 OK  
 < Date: Sat, 31 Aug 2013 19:16:41 GMT  
 < Content-Type: application/json  
 < Transfer-Encoding: chunked  
 < Server: WSO2 Carbon Server  
 <   
 * Connection #0 to host localhost left intact  
 * Closing connection #0