The WSO2 Identity server 4.5.0 and WSO2 Enterprise Load Balancer 2.1.0 is now released. So with this post I'm going to guide you with each and every and complete set of steps on how to cluster Identity server with two nodes and loadbalance it with ELB. And this setup is going to be in one Ubuntu machine.
Prerequisites:
1. Ubuntu 12.04 or a Linux similar platform
2. Java 1.6 or above (I'm using 1.7.0_25 64-bit)
3. Mysql (I'm using Ver 14.14 Distrib 5.5.32)
Let's go through the steps on how to cluster IS
1. Pointing to same User store and User Management db.
Being a cluster of IS, every node need to share the same User Store and User Management databases. So that every request come through load balancer will be served by any node similarly.
For User Store you can use any LDAP as Active Directory, Apache DS, OpenLDAP etc. Here I'm using the JDBC userstore which is supported by Identity Server to use a relational database as the user store.
I'm using mysql database for this.
Create a new database
$ mysql -uroot -proot
mysql> create database is_user_store;
Populate the tables needed. For that you can use the db scripts in the dbscripts directory in Identity Server home.
${IS_HOME}/dbscripts/mysql.sql
${IS_HOME}/dbscripts/identity/mysql.sql
${IS_HOME}/dbscripts/service-provider/mysql.sql
mysql> source mysql.sql;
mysql> source identity/mysql.sql;
mysql> source service-provider/mysql.sql;
Now you have to use this database in the configurations. For the User Store you have to add the following configurations;
In the;
${IS_HOME}/repository/conf/datasources/master-datasources.xml
Have the configuration
<datasource>
<name>WSO2_USERSTORE_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UserStoreDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/is_user_store</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
And that jndi database reference have to be used in the user-mgt.xml as to use it as the userstore
${IS_HOME}/repository/conf/user-mgt.xml
<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
<Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>
<Property name="ReadOnly">false</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="IsEmailUserName">false</Property>
<Property name="DomainCalculation">default</Property>
<Property name="PasswordDigest">SHA-256</Property>
<Property name="StoreSaltedPassword">true</Property>
<Property name="ReadGroups">true</Property>
<Property name="WriteGroups">true</Property>
<Property name="UserNameUniqueAcrossTenants">false</Property>
<Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>
<Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
<Property name="UsernameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$</Property>
<Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>
<Property name="RolenameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\<>,\'\"]{3,30}$</Property>
<Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>
<Property name="UserRolesCacheEnabled">true</Property>
<Property name="MaxRoleNameListLength">100</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="SharedGroupEnabled">false</Property>
<Property name="SCIMEnabled">false</Property>
<Property name="dataSource">jdbc/WSO2UserStoreDB</Property>
</UserStoreManager>
And for that to be the Primary userstore of the IS that need to be on top in the user-mgt.xml as the first mentioned userstore configuration. Every other userstore configuration you specify will be secondary userstores with a domain name.
and this has to be done in both of the IS nodes.
Now lets create another db and set it as the User Management db. Run the scripts to populate tables and refer it in the master-datasources.xml as following.
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UserMgtDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/is_user_mgt</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
You have to refer this in the user-mgt.xml as in following.
<Configuration>
<AddAdmin>true</AddAdmin>
<AdminRole>admin</AdminRole>
<AdminUser>
<UserName>admin</UserName>
<Password>admin</Password>
</AdminUser>
<EveryOneRoleName>everyone</EveryOneRoleName> <!-- By default users in this role sees the registry root -->
<Property name="dataSource">jdbc/WSO2UserMgtDB</Property>
</Configuration>
Now the User management related configurations are done. The next step is sharing the config and governance registries among each IS nodes. That is called Registry Mounting.
2. Registry Mounting
Let's create a database to be used in the mount configurations so that it will be the same repository for both IS nodes to share as the Config and Governance collections. That will be called the Mounted db. However for each IS node, the local registry will still be pointed to each and every node's H2 database. That configuration is already there in the master-datasources.xml. That need not to be changed.
You'll have to create a database in mysql and populate the tables. Let's see how it will be refered in the configurations.
First in the;
${IS_HOME}/repository/conf/datasources/master-datasources.xml
<datasource>
<name>WSO2_MOUNT_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2ISMountDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/is_mount_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
And now this jndi database reference will be used in both nodes as in;
${IS_HOME}/repository/conf/registry.xml
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>
<dbConfig name="mounted_registry">
<dataSource>jdbc/WSO2ISMountDB</dataSource>
</dbConfig>
<handler class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyHandler">
<filter class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyMediaTypeMatcher">
<property name="mediaType">application/xacml-policy+xml</property>
</filter>
</handler>
<remoteInstance url="https://localhost:9443/registry">
<id>instanceid</id>
<dbConfig>mounted_registry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/nodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Now let's look at the clustering configurations.
3. Clustering of IS nodes
We need to cluster two IS nodes as management nodes, and load balance it with WSO2 ELB.
First let's look at Identity Server configurations.
This is for the IS node 1.
${IS_HOME}/repository/conf/axis2/axis2.xml
There is a section as "clustering". You have to find the place and edit as following.
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
enable="true">
<parameter name="membershipScheme">wka</parameter>
We have to specify a domain for Identity Server cluster. And this domain name need to be unique from other WSO2 product clusters. Even from ELB domain.
<parameter name="domain">wso2.is.chamara.domain</parameter>
<parameter name="localMemberPort">4001</parameter>
<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="mgt"/>
</parameter>
The next step is to define the well known member for IS nodes. (this will be kept as default)
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4000</port>
</member>
</members>
Here the well known member is WSO2 ELB. In the ELB in the file
${ELB_HOME}/repository/conf/axis2/axis2.xml
the property of localMemberHost will be kept as default.
<parameter name="localMemberHost">127.0.0.1</parameter>
and in the file
${ELB_HOME}/repository/conf/loadbalancer.conf
I will define group management port as 4000. This Hostname and the Port are the socket which each IS node will search for, as the well known member.
In the carbon.xml
${IS_HOME}/repository/conf/carbon.xml
<Offset>1</Offset>
<HostName>wso2.is.chamara.com</HostName>
<MgtHostName>wso2.is.chamara.com</MgtHostName>
This hostname will be the one that is used by IS cluster and the ELB. It need to be specified in the;
/etc/hosts
as;
127.0.0.1 wso2.is.chamara.com
In the carbon.xml we are specifying the deployment synchronizer configuration as well. The deployment synchronizer will be enabled for, each IS nodes to share the secondary user store configurations.
Here I'm using svn based deployment synchronizer. So you'll have to find your own svn server and specify a location. Or else you can use Registry Based Deployment Synchronizer
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://svnexample.wso2.com/svn/chamara450</SvnUrl>
<SvnUser>wso2</SvnUser>
<SvnPassword>wso2123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>
I'm going to run ELB on default HTTP/HTTPS ports which are 80/443.
So we have to specify those as proxy ports in the
${IS_HOME}/repository/conf/tomcat/catalina-server.xml
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="80"
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
In the IS node 2 the configurations are almost same as for node1. The only differences that need to be changed are; in;
${IS_HOME}/repository/conf/axis2/axis2.xml
<parameter name="localMemberPort">4002</parameter>
and in;
${IS_HOME}/repository/conf/carbon.xml
<Offset>2</Offset>
4. Sharing of same keystores in IS cluster and ELB
In this step I'm going to explain how to use the same keystore in three servers and having certificates in the truststores. We can always use the default keystore wso2carbon.jks and it's public certificates. But in this case since we are using ELB in default ports, the servers need to have same keystore with the Common Name as the domain name (CN=wso2.is.chamara.com). So here are the steps to generate your own keystore.
$ keytool -genkey -alias iscarbon -keyalg RSA -keysize 1024 -keypass iscarbon -keystore iscarbon.jks -storepass iscarbon
What is your first and last name?
[Unknown]: wso2.is.chamara.com
What is the name of your organizational unit?
[Unknown]: users.wso2
What is the name of your organization?
[Unknown]: wso2
What is the name of your City or Locality?
[Unknown]: Colombo
What is the name of your State or Province?
[Unknown]: Western
What is the two-letter country code for this unit?
[Unknown]: 94
Is CN=wso2.is.chamara.com, OU=users.wso2, O=wso2, L=Colombo, ST=Western, C=94 correct?
[no]: yes
Here for local testing purposes it is ok to use self signed certificate. If this is going to be used in production the public certificate need to be signed by Certificate Authority.
Now export the public certificate and import it to the server trust store. Here it need to be imported in two IS nodes and the ELB.
$ keytool -export -alias iscarbon -keystore iscarbon.jks -storepass iscarbon -file iscarbon.pem
$ keytool -import -alias iscarbon -file iscarbon.pem -keystore client-truststore.jks -storepass wso2carbon
Now we need to change the configuration files to use this new keystore as the server keystore. For that following locations needs to be updated accordingly.
identity.xml: <Location>${carbon.home}/repository/resources/security/iscarbon.jks</Location>
identity.xml: <Password>iscarbon</Password>
security/cipher-text.properties: Carbon.Security.KeyStore.Password=[iscarbon]
security/cipher-text.properties: Carbon.Security.KeyStore.KeyPassword=[iscarbon]
carbon.xml: <Location>${carbon.home}/repository/resources/security/iscarbon.jks</Location>
carbon.xml: <Password>iscarbon</Password>
carbon.xml: <KeyAlias>iscarbon</KeyAlias>
carbon.xml: <KeyPassword>iscarbon</KeyPassword>
axis2/axis2.xml: <Location>repository/resources/security/iscarbon.jks</Location>
axis2/axis2.xml: <Password>iscarbon</Password>
axis2/axis2.xml: <KeyPassword>iscarbon</KeyPassword>
axis2/axis2.xml: <Password>iscarbon</Password>
axis2/axis2.xml: <Location>repository/resources/security/iscarbon.jks</Location>
axis2/axis2.xml: <Password>iscarbon</Password>
axis2/axis2.xml: <KeyPassword>iscarbon</KeyPassword>
axis2/axis2.xml: <Password>iscarbon</Password>
5. Configuring the Elastic Load Balancer
We are done with configuring the IS nodes. Lets look at how to configure the ELB
${ELB_HOME}/repository/conf/axis2/axis2.xml
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="domain">wso2.chamara.lb.domain</parameter>
this is a individual port unique from is nodes' localMemberPort
<parameter name="localMemberPort">4100</parameter>
For changing Transport ports (Proxy ports in IS nodes)
<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
<parameter name="port">80</parameter>
<transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">
<parameter name="port" locked="false">443</parameter>
In the file
${ELB_HOME}/repository/conf/loadbalancer.conf
Remove all the services entries and add the following.
identity {
domains {
wso2.is.chamara.domain {
tenant_range *;
group_mgt_port 4000;
mgt {
hosts wso2.is.chamara.com;
}
}
}
}
In the ELB also you need to specify keystore configurations as above, in order to use the iscarbon.jks. Here you don't have add configure or add any identity.xml.
6. Running the cluster
Now we have come to the last part. First run the ELB. For run that you need to be the superuser of that computer since we are running ELB on default ports.
{ELB_HOME}
Then run two IS servers.
{IS_HOME}
You will see the following logs in ELB while IS nodes joining ELB.
[2013-09-04 08:21:15,211] INFO - HazelcastGroupManagementAgent Member joined [ad2e310c-bf27-4bf3-9e03-afbe2117b6dd]: /127.0.0.1:4001
[2013-09-04 08:21:18,275] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:21:18,276] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster
[2013-09-04 08:24:59,428] INFO - HazelcastGroupManagementAgent Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002
[2013-09-04 08:25:02,493] INFO - MemberUtils Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:25:02,494] INFO - HazelcastGroupManagementAgent Application member Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true joined application cluster
And also while IS node joins each other following log will be appeared.
Node 1
[2013-09-04 08:22:15,083] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
[2013-09-04 08:22:23,314] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8228ms
[2013-09-04 08:22:23,365] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [bb0a0482-aea0-4895-a2f1-68ee721a38a5] - Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:22:23,374] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:24:59,430] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [3db96e0f-d4e9-4c45-bbdf-6331682a61cd]: /127.0.0.1:4002
[2013-09-04 08:25:01,494] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
Node 2
[2013-09-04 08:24:53,226] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
[2013-09-04 08:25:01,451] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8223ms
[2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4001, HTTP:9764, HTTPS:9444, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:25:01,481] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [3db96e0f-d4e9-4c45-bbdf-6331682a61cd] - Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
[2013-09-04 08:25:01,492] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:127.0.0.1, Remote Host:null, Port: 4002, HTTP:9765, HTTPS:9445, Domain: wso2.is.chamara.domain, Sub-domain:mgt, Active:true
Now you can access the IS cluster Management Consoles using following url;
https://wso2.is.chamara.com/carbon/
Login using the default admin user {admin:admin}
