DirkReske.de

Overview

After my articles about setting up a glassfish cluster and enable session replication, its now time to put a load balancer in front of it.

A couple of times I’ve tried to get the Sun load balancing plugin working, but without success. This was because of  some operating system issues or other problems.
So after looking around the web, I’ve found some good solutions using apache and the mod_jk connector from the apache tomcat project.

Requirements

• Recent glassfish installation (I use v2.1.1).
• mod_jk tomcat connector.
• Apache tomcat server to get some files from (Use version 5.5.23, others won’t work).
• Apache commons logging
• Apache commons modeler

Installation

1. Configuring the glassfish cluster
   1. Copy the $CATALINA_HOME/server/lib/tomcat-ajp.jar to your $GLASSFISH_HOME/lib directory on each glassfish installation in your cluster.
   2. Also copy the commons-logging.jar and commons-modeler.jar to your $GLASSFISH_HOME/lib directory on each glassfish installation in your cluster.
   3. Now you have to define jvmRoute and com.sun.enterprise.web.connector.enableJK system properties for the cluster (this can be done from the domain administration server)

      asadmin create-jvm-options --target cluster1 "-DjvmRoute=\${AJP_INSTANCE_NAME}"
      asadmin create-jvm-options --target cluster1 "-Dcom.sun.enterprise.web.connector.enableJK=\${AJP_PORT}"

   4. Now you have to define the values for these properties for each instance running in the cluster (this can also be done from the domain administration server)

      asadmin create-system-properties --target instance1 AJP_INSTANCE_NAME=instance1
      asadmin create-system-properties --target instance1 AJP_PORT=8020

2. Configuring the apache web server
   1. Copy the mod_jk.so to a apache accessible directory.
   2. Copy the following lines to your apache main config file.

      LoadModule jk_module <path to your mod_jk.so>
      JkWorkersFile <path to your worker.properties (created later)>
      JkLogFile <path to your mod_jk.log>
      JkLogLevel debug
      JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
      JkOptions +ForwardURICompat -ForwardDirectories
      JkRequestLogFormat "%w %V %T"

   3. Copy the following lines to your virtual host configuration (or into the main config, if you don’t use virtual hosts).

      JkMount /* loadbalancer

   4. This will redirect all requests to your load balancer. (You can also use patterns like /*.jsp for only redirecting jsp file request to the load balancer or just any other sensefull pattern).
3. Writing your worker.properties configuration file.

   worker.list=loadbalancer
   
   # Set properties for instance1
   worker.instance1.type=ajp13
   worker.instance1.host=<host of instance1>
   worker.instance1.port=<AJP port of instance1>
   worker.instance1.lbfactor=50
   worker.instance1.connection_pool_size=10
   worker.instance1.connection_pool_timeout=600
   worker.instance1.socket_keepalive=1
   worker.instance1.socket_timeout=300
   
   # Set properties for instance2
   worker.instance2.type=ajp13
   worker.instance2.host=<host of instance2>
   worker.instance2.port=<AJP port of instance2>
   worker.instance2.lbfactor=50
   worker.instance2.connection_pool_size=10
   worker.instance2.connection_pool_timeout=600
   worker.instance2.socket_keepalive=1
   worker.instance2.socket_timeout=300
   
   worker.loadbalancer.type=lb
   worker.loadbalancer.balance_workers=instance1,instance2

4. Restart all cluster instances and the web server. Your load balancing should be working now.
   Each time a new session is creted by the glassfish, it appends its instance name to the JSESSIONID, so the load balancer can determine on which instance to route the request.
   If one instance crashes, the load balancer will notice this and redirects all requests to another instance. If you have enabled session replication, your old session will be resumed there.

Further reading

• enable session replication
• setting up a glassfish cluster
• mod_jk connector documentation

Overview

One of the high availability methods provided by glassfish, is session replication.
With session replication enabled, the server distributes session data across the cluster. If one cluster instances crashes, the session is still available on the other instances.

Requirements

• A recent glassfish installation (versions prior to 2.1.1 had some bugs).
• A configured cluster (Look here who to install a glassfish cluster).
• All cluster instances located in the same subnet. (later glassfish version should support different subnets)

Configure session replication

1. Add a <distributable /> tag to your web.xml

   <web-app>
     <distributable />
     ...
   </web-app>

2. Add following lines to your sun-web.xml

   <session-config>
     <session-manager persistence-type="replicated">
       <manager-properties>
         <property name="persitencerequency" value="web-method" />
       </manager-properties>
       <store-properties>
         <property name="persistenceScope" value="session" />
       </store-properties>
     </session-manager>
   </session-config>

   • persistenceType=”replicated” says, that the session should be replicated along the cluster.
   • persistenceFrequency=”web-method” says that the session state should be stored after processing a request, but before sending the response. (other values: time-based)
   • persistenceScope=”session” says that the whole session state should be stored. (other values: modified-session, modified-attribute)
3. Deploy application
   • using the web-interface: check the Availability checkbox.
   • using the cli: add –availabilityenabled=true parameter.

Notes

• Be sure that all classes that are going to be stored in a session implements the Serializable interface.
  If not, the server can’t serialize it and replication won’t work.
• As already mentioned, the instances have to be on the same subnet, otherwise session replication will not work.

Further reading

The replicated Persistence Type

Overview

After I looked around the web, I’ve found a few good howto’s about glassfish cluster configuration, but either they use only single machine clusters or they have some other disadvantages in my eyes. So I’ve decided to write my own article about this.

This article will describes how to set up a two machine cluster which consists of the domain administration server (DAS) and a “normal” node. Thereby the DAS also works as such a normal node.

This article uses Glassfish v2.1.

Before the installation

You have to ensure, that your network is configured properly, especially the dns system. If there are problems, there will be curious problems while setting all up. I’ve done this using the hosts file.

10.0.0.1   das-host
10.0.0.2   node2-host
10.0.0.x   nodex-host

The next thing is the firewall. The nodes communicating using a couple of ports. For setup and testing I’ve configured my firewalls to accept everything from all the other nodes, so there are no problems for the nodes to communicate with each other.

Setting up the DAS

Because the DAS server needs cluster support, we have to configure this first.
There are two ways to do this.

• install the glassfish server using:

  ant -f setup-cluster.xml

• activate cluster support using the admin console
  1. go the the admin console (e.g. http://localhost:4848)
  2. select from the common task: Enable Cluster Support

Now start up the server using:

asadmin start-domain domain1

In the next steps, I will create the cluster, then a node-agent and one instance running on the DAS server using the cli.

In most cases you have to specifiy your admin username and password. This can be done using the –user and –passwordfile parameters, but for simpler reading I will omit this. The server will ask you in this case.

1. Create the cluster:

   asadmin create-cluster cluster1

   This creates a cluster on the local server (the DAS) with the name cluster1.

2. Create the node agent:

   asadmin create-node-agent das-agent

   This will create the node agent for the current node.

3. Create the instance for the current node:

   asadmin create-instance --nodeagent das-agent --cluster cluster1 instance1

   This will create the instance instance1 associated with the node agent das-agent and the cluster cluster1.

4. Now start up the node agent and the associated instance:

   asadmin start-node-agent --syncinstances=true --startinstances=true das-agent

   • –syncinstances=true says that the node agent should synchronize its instances with the central repository of the DAS
   • –startinstances=true says the the node agent should start all its instances (instance1 in our case)

Now the domain administration server is up and running.
The first node is also running.

Setting up the second node

Now come to setup the second node.
Because the second node should only work a “computing” node, we do not need to install the whole application server, just a node agent.
I omit user and password parameters here too. You have to enter the DAS login informations when asked for.

1. Install the node agent:

   ant -f setup-cluster.xml create-node-agent -Ddas.host=das-host -Dnodeagent.name=node2-agent

   • -Ddas.host=das-host sets the hostname of the domain administration server
   • -Dnodeagent.name=node2-agent sets the name of the local node agent
2. Create the new instance:

   asadmin create-instance --host das-host --nodeagent node2-agent --cluster cluster1 instance2

   This will create the new instance associated with the node agent node2-agent and the cluster cluster1. The –host parameter specifies the hostname of the domain administration server.

3. Start the instance:

   asadmin start-node-agent --syncinstances=true --startinstances=true node2-agent

   Now the local node agent starts, synchronizes with the das server and starts all local instances (only instance2 in our case)

Now your cluster is running with two nodes. To add more nodes, just repeat the three above steps as much as you want.

Deploying applications to the cluster

For deploying applications to the cluster, just type on the das server

asadmin deploy --user admin --target cluster1 MyApplication.ear

This will deploy the application to the whole cluster.

For further reading

• Enable glassfish session replication
• create-cluster command
• create-node-agent command
• create-instance command
• start-node-agent command