A Framework for Differential Quality of Service Infrastructure for a Java Application Server 

RAJESHWARI GANESAN 
 INFOSYS TECHNOLOGIES LIMITED
 
& 
VARUN CHHABRA
SEIBEL SYSTEMS 

Abstract: - Most Internet-sites are subjected to a huge portion of load from casual visitors who use and view free content, to an extent that the performance of paid users often degrades. Giving such a situation can we device schemes without increasing infrastructure, such that the requests can be distinguished at application-level i.e. based on user type or business functionality or both? Currently, the application servers that host business logic do not have differential Quality of Service (QoS) support. With this in mind, we build a prototype QoS infrastructure on the open-source application server JBoss 2.4. The paper will describe the architectural components in building the differential QoS infrastructure present the appropriate theoretical background and empirical observations with the QoS infrastructure. This idea can be effectively applied in the Enterprise context to give a quantifiable performance boost in-terms of throughput and response times for preferred request classes.  

Key-Words: - Differential Quality of Service, Application-level scheduling 

1	Introduction
Consider an Internet-based e-Commerce site that has paid users and offers a limited portion of its features for free use. It is common phenomenon that such sites are subjected to a huge portion of load from casual visitors who use and view free content, to an extent that the performance of paid users often degrades. Giving such a situation, 
a)	Can we device schemes without increasing hardware/software infrastructure, such that the requests can be distinguished at application-level (i.e. based on user type or business functionality or both) to boost the performance of certain types requests i.e. differential QoS? 
b)	Can we additionally regulate the performance of these requests types in a predictable and controlled manner, instead of a strict priority processing of request? 
These requirements are applicable for many Internet systems where the request arrivals are chaotic or bursty in nature and a considerable portion of it comes from casual visitors. For these situations, one may have to look for optimal ways to use the existing infrastructure.
QoS [1,2,3] can be addressed by classifying the requests into different priority classes based on factors such as user type or business functionality. Further, these different priority classes will be serviced in manner to achieve differential QoS in-terms of the system performance i.e. throughput and response times of the request classes. There are two options to implement the differential QoS infrastructure. In the first type, the requests are assigned priority and are then routed to different processing elements in the infrastructure. With the second type, the requests after prioritization are scheduled to determine the order in which they would be processed by the infrastructure. Here our focus is on the latter option.
For setting the prioritization based on the user type or business functions residing in the Application server, the differential infrastructure would be required in the application server. Currently, none of the current Application servers have differential QoS support. With this in mind, we build a prototype infrastructure based on scheduling on the open-source J2EE application server JBoss 2.4 [4]. 
We will in this paper describe the components in building the differential QoS infrastructure based on scheduling of requests. We will also present some appropriate theory behind the concepts and show some empirical observations with the infrastructure on JBoss 2.4. Though we consider the application server in isolation, the idea can be effectively applied in the enterprise context to give a quantifiable performance boost in-terms of throughput and response times for preferred request classes. The next section will briefly introduce the reader to the JBoss 2.4 architecture.

2	JBoss Architecture
JBoss is an open source application server that only host an EJB container. The client calls to JBoss application server are made using JRMP, SOAP  or IIOP protocols. The current architecture of the JBoss provides different entry points called the ContainerInvoker for each of these protocols to abstract the specific protocol level details. The server hosts the components (the EJB beans) that are responsible for execution of the business logic. Each type of bean has a container instance associated with it that handles services such as its caching policy, security, transaction, persistence and so on. Thus, for an example let us consider an Internet digital library that implements two types of EJB’s - the MemberBean that is invoked for member creation, deletion, updating of member related information; and an SearchBean that retrieves documents from the digital library. There would be two container instances, one for each of them. All the client requests that invoke a particular bean type access the same instance of ContainerInvoker that in turn forwards the call to its the corresponding container. 
To understand the call structure we take the specific case of RMI call and trace it through the container. 
-	A remote call from an RMI client enters the JRMPContainerInvoker in the form of a marshalled object that encapsulates the semantics of calls made from clients. The marshalled object is unpacked to reveal the object ID, the method invoked, security credentials, principal of the caller (identity) and the transactional context. 
-	The JRMPContainerInvoker delegates the call to the container. The container forwards the call to various interceptors that manage the services such as transactions, security and persistence for the call. (Using the two deployment descriptors for every bean --- the standard ejb-jar.xml and application server specific extensions in jboss.xml, the behavior of the container can be tweaked; we will soon see how this can be useful.) The container creates or acquires a bean instance.  When all the interceptors have been passed through, the container invokes the business methods on the bean instance. 
-	The results are obtained by the container and sent back to the client through the JRMPContainerInvoker in the form of a marshalled object. 
JRMPContainerInvoker serves as a single point of entry for all requests across all clients for a single bean type for RMI calls. This observation paves the way for building a prototype infrastructure on JBoss to effect prioritization of requests (coming as RMI call) by scheduling. The next section describes at a conceptual level the different components in a prioritization-cum-scheduling architecture.

3	 Conceptual Architecture of QoS Scheduling Layer

The architecture is built around with a collection of subjects, a request basket and an observer (see Fig.1) which interact as follows: 
 Fig. 1 The conceptual scheduler architecture

The Collection of Subjects represents logged-in clients; each subject pushes the requests from its client into the request basket. For each request, the subject waits for a signal from the observer when its turn arrives, indicating that it can proceed with the processing of that request. 

The Request Basket is a collection of data structures that store pending requests. In the digital library example, it could consist of a collection of queues, each containing requests of different priority. Each incoming request is analyzed, its priority assessed, and assigned to an element of the data structure (queue) accordingly. The priority assessment is based on the differential QoS requirements based on user type, business logic or a combination of those.

The Observer is responsible for observing the request basket and implementing the request scheduling policy. This policy can be either deterministic or probabilistic. A deterministic strict priority rule would mean that the observer is always configured to pick a particular class of high priority request each time from the basket in preference to the pending low priority requests. The popular shortest remaining processing time policy also falls in this category. The deterministic policies have an important disadvantage of request starvation, which can be overcome by a randomized scheduling policy. 

The prototype implementation follows a randomized approach. We apply the policy in a manner that ordering of requests for execution happens in a probabilistic manner. To understand consider two requests of type A and B that have to be picked for processing by the server in the ratio of 3:2.  This can be simulated by generating a random number say between 1 to 5, where for random values of 1,2,3 request A is picked, and for random values of 4 or 5 request type B is picked for processing.  The ordering will happen so that the differential QoS infrastructure will meet the expected performance levels of the different request classes. 
During an actual implementation, the number of request baskets, the number of different priority queues within each request basket, the number of observers per request basket, and the hierarchy of all these components are all mainly dependent on the requirements of the scheduler to be built. Our prototype infrastructure will do two-level request prioritization of requests at method level. We move to digging out details of how this requirement was implemented in JBoss.

4	Implementation
The JBoss request forwarding mechanism was aligned to meet the conceptual requirements of the scheduler. However we faced several challenges, which we list here:
a) Where to insert the scheduling layer?
At the beginning, it seemed that there were various architectural choices to decide the point at which the scheduling layer would be “inserted”, i.e., where the client requests would be assigned differential priorities and pushed into the queues. However, probing a little further clearly eliminated certain possibilities. One of those options was in creating a new entry point before it hits a particular ContainerInvoker. However, one of our objectives was to ensure that the syntax of the calls made by the client remain the same with or without the scheduling infrastructure over the application server. Hence this option was eliminated. The other alternative was to introduce a new interceptor to the container that would do QoS scheduling. Some analysis indicated that the sources impacted by adopting this approach were far too many and such an infrastructure could also not abstract away the specifics of the container implementation. This certainly seemed like a more complex and less extensible implementation, with no extra benefits or advantages. Finally, we decided that the optimal solution was to have the layer inserted into the container invoker. 
The observer was introduced as a startup service and the infrastructure was built for JRMP protocol. With the modified setup, the request would no longer make direct calls from JRMPContainerInvoker to the container. Instead, each client accessing JRMPContainerInvoker would wait there till the observer notified it to “proceed” with the call. 
b) With one invoker for a type of bean, so how to distinguish the different requests of the bean?
There was only one instance of JRMPContainerInvoker for a given bean type. Hence we needed to introduce a mechanism whereby, after entering JRMPContainerInvoker, each request would create an instance that serves as a proxy for it. This instance would push its reference on to the request queue and would enable the observer to identify it. Finally, the proxy instance would return control back to JRMPContainerInvoker to continue with rest of the processing as usual. This proxy was implemented as the ProxyClient in the modified JBoss. 
c) How to minimize the overheads of the scheduling layer with introduction of asynchronous communication?
The biggest challenge of the infrastructure was to minimize the overheads of the scheduling layer itself. This was primarily dependent on the implementation strategy used for managing concurrency and asynchronous communication between the active instances of ProxyClient and the observer thread. We did two thing –
(a)	The ProxyClient instance executed a synchronized block that pushed the request in the queue. While executing the block, it would grab an internal lock (called a mutex) on a java object to ensure that no one else is able to execute the code. The usage of synchronized reduced the concurrency of the system. In our case, the instances of ProxyClient were contesting to update the request queue, and the observer thread was simultaneously checking it in a synchronized fashion. We noticed that this was quite a bit of overhead. The solution we resorted to was to create a secondary request basket for exclusive use by the observer. Certain conditions would trigger the elements to be moved, from the primary, to the secondary request basket. This change improved the concurrency of the system considerably( see Code Listing-1) . 
(b)	The second issue addressed was related to the implementation of asynchronous communication between the observer and the instances of the ProxyClient. We realized that the only acceptable and elegant solution was to use the wait/notify mechanism of Java that allowed them to signal each other. The wait() operation of the wait/notify mechanism uses almost zero processor resources. This was preferable to sleep() that used processor resources continuously for checking for changed value, and consequently leading to problems of missed status. In the prototype, the wait/notify mechanism was used in conjunction with variables (e.g. Main.notifyflag, goahead). These variables were set by one and checked by another. This helped the ProxyClient instances to eliminate redundant notification to the observer. Also, it was specially useful in case of notifyAll() by the observer, when all instances of the ProxyClient would come out of their wait state. However, only the specific instance that had to proceed with execution had its variable goahead ‘turned on’ by the observer. This way, the remaining instances of ProxyClient would re-enter the wait state (see Code –Listing 2).
The next section describes the prioritization rule specification and the policy used for scheduling.

5	Scheduling Rules and Policies
The Infrastructure was designed in a manner such that the prioritization rules could be specified as a deploy time option. As mentioned earlier, the current architecture provides jboss.xml, where any extensions to the functionality could be placed. Since every bean had one of those files associated with it, any bean level QoS requirement could be mentioned there. Correspondingly, JBoss code could be made to pick that up. However, our prototype intended to do overall scheduling, and so we created a global XML deployment descriptor file called the qos.xml.

<?xml version="1.0" encoding="UTF-8"?>
<jboss>
	<QoS> 
       <ejb-name>SearchBean</ejb-name>
       <method>PaidContent</method>
   	 <priority>5</priority>
     </QoS> 
</jboss>

In qos.xml we have consider the example where SearchBean that has 2 methods - PaidContent() accessed by paid users for searching and FreeContent() accessible for guest users. Let us assume for simplicity that these are the only two methods in the entire application. We want to ensure that PaidContent() is assigned higher priority over FreeContent(). Further, we also specify here that the response times of PaidContent() has to be 5 times scaled-up as compared to FreeContent(). 

We will examine shortly, how such throughput or response time guarantees can be achieved by ordering of requests for scheduling appropriately.

Interactive Response time law relates the performance metrics such as throughput, response time and think times with the number of users in a closed system. The relationship is,
N  = X * ( R + Z) where,
N is the number of users
X is the throughput
R is the response time
Z is the think time
Example. Let us suppose that we have a user base of Npaid and Nguest users. We want to achieve a response times of Rpaid to be under all load conditions be 5 times lower than Rguests. Assume that the average think times are extremely small as compared to response times that we ignore it for both types of users. Using the interactive response time equation we know that the throughputs ratio has to be:
Xpaid / Xguest = ( Npaid  / Nguest ) * (Rguests / Rpaid)
A special case would be where, the number of users Npaid  = Nguest, this would translate to Xpaid / Xguest = 5.
This will be the QoS requirement with priority set as ‘5’ in the deployment descriptor. Thus, here a throughput ratio of 5 translates into a response time ratio of 5 too: the high priority requests experience a response time that is one fifth that of that of the low priority ones. (Notice in fact that this reasoning does not depend on the relative size of the requests!)

As mentioned previously, we opted to have a randomized scheduling policy to avoid the problem of request starvation. Thus in the above example the implementation ensured that the probability of high priority request being picked up was 5 times higher that picking up a low priority request. This means that the throughput of the high priority requests would be boosted by 5 times compared to that of the low priority ones. This has a direct consequence on the response times. This can be understood by applying response time law [5], which we explain in side bar below.

6	Test Results & Inferences
Experiments were conducted by setting up a virtual client generator program on one machine and the JBoss application server, for cases of with and without the QoS scheduling infrastructure, installed on another machine. The load generator spawned a fixed number of virtual users Npaid and Nguest to represent users who would fire requests to methods PaidContent()  and FreeContent() respectively. Tests were modeled in a manner where each user would fire the corresponding request, get a response and fire again (no think times). The tests were conducted under steady state conditions and the average response times, throughputs (request/second) were noted for the two classes of users in original JBoss, and then with modified JBoss with scheduling infrastructure. Fig.2 summarizes the results of the experiment, which had their relative priorities as 5:1 in qos.xml. 

 PaidContent()	Throughput  (request/second)	Average Response times (seconds)	
No Scheduling	26.31	0.038	
Scheduling with QoS priority set as 5.	41.67	0.024	
FreeContent() 
	Throughput  (request/second)	Average Response times (seconds)	
No Scheduling	26.31	0.038	
Scheduling with QoS priority set as 5.	8.20	0.122	
Relative ratio of response times		5.08	
Relative ratio of throughput	5.08		
Fig.2  Performance measures with and without scheduling infrastructure
Please note that the methods were identical. The results confirm that their relative response time ratio and throughput was close to 5. We confirmed this under varying conditions of load, and avoided very low load conditions where the scheduling efficiency was expected to be low.  The ratio of response times could however vary if the requests were fired by different proportion of clients, even though the relative throughput ratio is 5. Second part of the analysis was to determine the absolute improvement in response times of the high priority requests with scheduling. To explain, we define a High Priority Improvement Factor (HPIF) that is a percentage improvement of response time of the high priority request with QoS scheduling. Higher HPIF indicates higher efficiency of scheduling, and, as you would expect, is dependent on the ratio of the sizes of the requests being prioritized.  We quantify the sizes of requests by taking into account the service demands [5] of the system resource. Here the operations were processor intensive and thus we measured their service demand on the processor of the hardware box that hosted the application and the application server. Fig.3 shows the results of the experiment conducted with 30 concurrent users with two requests whose ratios of the request service demands (high versus low priority) were altered, and the absolute response time of the high priority request was then noted.  It was noticed that with increase in this ratio of service demands from 0.5 to 2, the HPIF decreased. In other words, while the absolute improvement for the high priority request was only 22% with ratio of 2, it increased to 56% with a ratio of 0.5

Fig.3. HPIF vs. Ratio of Service Demands

7	Conclusion
Traffic to most web sites is highly bursty, and the usual approach is to overprovision servers to handle peak loads. However prioritization of requests can be good way to utilizing server resources without over provisioning. The paper has described the architecture, backed it with some theoretical foundation and empirically proved the efficiency of the setup.
 References:
[1] J. Almeida, M. Dabu, A. Manikutty and P. Cao, “Providing differentiated levels of service in Web content hosting,” in Proc. First Workshop on Internet Server Performance, June 1998.
[2] L. Eggert and J. Heidemann, “Application-level differentiated services for Web servers,” World Wide Web Journal, 2(3), pp 133-142, August 1999.
[3] R.Ramaswamy, “Probabilistic application-level connection scheduling in Web servers,” IEEE Distributed Systems Online, Vol.2, No.8, 2001
[4] JBoss: Professional Open Source,at URL: http://www.jboss.org,
[5] Mitrani I., “Queuing Theory,” in J. McDermid (Ed.) Software Engineer’s Reference Book, Butterworth-Heinemann, 1991.
Code-Listing 1

public class ProxyClient{
...
public void insertNotify(){
synchronized (Main.obj) {
...
			if (Main.notifyflag == true){
				try {
					Main.notifyflag = false;
					Main.obj.notify();
					} catch (Exception e) {}
				}
	...
     } //end of insertNotify

public void waitForSignal() throws Exception{
			goahead = false;
			insertNotify();			
			while (!goahead)
			{
				synchronized (Main.obj3)
				{
					Main.obj3.wait();
				}
			}
		}//end of waitForSignal
}// end of ProxyClient

 Code-Listing 2

public class ObserverThread extends Thread{

...
public void invokeProxyClient(ProxyClient object){
				object.goahead = true;
				synchronized (Main.obj3){
					Main.obj3.notifyAll();
				}
}

public void run(){
		...	
			synchronized (Main.obj)
				{
				try {
					Main.notifyflag = true;
					Main.obj.wait();
					Main.notifyflag = false;
				} catch (Exception e) {}
			}
...
} // end of observer