Grinder Test Automation for the WebLogic Server

Todd Nichols, Anser Enterprise

Introduction

This is a description of Grinder source code modifications made to automate testing the effects of WebLogic server configuration changes. For Grinder users, this tutorial may prompt ideas for general-purpose Grinder feature requests. For Grinder developers, it discusses implementation techniques, which may be of interest, while not exactly fitting with plans to provide a Grinder API for test automation.

 

Todd Nichols

 

Part I User Interaction

Automation Tasks

Source code modifications and other techniques address these test automation functions:

 

  1. Inputting ranges of Grinder workloads and settings of WebLogic configuration parameters to use.
  2. Computing all permutations of possible values in workload and configuration setting ranges.
  3. Starting and stopping a sequence of tests.
  4. Distributing grinder.properties files with the number of Grinder processes and/or threads before each test.
  5. Changing one or more WebLogic server configuration parameters before a test.
  6. Re-starting the WebLogic server before a test, when required.
  7. Recording the number of Grinder processes and threads used for each test.
  8. Recording statistics presented in the Console Results Tab for each test.

 

Console GUI Modifications

 

This screenshot shows new controls that were added to the Console:

  1. Target Tab (similar to Script Tab, but with a separate file selector tree)
  2. Start test series (on the Action dropdown control after Stop Collection menu item)
  3. Stop test series (also on the Action dropdown control)

 

Target Tab

 

As with the Script Tab, users can select a root directory and a file in the Target Tab. These selections are independent of selections for the Script root directory and file made in the Script Tab. In the Target Tab, the user selects a XML file that contains the ranges of values for WebLogic configuration settings for the series of tests. In the Console picture, the selected file is “configTarget.xml”, whose format and origin is described below.  In addition, this file contains the range of processes and threads values to be inserted into a grinder.properties file and distributed to Grinder agents.

 

Test result files are created in the same directory that contains the configuration XML file. As shown above, those files are named InputParmNames.txt, InputParms.txt, StatsMean.txt, and StatsNames.txt. The formats for these files will be described below as well.

 

Start test series

 

Like the existing Start processes menu item under Action, Start test series initiates the agent’s workload. However, it initiates a series of tests instead of a single test. To proceed through all of the tests, the Collect samples forever/Stop after XX samples text box must be non-zero. Once this sample limit has been reached for a test, the test automation code re-configures the WebLogic server, distributes a newly configured grinder.properties file to the agent(s) and begins the next test.

 

Stop test series

 

Like the existing Stop processes menu item, Stop test series stops the agent workload but for the remainder of the test series.

 

Test Configuration XML File

 

  <? xml version="1.0" encoding="UTF-8" ?>

< Domain ConfigurationVersion =" 8.1.2.0" Name =" workshop">

   < Server AcceptBacklog =" 10,20,30,40" AdministrationPort =" 1" DefaultProtocol =" t3" DefaultSecureProtocol =" t3s" HttpdEnabled =" true" ListenAddress =" localhost " ListenPort =" 7001" Name =" cgServer " NativeIOEnabled =" false,true" ReliableDeliveryPolicy =" RMDefaultPolicy " ServerVersion =" 8.1.2.0" StdoutDebugEnabled =" false" StdoutSeverityLevel =" 32" TransactionLogFilePrefix =" ./logs/" TunnelingClientPingSecs =" 45" TunnelingClientTimeoutSecs =" 40">

      < WebServer AuthCookieEnabled =" true" LogFileName =" ./logs/access.log"     LoggingEnabled =" true" Name =" cgServer " />

      < COM Name =" cgServer " />

      < ServerStart Name =" cgServer " />

      < SSL Enabled =" true" IdentityAndTrustLocations =" KeyStores " ListenPort =" 7002" Name =" cgServer " />

      < Log FileMinSize =" 20000" Name =" cgServer " RotationType =" bySize " />

      < KernelDebug Name =" cgServer " />

      < KernelDebug Name =" cgServer " />

      < ServerDebug Name =" cgServer " />

      < ExecuteQueue Name=" weblogic.kernel.Default " QueueLength =" 65536"     QueueLengthThresholdPercent =" 90" ThreadCount =" 10,15,20,25,30" />

      < JTAMigratableTarget Name =" cgServer " UserPreferredServer =" cgServer " />

  </ Server >

  < Grinder FileName =" C:\grinder3_0\test1_script\grinder.properties" Processes =" 1" Threads =" 15,50,100,200" />

 

 

The partial listing above is from the “configTarget.xml” file shown selected in the Target Tab. It specifies the range of values for the Grinder workloads and WebLogic configuration settings as comma-separated integers, Booleans, etc. in the attribute value strings. Most of the file XML content is copied from a WebLogic config.xml configuration file for the app server to be tested. For the configTarget.xml file, single attribute values from the config.xml file are replace by multiple values. For example, AcceptBacklog=”30” is changed to AcceptBacklog=”10,20,30,40” and NativeIOEnabled=”false” becomes NativeIOEnabled=”false,true”.        

The <Grinder …/> element is added to the beginning of the config.xml-derived file to add specifications for the workloads that the Grinder agents will have. Here, the file, C:\grinder3_0\test1_script\grinder.properties, will be changed between tests so that its grinder.threads property will have values, 15, 50, 100, and 200.

 

The test automation code computes all permutations of these test values for the series of tests. See the InputParms.txt file below for an example of these permutations.

 

The order of appearance of an attribute in the XML file determines how fast the associated configuration settings will change for subsequent test in the series, just as bits to the right in a binary register toggle faster than bits to the left as you count up. In many cases when WebLogic configuration settings are changed, the app server must be re-started for the change to take affect. The test automation takes care of this re-start. Therefore, it’s best to place the <Grinder…/> element at the end, after all of the WebLogic configuration elements, to minimize the number of WebLogic server re-starts.

Test Series Results Files

For each test in the series, its configuration settings (input parameters for the test) and its ending statistics are appended to two tab-delimited files. The names of the input parameters and statistics are recorded as well. 

 

Input Parameter Files

 

The InputParmNames.txt file contains the names of Grinder workload and WebLogic configuration parameters for the tests as well as units of measure for charting purposes. For example below, “Grinder_Processes” is the name of an input parameter and its units of measure is “Agent Processes”. According to the configTarget.xml file above, its value in the next to last column of the InputParms.txt file discussed below will always be “1”.  To make charting easier, “false” and “true” values such as for psServer_NativeIOEnabled are converted to “0” and “1” as in column 3 in the InputParms.txt file.

 

psServer_AcceptBacklog  Messages

psServer_NativeIOEnabled     0/1 (No/Yes)

weblogic.kernel.Default_QueueLengthThresholdPercent  Percent

weblogic.kernel.Default_ThreadCount   Threads

Grinder_Processes  Agent Processes

Grinder_Threads    Agent Threads

 

The InputParms.txt file contains the values for each of the input parameters named in the InputParmNames.txt file above in columns. The first column contains the run number of the test. Here’s a partial listing of this file’s contents.

 

1    10   1    90   10   1    300

2    10   1    90   10   1    400

3    10   1    90   10   1    500

4    10   1    90   12   1    300

5    10   1    90   12   1    400

6    10   1    90   12   1    500

7    10   1    90   14   1    300

8    10   1    90   14   1    400

9    10   1    90   14   1    500

...

31   10   1    90   30   1    300

32   10   1    90   30   1    400

33   10   1    90   30   1    500

34   15   1    90   10   1    300

35   15   1    90   10   1    400

36   15   1    90   10   1    500

37   15   1    90   12   1    300

38   15   1    90   12   1    400

39   15   1    90   12   1    500

...

           

Statistics Files

 

The StatsNames.txt file contains the names of statistics presented in the Grinder Accumulated Test Statistics table in the Console Results Tab at the end of each test in the series. Here is an example:

 

Successful tests   Grinder   transactions

Errors    Grinder   transactions

Mean timeGrinder   milliseconds

TPS  Grinder   transactions/second

Peak TPS  Grinder   transactions/second

Mean response length    Grinder   bytes/transaction

Response bytes per second    Grinder   bytes/second

Response errors    Grinder   transactions

Mean time to resolve host    Grinder   milliseconds

 

This file is tab-delimited. The first column contains the name of the statistic. The word “Grinder” is inserted in the second column for charting purposes. The last column contains the unit of measure for the Grinder statistics, also used for chart scaling and labeling the Y axis.

 

The StatsMean.txt file contains the measured values for each of the statistics named in the StatsNames.txt file. There will be as many rows in this file as there are in the InputParms.txt file. Similarly, the first column contains the run number of the test.

 

1    1686102  593040.9115  14500     593000    0    1.68

2    16290    956040.5120  16000     647000    0    1.48

3    170929   12100     40.6102  17400     705000    0    0.427

4    178210   576042.379.014700     622000    0    0.788

5    17152    884041.5107  15900     660000    0    0.996

 

The contents of the input parameter and statistics files can be imported in to a spreadsheet for charting. Also, we’ve built a Java charting applet that will download these four files and chart the results in a browser. It requires the Java 1.5 plug-in. To see an example of these test results using the Java applet please visit, http://www.anser-e.com/run6/Run6a.html.

Part 2 Implementation Details

 

Grinder Version 3.0 beta 24 was used for these source code modifications. The following sections describe some development experiences, a functional overview and a sequence diagram and further discussion of the test series flow.

Grinder Code Reuse

I started reusing source code in the net/grinder/console/swingui directory and the ConsoleUI class in particular. Some new Grinder test series objects were instantiated from existing classes, such as the ProcessControl, FileDistribution, and EditorModel classes below. Other Grinder classes were cloned and modified somewhat. For example FileTreeSUT for selecting the app server configuration file was cloned and modified from the existing FileTree class. I thought this would be the fastest approach and provide a level of isolation of new and existing functions. From the ConsoleUI.java file:

 

public final class ConsoleUI implements ModelListener {

 

 private final ProcessControl m_processControl;

 private final ProcessControl m_process SUTControl;

 

 private final FileDistribution m_fileDistribution;

 private final FileDistribution m_fileDistributionConfig;

 

 private final EditorModel m_editorModel;

 private final EditorModel m_editorModel Config;

 

 private final FileTree m_fileTree;

 private final FileTreeSUT m_fileTreeConfig;

  …

 

However,   I found that I needed to go deeper into the code and create more similar classes and files such as my ‘ConsoleSUTCommunicationImplementation.java’ file derived from ‘net/grinder/console/ConsoleCommunicationImplementation.java’.  I ended up going as far as modifying the ‘net/grinder/console/Console.java file, added lines such as these bolded ones:

 

public class Console {

 

  private final ConsoleCommunication m_communication;

  private final ConsoleCommunication m_ SUTcommunication;

  ...

  public Console() throws GrinderException {

 

 

    final ConsoleProperties properties =

      new ConsoleProperties(resources,

                            new File(homeDirectory, ".grinder_console"));

 

    final ConsoleOtherProperties otherProperties =

      new ConsoleOtherProperties(resources,

                            new File(homeDirectory, ". grinder_console2"));

 

  ...

    m_userInterface =

      new ConsoleUI(m_model,

                    m_communication.getProcessControl(), m_SUTcommunication.getProcessControl (),

                    fileDistribution, fileConfigDistribution ,

                    agentStatus);

 

This went on for longer that I would have liked, but now that it’s done I can live with the changes. It works for what I need. But I’m sure there is a more elegant approach for plug-in’s that would first involve some architecture changes. 

Functional Overview

 

Inputting XML Configuration Data

 

The XML configuration template file selected in the Target Tab (described in Part 1) is parsed using the DOM API from these packages:

 

import org.w3c.dom.*;

import org.xml.sax.*;

 

Currently, only a subset of all attributes in a WebLogic config.xml file is supported. I can support others with only small amounts of additional coding as I need to test variations of other configuration parameters.

 

Creating and Executing Configuration Permutations

 

As XML nodes are parsed and attributes are recognized, “ExecutableJMXCommand” objects are created and added to a list. The ExecutableJMXCommand class has members for WebLogicMBean objects and Method objects for verifying and setting WebLogic configuration parameters using the WebLogicMBean objects. During this discovery phase, the ServerMBean is found using MBeanHome’s getMBean method and other ConfigurationMBean’s are found from it. To complete the JMX command objects, the ranges of configuration values, which these JMX commands will use, are store with them as well.

 

There are <Grinder…/> elements in the XML file as well that are processed. They cause similar executable command objects to be created, but these commands modify a grinder.properties file, primarily changing the grinder.threads property value that determines a test’s workload.

 

Execution of JMX commands happens in a separate thread spawned by a “Start test series” menu item event. Once the entire XML file has been processed, the first JMX command object on the list is executed for the first configuration value in its range of values. The configuration value is recorded in an Input Parameters file. Then, other JMX command objects on the list are executed in turn until there are no more on the list. Their configuration values are appended to the same line in the Input Parameters file for a given test run. Going down this list is done in a recursive method. Once the bottom of the list is reached, the recursive method returns to its previous calling. There, it moves horizontal across to the next configuration value in a range for the JMX command object at that level. If there is another value in the range, the JMX Command at that recursion level is executed again with its new value and we go back down the recursion path. Each time we go back down, the JMX command that exhausted its range of values before starts over with its first configuration value.

 

Each time at the bottom of the recursion path, when all configuration parameters for a test have been set, the thread re-starts the WebLogic server. That is, if a re-start flag has been set somewhere along the way. This flag is set for any configuration change performed that requires the app server to be re-started. The thread waits until a connection has been re-established with the app server and then it signals a separate thread, which starts the Grinder agents.

 

Whenever there are no more values looking horizontally across configuration values in a range, we move back up the recursion path. When we get back to the top of the recession path, all permutations of configuration parameters and grinder workloads, and therefore all tests in the series, have been completed.

 

Starting Grinder Agents

 

Existing Grinder Console events are used to start a test for the new WebLogic server configuration and new Grinder agent workload. The separate thread mentioned above first fires the distribute files event. This passes the grinder.properties file with the new grinder.threads value to the agent(s). Then, the thread fires the start processes event to start the test and waits until it finishes.

 

A test in the series is finished when the specified number of samples is reached, as shown in the Console UI while the test is running. A listener callback signals this thread to wake up. At that point statistics from the CumulativeStatisticsTableModel object are read and written to the Statistics text file. Finally, this Grinder Process Control thread signals the JMX Command Process Control thread described above and the configuration process continues.   

 

End of the Test Series

 

When the JMX Command thread finishes the last permutation of parameters and signals for the last test to run to completion, it returns to the top of the recursion path. As the last step before this thread terminates it sends an interrupt signal to the Grinder Process Control thread, which causes it to exit as well. This finishes the test series.

 

Alternatively, the Console user can click the “Stop test series” menu item, which sends a StopSUTConfigAction event. This interrupts the Grinder Process Control thread, which in turn interrupts the JMX Command thread, causing both to terminate.

Test Series Sequence Diagram

 

Text Box: http

 

This sequence diagram depicts five threads of control as vertical dashed lines. The Grinder Console UI thread on the left causes the two additional threads for Grinder Agent control and JMX Process control to be spawned within the same Grinder Console JVM. The right two threads represent the Grinder agents running on a test driver machine and the WebLogic server on the system under test. Two additional dashed lines are included to show where reading the XML configuration file and writing Input Parameter and Statistics files happens in the flow, although these do not actually involve additional threads.

 

The horizontal block indicated by the green background color indicates the iterative setting of WebLogic configuration parameters followed by the communication with the Grinder agents and running the individual tests. Breaks in the vertical rectangles over the Grinder dashed lines indicate where a thread is blocked waiting for the active thread to relinquish control. Breaks in the flow for the WebLogic server indicate where the server has been terminated by the JMXProcessSeriesControl thread executing a ServerMBean stop() method. The WebLogic server start batch file, “startPetStoreWebLogic.cmd”, was modified to include an infinite loop to accomplish its re-start, as follows:

 

:LOOP

"%JAVA_HOME%\bin\java" %JAVA_VM% %MEM_ARGS% %JAVA_OPTIONS% -Dweblogic.Name=%SERVER_NAME% -Dweblogic.management.username=%WLS_USER% -Dweblogic.management.password=%WLS_PW% -Dweblogic.ProductionModeEnabled=%PRODUCTION_MODE% -Djava.security.policy="%WL_HOME%\server\lib\weblogic.policy" weblogic.Server

GOTO LOOP

 

At the end of the test series, a Thread interrupt() method is used instead of a Thread notify() method to break out of an infinite loop in the GrinderProcessSeriesControl thread. Within the catch block for the InterruptException, StopAction and ResetProcessAction  event are fired to terminate the Grinder agents’ workload.

Conclusion

Having access to Grinder  source code made creation of a special version for testing the effects of WebLogic configuration settings possible. Having a Grinder API designed for plug-ins would make development easier. The advantages of the centralized control of the Grinder Console were retained while automation of its functions makes running a large number of tests feasible and less prone to error. A lot of additional work could be done to improve the test automation. Specifically, this includes 1) handling of WebLogic configuration settings that are not in the config.xml file or can’t be changed through JMX, 2) recovery from stoppage in the test series if a particular test hangs and 3) recording statistics for test components in addition to recording just the total statistics as presented in the Accumulated test statistics table.

 

Todd Nichols, Anser Enterprise

Copyright © 2014 Anser Enterprise