Load and performance test tools

Load Testing and Performance of WebSites Developers typically measure a Web application’s quality of service in terms of response time, throughput, and availability. Poor QoS translates into frustrated customers, which can lead to lost business opportunities. At the same time, company expenditures on a Web site’s IT infrastructure are a function of the site’s expected traffic. Ideally, you want to spend enough, and no more, allocating resources where they’ll generate the most benefit. For example, you shouldn’t upgrade your Web servers if customers experience most delays in the database server or load balancer. Thus, to maximize your ROI, you must determine when and how to upgrade IT infrastructure. One way to assess IT infrastructure performance is through load testing, which lets you assess how your Web site supports its expected workload by running a specified set of scripts that emulate customer behavior at different load levels. Here, I describe the QoS factors load testing addresses, how to conduct load testing, and how it addresses business needs at several requirement levels.

QoS Measures QoS is key to assessing how well Web-based applications meet customer expectations on two primary measures: availability and response time. Availability Availability measures the percentage of time customers can access a Web-based application. Availability goals typically vary according to the application type. Critical applications, such as online brokerage, often have more stringent requirements than other applications, such as online travel sites.

However, even in industries where availability is less critical, it is always crucial to a company’s bottom line. In addition to application type, availability requirements can vary according to the time of day or for special events.

During high market volatility, for example, online brokerage sites must be available as close to 100 percent of the time as possible. Likewise, online ticket brokers must exhibit high availability when tickets for a concert or sporting event go on sale. Unfortunately, at just these times, Web sites are subject to “flash crowds” that stretch resources to their limits, sometimes leading to rejected user requests and decreased availability.

Finally, Web or e-commerce site availability also varies for different customers: ISPs and networks provide different levels of service to customers, depending on their geographic location.

Response Time Another key QoS factor for Web and e-commerce sites is response time. With Web-based environments, you must measure end-to-end response time to determine how customers perceive such things as page download and keyword search times. Also, when defining end-to-end response time, you must distinguish between the download time for the base HTML page and that for other page components, such as images and ad banners. A customer’s perception of a Web application’s response time varies according to many different factors — some of which are outside your site’s environment. These factors include :

your site’s ISP,
the customer’s ISP,
the customer’s ISP-connection bandwidth,
which networks route packets from the customer to your Web site, and
the delays imposed by your Web site’s thirdparty

services (such as content delivery networks that provide images and streaming media, or ad networks that provide banners).

Load Testing

load testing lets you measure your site’s QoS performance based on actual customer behavior. When customers access your site, a script recorder uses their requests to create interaction scripts. A load generator then replays the scripts, possibly modified by test parameters, against the Web site.

How It Works The load generator mimics browser behavior: It continuously submits requests to the Web site, waits for a period of time after the site sends a reply to the request (the think time), and then submits a new request.

The load generator can emulate thousands of concurrent users to test Web site scalability. Each emulated browser is called a virtual user, which is a key load-testing concept. A load test is valid only if virtual users’ behavior has characteristics similar to those of actual users.

You must therefore ensure that your virtual users:

follow patterns similar to real users,
use realistic think times, and
react like frustrated users, abandoning a Web session if response time is excessive.

Failure to mimic real user behavior can generate totally inconsistent results. Because customers who abandon a session use fewer site resources than those who complete it, for example, planning your infrastructure capacity assuming that all started sessions will be completed can lead you to overprovision the site. Also, if you fail to consider session abandonment, you cannot accurately quantify important business metrics1 such as:

revenue throughput, which measures the amount of money a Web site generates per unit time (dollars per second, for example), and potential lost revenue throughput,1 which is the amount of money in customers’ shopping carts that was not converted into sales per unit time due to session abandonment.

During the time a Web site is subject to the load generated by virtual users, we measure its performance and obtain metrics such as response time and throughput for each load intensity value that is, based on the number of virtual users.

When to Use It Several circumstances call for load testing. Suppose, for instance, that you anticipate a significant traffic increase to your site following a marketing campaign. In place of what is now a peak of 3,000 session starts per hour, you’re expecting twice that. Currently, your dial-up customers experience an average 6.5-second response time on search requests, the most critical e-business function.

What will be the response time when the site’s load increases to 6,000 sessions per hour? As another example, suppose that you’re adding new functionality to the site or redesigning Web pages. You must know how this will affect response time before your customers find out; doing so lets you detect potential performance problems and fix them before they occur.

Another good time to perform load testing is when you plan to implement IT infrastructure changes.

Testing Parameters and Results

There are three main parameters to vary during a load test:

Workload intensity, typically measured in session starts per hour.

Workload mix, described by the scripts, which define typical sessions and what customers do in each session type.
Customer behavior parameters, including abandonment threshold and think time.

Typical load test results include

Number of completed and abandoned sessions per hour, as a function of the number of started sessions per hour.

Revenue and potential lost revenue throughput, as a function of the number of sessions started per hour.

Individual page download times and transaction completion times versus the number of sessions started per hour.

Load Testing and Performance Relationships

You can use load testing to predict your Web site’s performance at any load level by simply increasing the number of virtual users until you achieve the desired load.

However, running load tests for numerous values with numerous virtual users can be timeconsuming and expensive. You can obtain faster, albeit less accurate, results by combining load testing with analytic or simulation performance models.

You can use a few basic performance relationships to speed up scalability analysis with load testing. Consider a scenario in which several virtual users submit requests to a Web site, and let

         * N_VU = number of virtual users.
         * N_C = number of concurrent requests a Web site is processing.
         * Z = average think time, in seconds.
         * R = average response time for a request, in seconds.
         * X₀ = average throughput, in requests per second.

Using the Response Time law,2,3 we get the following relationship: R = N _VU / X_o - Z

A Web site’s throughput is a function of the load level — the number NC of concurrently executing requests — and the service demands these requests make on individual site resources (processors,storage devices, and networks, for example).

We define a request’s service demand — Di — at resource i as the average total time the request spends receiving service from the resource.3 This time does not include queuing time, and is therefore independent of the load level. Given this, we can write that X0(NC) = f (D1,...,DK,NC) to indicate that throughput is a function of load level and the service demands on a Web site’s K resources. Because the same is true for response time, we can write that:

         R_(NC) = g(D1, ... ,DK,NC). (3)

So, combining equations 1-3, we get

         N_VU = [R(_NC) + Z] × X0(NC). (4)

We can now use either an analytic or simulation model to predict response time and throughput for different values of the NC load level, and use equation (4) to estimate the number of virtual users we need to generate a given value of NC.

Load-testing tools are quite useful here. They can, for example, generate scripts for a few virtual users to measure service demands, which are load-independent. You can then use the service demands as input parameters to performance models. 3 Consider, for example, a Web site that has service demands for processing and I/O of 8 and 9 milliseconds, respectively. Using the Mean Value Analysis method,3 you can compute X0(NC) and R(NC) for various values of NC.

You can also estimate the maximum value of site throughput from the service demands by using the upper bound3: X0_(NC) ≤ 1/max{Di}. (5) Thus, in our example, the maximum throughput would be 1 / 0.009 = 111.1 requests per second, which is the horizontal asymptote of Figure 2’s throughput curve.