Performance Testing

Purpose

1. To discover bottlenecks, perform capacity planning, optimize the system’s performance

Pre-Requisites

1. Scope of each PSR test
2. Timelines and environment assumptions
3. Details should be obtained from the infrastructure team like network topology diagram to assess which network segments and routers will be affected as a result of the PSR tests. [The network topology diagram and the KBs of data sent per day should help the test engineer plan and mitigate risks associated with the PSR test.]
4. List of components to be monitored like database, application servers, infrastructure etc. and respective contact persons to report the issues.
5. Find out how many KBs of data are currently being transmitted for a typical workday for all applications
6. Templates to collect Environment details
7. Get the detailed requirements to create the expected volumes of transactions and business processes within the environment for a given number of concurrent users
8. Creating a folder structure for the artifacts
9. Tool Evaluation Criteria

Normal scenarios

1. Identify the current response time (backend and client end)
2. Identify current Areas of Bottlenecks
3. Trial run with minimal load
4. Isolate how many KBs of data the application under test sends on a given day for an end-user.
5. Find how many end users can be emulated from a desktop/endpoint
6. Functionality of the application should be robust and stable before initiating a PSR test.
7. Trigger events like emails etc should be tested separately and disabled for a general PSR run
8. Identify component level, architectural level, Infrastructural level and end to end level tests

Data Load

1. Obtain unique data values from the subject matter experts to re-execute the tests with multiple iterations of data for processes that have unique data constraints
2. Create test scripts with enough unique data records to prevent the data-caching problem from occurring.

Ex: Using the same set of data could cause data to be stored and buffered which may not fully exercise the database. Otherwise DBA may need to restore and refresh the database to cleanup the cache.

3. Setting up LAN to handle the Load test
4. Setting up the hardware to handle the Load test
5. Need to clear the end-point browser cache of JS files and CSS to for each run to simulate actual time taken to render the objects.
6. Need to figure out a strategy for clearing browser cache for the apps (like UMS only) but not the cache entries from the login. This is required to measure the timing for doing different operations on an extend period of time.
1. Should not clear the application cache and go ahead with multiple operations to capture the performance.

Execution

1. Inform other users before starting the PSR tests
2. Contingency Plan if the LAN crashes
3. Having all the personnel present
4. Documentation
5. Run on a known environment and make a benchmark/baseline and run on different environments to identify system bottle necks, measuring response times etc and increase the numbers gradually to find the deviations and the system limits
6. Capture and get screen shot print outs of all error messages that the application under test and place it in an organized repository
7. Scrum [interpret test results, monitor the test, and coordinate the testing efforts with multiple parties]
8. If the test environment is small compared to the Production then, the results has to be extrapolated with respect to the production environment. Strategy to extrapolate the results.
9. Actual results of performance with different versions
10. Identify bottlenecks using tools like Windows Task Manager, Windows Performance Monitor, Component Services administrative tool etc
11. Initial few rounds of testing should be followed by a development process to tune the application.
1. Tune the Database
2. Rewriting programs with inefficient SQL statements or network roundtrip calls to server from the client.
3. Application Server Configuration, like Java memory allocation or any other parameters

User Load Plan

1. Steady Load: Same/constant number of users
2. Increasing Load: Begin with small number of users and then the users are increased in sets gradually.
3. Dynamic Load: Change the frequency and the number of users randomly
4. Scheduled Load: Schedule the transactions with random intervals
5. Identify workflows and categorize/generate data in volume for them as given above
1. Login - Number of users logging/logged in at a given instance
2. Reports – Number of users accessing the reports at a given instance
3. CreateUser – Number of users created at a given instance
4. Search option – Number of users accessing the search option at a given instance
5. Billing

Example

SNO	Workflow Name	Parameters	Different sets: Distribution of users	Volume in each set	DataFile
1	Login	Username, Password	Eg: Admin, Helpdesk, finance, readwrite users, readonly users etc.		Login.csv

Test Variables

Variable	Description
TD	Duration of the Test
NU	Number of Users to Start
INU	Intervals to Increment number of users
NUi	Number of users to increment for each interval
TU	Total Number of users

Metrics

1. Response Times

1. Total Response Time: From the time the request sent from client and the response received from the server including network-time, test-environment, database dependencies and network-latency
2. Measure response time from server logs and compare with total response time to determine how the response times vary depending on the network traffic during different intervals

2. Average No. of transactions/sec [user transaction, file transfer, batch process]
3. Average Number of hits/sec
4. Throughput: Number of Transactions/sec, maximum concurrent users & byte/sec
5. Average, minimum and maximum limits of the system
6. Utilization: Percentage of time that the system resources are in use. If utilization is too high, there will be longer queuing delays, leading to higher response times. If utilization is too low, then there is an excess of capacity
7. Analyze why a certain transaction with certain input generates a good or a bad response
8. Number of bottle-necks, memory leaks identified
1. Strategy to fix the problem in short term and in long term
2. Capture all the decisions in one artifact
3. Running the test after the fix
9. Monitoring webserver
1. Processor time
2. Number of bytes
3. Thread counts

Scalability

Purpose

To discover the performance ability of the system, while high volume use to get non-disruptive growth, continuous availability, and consistent response times, even during peak usage times

Architecture level

1. Evaluate whether there is enough memory and processors to handle the expected user volumes.
2. Good Understanding of how easily the configurations of Web, application, and/or database servers can be expanded.
3. Tune database indexing [Incorrect database indexing of information can impact performance when requests are either unable to find information or get stuck in an infinite loop trying to fill a request]
4. For high-volume consider load balancing software/hardware in order to efficiently route requests to the least busy resource.
5. Security and its effect on performance [Adding a security layer to transactions will impact response times because information will require added steps to encrypt and decrypt transaction data.]
6. Identifying the parts of application that requires secure requests, and dedicating specific server(s) for handling secure transactions

Reliability

Search user – Number of users performing various actions like view, delete, edit change password on the same record/user