We are in an era where digital businesses understand the importance of building high-performance systems rather than reactive
performance testing strategies. It’s no more the core responsibility of the performance testing and engineering team that works
in silos to ensure the system performance is assessed and certified a few days before the move to production. With the radical
shift to the Agile/DevOps development model, the adoption of early performance testing strategies (shift-left approach in performance testing)
could bring exponential benefits to build and validate the system performance.
Whether it a legacy monolithic application, or an SOA application that consists of three server tiers - namely web, application,
and DB servers, or a microservices architecture that consists of more fine-grained components to accomplish the
same objective, a robust performance testing strategy is vital to identify bottlenecks at all components/layers.
Different types of performance tests might need to be executed in the correct sequence to find performance bottlenecks
in the system quickly.
The test strategy should focus on end-to-end performance testing and bottleneck analysis covering all the layers - namely
client-side performance testing (page loading and rendering time), network performance testing (impact due to
network type), and server-side performance testing that includes load balancer distribution checks (equal load
distribution analysis across the server farm), web server performance bottleneck analysis (HTTP requests and
thread pool size analysis), application server/API server(s) performance bottleneck analysis (heap analysis and
code profiling), and DB server performance bottleneck analysis (query performance analysis and profiling). This
not only requires expertise in several testing tools and techniques to identify the bottlenecks but also expertise
in providing performance optimization and capacity sizing recommendations (performance engineering).
A robust performance test strategy should focus on end-to-end performance testing of various layers to measure all the relevant
Key Performance Indicators (KPIs) to certify a system for its performance and scalability characteristics.
In general, there are top #3 mistakes why production failures due to performance issues occur despite having focused performance
testing activities:
-
Quantitively and/or qualitatively invalid workload model (not derived from production usage patterns)
- Improper performance test environment sizing strategies (leading to improper extrapolation of test results)
- Use of low database volumes while performance testing (non-realistic data volumes used in DB)
A sound performance test strategy should consider these key aspects to make the right conclusion from the performance
tests.
Prevalent performance bottlenecks at various tiers
Maintaining a database of frequently noticed performance bottlenecks of each tier can help performance engineers quickly
connect the dots to drill down the bottlenecks. A robust performance test strategy should also recommend running
the right type of tests to quickly identify the bottlenecks at all layers of the system. Here is a list of typical
performance bottlenecks identified in various layers during performance testing:
Client-side (browser/mobile device) performance bottlenecks: Client-side performance issues have become
a key contributor to overall system performance during recent years due to the advancement of Rich Internet Applications
(RIA). For mobile applications, analysis of device-side performance issues has become essential. Prevalent client-side(browser)
performance issues are listed below:
- Complex DOM traversal and manipulations leading to high page loading time
- Large event handler bindings
- Slow CSS selectors and a high number of CSS/JS files
- Image handling issues
- JS code performance
- A high number of XHR calls
- A high number of network calls and requests served by a high number of domains
- High utilization of device resources – processor utilization, memory usage, battery usage
Network performance bottlenecks: Network bottlenecks have become a major contributing factor particularly
in mobile applications. Following are the major network performance issues that impact the performance of the
application:
- NIC card configuration issues
- Impact due to WiFi cellular networks – 2G/3G/4G
- Network bandwidth issues
Web server performance bottlenecks: The web server serves the static contents (image files, CSS/JS
files, etc.) to user requests. Prevalent web server performance issues are listed below:
- Server configuration issues
- Disk issues due to log size increase | Enabling of server logs
- Hardware saturation issues
- Limited throughput due to HTTP connection pool configuration issues
- Improper hardware capacity sizing to meet target user loads
Application server performance bottlenecks: The application server handles the dynamic user requests
and executes the business logic of the application. Prevalent application server performance issues are listed
below:
- Ineffective application code
- Performance and scalability issues of third-party APIs
- Memory leaks
- Garbage collection issues
- Heap size configuration issues
- DB connection pool configuration issues
- Hardware saturation issues
- Application server configuration issues
- Improper hardware capacity sizing (& scaling strategy) to meet target user loads
Database server performance bottlenecks: The DB tier has more performance bottlenecks when compared
to other tiers. Prevalent database server performance issues are listed below:
- Cursor handling issues
- Ineffective SQL queries
- Issues due to cache settings
- DB server configuration issues
- Ineffective indexing
- Ineffective scanning
- Locks and deadlocks
- Hardware saturation issues
With the rise in new disruptive technologies and heavy competition across business players, there is an increase in the demand
for enriching the end-user experience. Due to these reasons, there is an increase in the importance of proactive
and continuous performance testing strategies to meet the ever-growing market needs. No doubt, the roles and
responsibilities of performance test engineers are expanding and they need to gear up to face the new challenges
by adopting smarter, innovative, and continuous performance measurement techniques with appropriate shift-left
and shift-right strategies that assure accurate performance assessment. This justifies why Performance Subject
Matter Experts (SMEs) are required to strategize the continuous performance testing, engineering, and performance
monitoring methodologies for modern enterprises.
Our Performance Testing & Engineering (PT&E) CoE at Nous-Testree has more than 20 years of industry experience in
performance test solutions using various open-source and commercial performance testing tools. Supported by our
performance testing and monitoring methodologies, client-side performance analysis strategies, differentiators
solutions, Queuing Network (QN) theory based accelerator tools, CI/CD enablement services, we have helped many
of our clients in assuring the system performance.