June 22, 2011

Performance Testing is not necessarily Performance Engineering

Written by

I’ve been in Performance Testing for almost 10 years and consider myself an experienced Performance Testing consultant. As a Performance Testing consultant I have worked for a variety of organisations over this time and of late have noticed an increased trend within organisations referring to the “Performance Testing” pipeline as “Performance Engineering”.

In a recent role, I was referred by a client as a “Performance Engineering expert”.   Which brings me to ponder this thought, “do Performance Testers and Performance Engineers perform the same role?”.  How similar are Performance Testers and Performance Engineers?

In my humble opinion, there is a clear distinction between the services provided by a Performance Tester and Performance Engineer.

Let me expand firstly on the roles of each. Firstly, for a bit of clarity, I am grouping the roles of the Performance Analyst, Performance Lead and Performance Manager within the boundaries of a Performance Tester. Therefore, some of the roles and responsibilities of a Performance Tester are as follows (not a definitive list), design and execute a Performance Test Strategy and Plan; design a Application Simulation Model (ASM); build the automation suite and artefacts;  build monitoring and metric gathering solutions; execute test and tuning cycles; analyse results against requirements; and reporting to business stakeholders. These are some of the roles that are considered and commonly accepted as Performance Testing resource role and responsibilities.

Whereas, a Performance Engineer has in-depth expertise in development and infrastructure components that assist them to provide services such as code reviews; running up profilers (application profiling); investigating environment configurations; reviewing performance best practices; tuning and supplying remediation activities to performance problems. In short, the Performance Engineer provides the in-depth expertise to resolve performance related bottlenecks.

Just to highlight the difference, a former Performance Test colleague once mentioned to me that their role is to discover the performance defects and report against the requirements. If a system or application does not meet its requirements, a defect is raised which then becomes a development/project problem to resolve. This is the true definition of a Tester on any project (albeit Functional or Performance). Their strengths are within the delivery and scope of “testing” processes and as l mentioned before, their roles as a Performance Tester would be focused on delivering a performance quality management solution (test processes, test artefacts, test metrics and reporting). This is where the Performance Engineer provides the value add. In this instance they will analyse the outputs of the test metrics and provide remediation activities or solutions to the development and infrastructure teams to ensure performance problems are resolved.

Having said all of the above, an excellent Performance Tester must have some (but not detailed) knowledge of Engineering activities. The accomplished Performance Tester is able to identify the bottlenecks, whether infrastructure, application code or JVM behaviour and assist to resolve these issues. An example might be that a good Performance Tester should be able to use tools such as HP Diagnostics to identify time spent within an application under loaded conditions.

And this is why Performance Testers are in a unique field. They are expected to have the knowledge and expertise in the Performance Test domain best practises and also have the expertise in resolving and remediating performance defects via engineering initiatives.

This is where, at times, the distinction between the Performance Tester and Engineer can become grey and perhaps why the terms are commonly used in place of each other.

From my experience, for a performance program to achieve best results a performance team must be compromised of both performance testing and performance engineering disciplines (SME’s). My experiences over the last few years have highlighted that to deliver successful performance programs these two roles must work closely together in a performance team, with one supplying the testing methodologies/artefacts and the other analysing such results from the testing to determine and remediate the performance defects.

I will leave you with this thought, could (and should) a functional tester inspect code to resolve a defect?  My humble opinion is “no”, that is the responsibility of the development team. We don’t refer to a functional tester as a functional engineer? In the performance version of testing and as aforementioned at the start of this blog, I believe that the Performance Testing and Performance Engineering roles are two distinct roles with differing deliverables and responsibilities. These two distinct roles are important cogs in any delivery of a successful Performance program.

Categorised in: Uncategorized

This post was written by John Natsioulas