Common Penetration Testing Approaches

A common question asked when scoping a penetration test is the difference between black box, white box and grey box testing, and which is the best method. In this post we will cover the difference between the three and how to decide which is best suited, as well as how to decide when to engage a penetration tester for a project.

tldr: Grey box is the best middle ground between the coverage of white box, and the effort required of black box. Organisations should also consider engaging penetration testers at the start of the project, rather than the common method of engaging just before release.

Black Box vs White Box vs Grey Box

Background

Black box, white box and grey box testing are commonly used terms when talking about penetration testing, and they all come with their own trade-offs.

Black Box testing

When doing a Black box test, testers are given limited or no information about the system. This is intended to simulate what a typical adversary on the internet could do. In this method, no credentials are provided, there is no access to source code or other internal documentation regarding the system. The tester must rely on the exploiting what they can access, and attempt to identify further issues using the limited knowledge of the system and any information they can identify online.

Pros

• More realistic as a typical adversary will not have access to source code or design documentation.
• Less requirements on the client side (Access doesn’t need to be provisioned, testers don’t need to be on-boarded etc.)
• Can require less time to complete than other options.

Cons

• Low visibility and coverage of the system.
• Largely dependent on the tester gaining initial access, otherwise internal attack surface may not be explored and assessed.
• Provides a false sense of security if issues are not identified. In this approach a tester has a fixed timeframe to assess the system, whereas a motivated adversary can dedicate significantly more time and resources.

White Box testing

White box testing is the complete opposite of black box testing. In a white box test, the tester is provided with all the information and details available. This likely includes full source code, internal design documentation and any other details the tester may require. White box testing enables a tester to cover all the attack surface for the system, identify vulnerabilities that might otherwise not be identified, including things such as logic flaws, configuration issues and even vulnerable blocks of code that are not exploitable in the systems current state.

White box testing can also allow testers to highlight issues that may be present in the overall design and architecture of the system, and provide guidance to remediate and appropriately harden the design.

Pros

• Full coverage of the target system. The tester has access to everything they need and can attack all components without needing to spend time manually identifying how it all pieces together.
• Testers can work collaboratively with the developers of the system to remediate issues in a more timely and effective manner.

Cons

• More data and setup is required before the tester can begin testing. This may include onboarding a tester to have internal access.

Grey Box Testing

Grey box testing is a mixture of Black box and White box. This means a tester may be provided with some level of access into the system (standard user level credentials etc.), and access to source code and some internal documents may be provided if required/available. Grey box testing can help streamline the way the tester approaches testing, and can be a more efficient approach than either Black or White box.

Grey box tests may include providing the tester with multiple levels of access in the system, meaning configuration review is possible, and could even include source code to make triaging vulnerabilities quicker and more effective. This method allows for testers to focus their efforts on high value vectors from the start of the engagement, without needing to spend time to identify the vectors without any details.

Grey box testing is the most common kind of testing due to the combination of high coverage, time efficiency and low client setup requirements.

Pros

• Easier for the tester to identify vulnerabilities when source code access is provided.
• Allows for more testing coverage than Black Box testing.

Cons

• Can be difficult to decide how much access and details to provide the tester.

How do we decide?

Understanding the differences helps streamline the process when engaging a penetration tester as the client already understands the system and the organisation. The decision for which method to go with ultimately is the decision of the client, however testers will usually have their own thoughts once they understand the scope of the engagement, and will work with the client to determine which method is the best way to ensure the goals as an organisation, and the security of the system is best assessed.

A good note to remember though is that a penetration test is meant to help your security posture. Often times organisations will have a time constraint that testing will need to be completed in, and will opt for a black box approach as the quickest way to meet their compliance requirements. Its worth considering that the best use of that testers limited time is more often than not, a white or grey box approach. While this approach might be less “realistic” it allows for a more comprehensive assessment, and an overall better outcome for the client.

When to engage a penetration tester

Understanding the driver behind the penetration test allows a client to better determine the timeline for engagement, and also assists the tester to frame their testing to the organisations requirements.

Drivers of penetration tests

There a many drivers for initiating a penetration test, at Proactive Labs we most commonly see the following approaches.

• Change based
  • Engaging penetration testers after a major upgrade or change has occurred in a system, to ensure that the changes did not degrade the security posture of the system as a whole.
• Compliance based
  • This can include things like annual testing to meet system certification requirements, or as part of an IRAP assessment.
• Annual / Periodic
  • Conducting periodic penetration tests against systems to ensure that their security posture isn’t degraded over time.
• Post major breach
  • After major breaches clients worry that they are vulnerable to the same kind of attacks, with this in mind they seek testing to ensure that they are not vulnerable themselves.
  • Also used after a breach of their own organisation has occurred, to ensure the remediation efforts were successful and the exploitation issues have been removed. Potentially used to identify what could have occurred if visibility wasn’t complete.
• Internal funding
  • This can occur when an IT department (or any other area of the organisation) is seeking funding to implement more security controls or uplift the posture of their environment. Engaging penetration testers helps demonstrate the current weaknesses and can assist proving the value of the requested security uplifts.

When should we engage penetration testers

Regardless of the driver for engaging penetration testers, there are a few different times during a project lifecycle that the penetration tester can be engaged.

At the completion of a project, before go live

This is the most common time organisations choose to engage penetration testers. The project has reached a stable state where the architecture and design are not going to change, and the system itself is in a position that it can start being used. This allows the tester to assess the system in a state that most closely resembles what will be used in production.

This also means however, that if the tester identifies high risk vulnerabilities that require significant changes (architecture, design, systemic vulnerabilities) it can be more costly for an organisation to remediate due to how far into the project they are. If this occurs it can lead to the pushing of go live dates, not meeting project deadlines and potentially having to re-do work that has already been done.

At the beginning of a project

Organisations rarely engage penetration testers at the beginning of a project, however we believe that there are significant benefits to doing so. Having a tester embedded from the start of a project means that they become intimately familiar with the technology stack, the development processes and they can be a part of the design decisions that are made.

Having a tester embedded from the start of the project means that those large issues that are harder to fix can be identified and remediated early on in the projects lifecycle. Another benefit is that developers are often not as familiar with security practices that penetration testers are. Having the tester engaging regularly with the developers can help ensure that code is written in a secure manner, and vulnerabilities are not introduced unknowingly. The tester can then also perform staged penetration testing across the different areas of the application as they are developed, to help reduce the findings identified at the completion of the project.

When the project is wrapping up and getting ready for go-live, a penetration test across the whole system should still be conducted, but due to the previous work and tests completed, the time requirement and findings should be considerably lower.

What does this all mean?

There are a lot of factors when considering what kind of penetration testing approach to use, and when a penetration testing provider should be engaged. An organisation should take into consideration what they aim to achieve with the penetration test and what their drivers are for testing when making these decisions. Having these discussions internally can then frame the conversation for when a penetration tester is engaged.