Accept Testing with Codeception

Acceptance testing is a method of verifying our application behaves exactly as expected, often utilizing a web browser. We will write test scenarios that will be acted out by a user interacting with our application such as logging in or performing a specific task. Acceptance tests are slower than unit tests because they rely on a web browser. What we lose in the speed of running acceptance tests we gain the confidence that our application works as designed with a real web browser. For small teams, acceptance testing can be your first step into a formal quality assurance (QA) process. Larger teams can use acceptance tests to prove new features behave as expected. The single developer that knows you should be writing tests but still doesn’t for whatever reason: acceptance testing can help you jump-start your application’s test suite.

Acceptance testing is my favorite tool to reach for when working with legacy applications that may have low test quality or no tests at all. Because acceptance testing approaches the application from outside of the source code we’re able to greatly increase test coverage without having to touch the application’s code itself. One of the most common test scenarios might be “Can a user log in to our application and land on a specific URL we expect?” which covers many methods of our application giving us more test coverage at the expense of our test failures being more difficult to debug because so many methods are involved. If we find ourselves building a new feature we can leverage acceptance testing to prove our feature works as expected and we can also test our failure scenarios to prove our validation logic and form processing is what we expect.

Where unit tests focus on testing isolated methods and feature tests focus on testing specific methods work together; acceptance testing exercises our application just as real users by clicking on links, filling out forms, and processing data. We’re able to easily create users and set permissions and then verify those permissions are enforced. This level of detailed step-by-step testing provides incredible confidence the application behaves as expected. Acceptance testing is not a replacement for unit tests. Even in situations of legacy applications where acceptance tests are the only tests we still write unit tests for new functionality.

How much of our tests should be unit or acceptance? Do we need 100% coverage? It depends. Test coverage is about confidence in your code. Does 100% unit test coverage give you confidence? What gives me confidence is knowing my applications behave as I expect them to. Acceptance testing also allows me to turn user stories into test scenarios and easily translate business requirements into acceptance tests. When our tests exercise the application in the exact same way real users will the result is confidence in our test suite.

Acceptance testing isn’t the solution for fast test feedback and should not replace unit tests. Acceptance tests should augment your existing test suites and add a layer of confidence that your users can interact with your application exactly as you’ve designed and tested.

Why Codeception and not something like Behat or [Kahlan](https://github.com/kahlan/kahlan Behat and other Behavior Driven Development (BDD) style testing frameworks are designed to test the application’s behaviors. Read about Kahlan in the September 2018 issue. With Codeception’s acceptance test suite we’re able to run scenarios with real users in real browsers. Codeception has been around the PHP ecosystem for 10 years or more and the project has matured into a robust solution for full-stack testing. Codeception supports functional, unit, and API test suites out of the box which makes it a perfect choice for new applications as well as legacy applications that may not have a test suite at all.

Getting Real

How we wire up this real web browser with Codeception is via Selenium. Selenium is a suite of tools for automating web browsers, a very wide and open-ended topic. We are going to skip most of the configuration and complexity that often comes with Selenium by utilizing selenium-standalone from NPM. You can install selenium-standalone as a global NPM package with npm install selenium-standalone -g , save the package to your project: npm install selenium-standalone --save-dev, or run via docker: docker run -it -p 4444:4444 webdriverio/selenium-standalone. If you have installed the NPM package the next step is to run sudo selenium-standalone install to download the Chromedriver and selenium packages. By default, we’ll get the Selenium, Chrome, Firefox, and Chromiumedge drivers installed for us to use. To start the service we’ll use selenium-standalone start which will listen on port 4444 by default.

selenium-standalone start example output

Since we’re using real browsers to test our application I felt it was fitting to pull a real PHP project: Snipe-IT. Snipe-IT is an open-source IT asset/license management system. Think of large companies that have thousands of laptops, desktops, monitors, Windows licenses, and more deployed to their workforce, Snipe-IT is a tool they can use to help manage all of that headache. The project is so successful snipeyhead founded a company to provide a hosted version and offer support to customers. Over the years I’ve been able to contribute to the project and it’s an incredible example of a modern Laravel based application. I’m currently using PHP 8.0 and working off the develop branch due to a new major version release (6.0!) coming in the near future. My development environment is Ubuntu 20.04 LTS however these commands should translate to macOS directly.

Snipe-IT example dashboard of asset management

I have my Snipe-IT running at http://snipe-it.test and once upon a time the project featured Codeception but if you’re just getting started with Codeception you can use composer require "codeception/codeception" --dev to install the package. Make sure you review the fantastic documentation for all the configuration options.

Because Codeception has previously been used we’ll review the configuration from the root of the project and verify we’re ready to run the Acceptance test suite found in tests/Acceptance folder of the project. The root configuration defines where the Codeception test suites will be located along with support and data directories as well as extension configuration which we can leverage to run only failed tests. This configuration can be found in codeception.yml:

paths:
    tests: tests
    output: tests/_output
    data: tests/_data
    support: tests/_support
    envs: tests/_envs
actor_suffix: Tester
extensions:
    enabled:
        - Codeception\Extension\RunFailed

Our acceptance test suite has it’s own configuration file to specify how this suite should operate. Most importantly we configure the WebDriver module to use the URL for our local development environment and specify the chrome browser. We’re also going to configure the Laravel module to utilize the ORM so we have access to Eloquent in order to perform any setup or teardown actions we may need for specific scenarios.

The configuration file is located at tests/acceptance.suite.yml which contains:

actor: AcceptanceTester
modules:
    enabled:
        - WebDriver:
            url: http://snipe-it.test
            browser: chrome
            capabilities:
                chromeOptions:
                    args: ["--headless", "--disable-gpu"]
        - Laravel:
            part: ORM
            cleanup: false # can't wrap into transaction
        - \Helper\Acceptance

Codeception uses these Yaml configuration files to create generated code helpers for your test suite. When you make a change to a configuration file you need to run php vendor/bin/codecept build to build the support classes Codeception needs:

codcept build command to parse configuration Yaml files

The Snipe-IT projects makes use of Laravel’s database seeders so we can set up our application with realistic data generated randomly by running php artisan db:seed. This will also create an administrative user with the username of snipe and password of password which we can use to validate our user’s ability to log in to our application. There’s an existing tests/acceptance/LoginCest.php test scenario we can start with:

<?php

class LoginCest
{
    public function _before(AcceptanceTester $I)
    {
    }

    // tests
    public function tryToLogin(AcceptanceTester $I)
    {
        $I->wantTo('sign in');
        $I->amOnPage('/login');
        $I->see(trans('auth/general.login_prompt'));
        $I->seeElement('input[type=text]');
        $I->seeElement('input[type=password]');
    }
}

The _before method can be used to do any setup required before we start executing our test cases. in our LoginCest we open the login page, expect to see some text and then see form elements for password and text fields. We can give this test a run by itself via php vendor/bin/codecept run tests/acceptance/LoginCest.php

Assuming everything went well we should see the following output in the console:

output from our LoginCest test scenario

we can use the debug flag to get more output from Codeception

Now that we can verify our login pages loads, let’s expand our LoginCest test by filling out the form and attempting to login to the application. We’ll add steps to our tryToLogin() method to fillField() the username and password fields, click() the Login button, and see() the success string:

public function tryToLogin(AcceptanceTester $I)
{
    $I->wantTo('sign in');
    $I->amOnPage('/login');
    $I->see(trans('auth/general.login_prompt'));
    $I->seeElement('input[type=text]');
    $I->seeElement('input[type=password]');
    $I->fillField('username', 'snipe');
    $I->fillField('password', 'password');
    $I->click('Login');
    $I->see('Success: You have successfully logged in.');
}

Running our tests confirms our user is able to log in:

Testing that our user can login to our application successfully.

What if we changed our username from snipe to joe? Let’s run our tests again and see an example of a test failure.

Example test failure output showing us a stack trace and output

We can see from the output that our username or password is incorrect, as we expected. The added bonus that Codeception brings is that we have the HTML output from our application AND a screenshot of when our test failed. We can inspect the contents of tests/_output and see LoginCest.tryToLogin.fail.html and LoginCest.tryToLogin.fail.png

Screenshot from our LoginCest.tryToLogin.fail.png test failure

Armed with the full HTML output from our application and a screenshot of what the browser saw we can debug what potentially went wrong. In our example test case we can verify the user is experiencing the “username or password is incorrect” error.

You might notice a lot of output happening in the window you ran selenium-standalone start in which is the back end of Selenium running our headless browser to execute our tests. If you run into a test failure that doesn’t make sense we can review these logs to see if something in selenium was breaking. In most cases, you should be able to safely ignore this output.

It’s important to understand what we’re testing. With our acceptance test LoginCest.php we are testing a lot of different parts of the application as well as UI elements, log-in form validation, database access, and generally everything required to allow a user to log in to our application is fully functional. This is how we can gain large amounts of test coverage of our codebase at a tradeoff of the test failures being more obscure and possibly more difficult to debug because so many methods are involved. This type of test is how we can validate all of our authentications and login-related unit tests work when we put them all together and authenticate and redirect a user. If you’re building these tests in a legacy application that does not have an existing test suite this is how you gain confidence in your refactoring work: by writing tests to validate the system is working in its current state (which you hope is working but maybe not always, the known state is more important here)

Codeception also supports using helper methods such as logging in a user to make this code easily reusable. We can add helper methods to tests/_support/AcceptanceTester.php and we see there is an existing test_login() method that attempts to login our snipe user. Our static test_login method contains:

public static function test_login($I)
{
    $I->amOnPage('/login');
    $I->fillField('username', 'snipe');
    $I->fillField('password', 'password');
    $I->click('Login');
}

With this helper method, we could refactor our tryToLogin() test to contain our helper method. We could also add our expanded test to the helper so we know we’re validating the login actually works for our test user.

public function tryToLogin(AcceptanceTester $I)
{
    $I->test_login($I);
}

Running our test again validates our LoginCept.php test functions as expected

We can utilize these helper methods to login administrators as well as less privileged users to verify our permissions are being enforced without having to copy and paste the same few lines of code in each of our test cases.

Everyone tests their application. Some of us automate it. You test your application when you deploy it and then click around production to “make sure nothing obvious is broken”. What if we could know there’s a problem before we deployed our code? Codeception gives you the ability to open your application in your browser of choice and click around and use the system exactly as your real-world users would. With selenium standalone, you can skip most of the complexity typically involved in Selenium. If you don’t already have a quality assurance team, acceptance tests can be your start. If you’re already working with a QA team Codeception is a fantastic way to expand your test coverage directly from the people testing your application manually already. If you’re not testing your application in an automated way, start today with Codeception and start small. Test coverage doesn’t happen instantly but you will gain confidence as your test suite expands to cover more of your application.

Codeception was built by Michael Bodnarchuk in Kyiv. Ukraine. The PHP world is better because of Codeception. Make tests, not war.