Writing Features and Scenarios

E2E test suite structure

All our test suites share the same structure, configuration files, options and way how we run the test suites. Below is common structure and location of important files within the test suite:

artifacts/
  |- temporary/
  |- reports/
  |- screenshots/
  ...
features/
  |- zendesk/
  |  |- zendesk-chat.feature
  |  |- zendesk-permissions.feature
  |  ...
  ...
fixtures/
  |- 01-base/
  |  |- 00-prepare-system.feature
  |  |- XX-create-users.feature
  |  ...
  |  |- 99-save-fixture.feature
  ...
src/
  |- steps/
  |  |- profile.steps.ts
  |  |- payments.steps.ts
  |  ...
  |- cucumber-hooks.ts                    
cucumber.mjs                              

Artifacts

Folder containing reports, screenshots, temporary files and other artifacts created by the test suite. This folder is not part of the Git repository as these files are created locally.

Features

Features folder contains all tests for the Features. It contains sub-folders for each of the features which in turn contain single *.feature file for a particular aspect of the feature. For example we may have a feature file for main functionality and additional one for the permissions related to that feature.

Fixtures

Fixtures contain numbered folders representing the a fixture, eg 01-single-language. These folders contain *.feature files, each creating certain aspect of the fixture.

Steps (src/steps/*)

Step definitions which are defined within the test suite for the product - are specific to the product - are located in the src/steps folder. Before adding any steps to the test suite check e2e.support shared steps!

Cucumber hooks (src/cucumber-hooks.ts)

Cucumber hooks file is a main entrance point merging Cucumber with Playwright and E2E Support. This file is used to create the FryWorld, customize Locations and Actors (user roles).

Cucumber configuration (cucumber.mjs)

Cucumber configuration file which consumes command line parameters and builds the configuration for the test run. This file contains Profiles used to amend the behavior of the test suite such as Retry, Exluding Tags, and so on…

Gherkin/Cucumber language

We are using gherkin/cucumber so a non-developer can write test scripts/features using plain English. Gherkin uses specific syntax to create the test steps (Given, When, And, Then). For every step, a developer will write the accompanying step definition (i.e. code) to make it execute in the web browser.

The goal is to have tests written before a new feature is released to customers.

Gherkin syntax

For detailed Gherkin reference see: https://cucumber.io/docs/gherkin/reference/

Given: is used to set up the system for the feature. It provides the context of the feature.

Given system state is loaded from fixture "written-ul-mwq-exams"
Given Admin is logged into Practique

When: is used to describe the action an actor is performing in the feature.

When Question author clicks the "Create new item" button
When Admin navigates to "Items" using the main menu

Then: is used to indicate the expected result of the previous action(s).

Then Admin sees 'Changes saved' text on the page

And: is used when there is more than one Given/When/Then step in a row.

Given system state is loaded from fixture "written-ul-mwq-exams"
And Admin is logged into Practique

When Question author clicks the "Create new item' button
And fills "this is a test item" into "Item title" form field

Terminology

Fixtures vs Features

Fixture is the system/data setup and Feature is the behavior we want to test.

Setting up data for a specific test is often the most time consuming task when testing so we make use of fixtures to set up the system/data for whatever feature we are testing.

For example, for the MWQ marking feature, we want to load an MWQ exam, therefore an MWQ exam fixture is created and in the feature, we just have to load that system state (i.e. MWQ exam). This same fixture can be used to test other features that need an MWQ exam.

Scenarios

Within each feature, there will likely be a series of scenarios. A scenario should only be testing one thing because if one step fails, the rest will be skipped and you won’t know if subsequent steps pass or fail.

For example, we might have a feature to create a written exam blueprint and all the scenarios to do with creating written exam blueprints.

Feature: Written exam blueprint
  Scenario: System set up
  Scenario: Create written exam blueprint (by items)
  Scenario: Create written exam blueprint (by marks)
  Scenario: Edit written exam blueprint
  Scenario: Delete written exam blueprint

Configurations

Some features will need a specific configuration set up. If a feature needs to have a non-default setting, we have to create a configuration file. We currently have the following configuration files: multilingual, unilingual, marking_discrepancy_per_item_total.

This will be the first scenario in the feature file:

Scenario: System set up
  Given system configuration "unilingual" is loaded

Writing a feature file

The first line is always “Feature: high level name of feature”

Enter a description of the feature and what is to be tested

The first scenario “Scenario: the first scenario name”

All the steps for the first scenario

The second scenario etc..

A typical feature will have a Given step (system set up), followed by a series of When steps (actions an actor takes), followed by a Then step (expected outcome).

Tips

When writing a new feature file, make use of the existing step library. The step you are wanting to write may already be created. As we create more and more tests, our step library will get more and more populated.

For example, the step to click a button is created. You only have to modify the name of the button.

When Admin clicks the "View" button
When Marker clicks the "Mark" button

We enclose exact text within double quotes to identify a button name or specific text the actor should see on the page.

Every When, And and Then step must have the actor identified.

For example, the following is part of the Item Preview - SBA feature file

Feature: Item Preview - SBA
  -preview an SBA item (html, pdf, MS Word, Exam preview (p4b))

  Scenario: Load items and system state
    Given system database is clean
    Given system state is loaded from fixture "written-ul-mixed-items"

  Scenario: Preview SBA item - html (uncheck all options)
    Given Admin logged into Practique
    Given Admin authentication credentials are stored for reuse
    When Admin navigates to "Items" using the main menu
    And Admin selects "40" items per page
    And Admin clicks the "View" button for "SBA with 10 answers" row
    And Admin clicks the "Preview & Export" button
    And Admin clicks the "Export Item Preview" link
    And Admin selects "HTML" format
    And Admin unchecks "Highlight correct answers" checkbox form field
    And Admin unchecks "Include blueprint tagging" checkbox form field
    And Admin unchecks "Show item information as table" checkbox form field
    And Admin unchecks "Include resources" checkbox form field
    And Admin clicks "Export" in modal box
    Then Admin sees "Single best answer question #21 (version 1)" heading on the page
    Then Admin screen matches the reference image "sba_item_preview_minimum_options"

  Scenario: Preview SBA item - html (select all options)
    Given Admin logged into Practique
    When Admin navigates to "Items" using the main menu
    And Admin selects "40" items per page
    And Admin clicks the "View" button for "SBA with 10 answers" row
    And Admin clicks the "Preview & Export" button
    And Admin clicks the "Export Item Preview" link
    And Admin selects "HTML" format
    And Admin checks "Highlight correct answers" checkbox form field
    And Admin checks "Include blueprint tagging" checkbox form field
    And Admin checks "Show item information as table" checkbox form field
    And Admin checks "Include resources" checkbox form field
    And Admin clicks "Export" in modal box
    Then Admin sees "Single best answer question #21 (version 1)" heading on the page
    And Admin waits for 3 seconds
    Then Admin screen matches the reference image "sba_item_preview_all_options"
    And Admin sees "Ultrasound PNG" text on the page

Notes from calls

Whateever