Boundary Testing

Nov 10, 2020

Peter Aitken Principal Software Engineer, Applications

Motivation: Avoid issues in boolean methods due to wide ranging test data or not enough test cases.

Let’s take an example of a boolean / question method, goal_achieved?

# app/models/sales_report.rb
class SalesReport < ActiveRecord::Base
  def goal_achieved?
    self.number_of_sales > 500
  end
end

The tests tend to look something like this:

# spec/models/sales_report_spec.rb
RSpec.describe SalesReport, type: :model do
  describe "#goal_achieved?" do
    context "sales below 500" do
      it "goal is not achieved" do
        subject.number_of_sales = 400
        expect(subject.goal_achieved?).to eq(false)
      end
    end
    context "sales above 500" do
      it "goal is achieved" do
        subject.number_of_sales = 570
        expect(subject.goal_achieved?).to eq(true)
      end
    end
  end
end

Looking purely at the tests, they are nice and clear. However, somewhere between 400 and 570, a goal becomes achieved. Yes, it’s at 500, but what are the specifics around that point?

Is the goal_reached? when the number of sales is 500 or not?

Let’s really tie down the wide test data range around the boundary, and what happens in our tests by checking just below, on and just above the boundary of 500:

• just below the boundary where number of sales is 499 rather than 400

• add a new test where the number of sales is exactly 500

• just above the boundary where number of sales is 501 rather than 570

# spec/models/sales_report_spec.rb
RSpec.describe SalesReport, type: :model do
  describe "#goal_achieved?" do
    context "sales below 500" do
      it "goal is not achieved" do
        subject.number_of_sales = 499
        expect(subject.goal_achieved?).to eq(false)
      end
    end
    context "sales of 500" do
      it "goal is achieved" do
        subject.number_of_sales = 500
        expect(subject.goal_achieved?).to eq(true)
      end
    end
    context "sales above 500" do
      it "goal is achieved" do
        subject.number_of_sales = 501
        expect(subject.goal_achieved?).to eq(true)
      end
    end
  end
end

Our new test “sales of 500” fails, due to

  def goal_achieved?
    self.number_of_sales > 500
  end

Which can be fixed by making the “>” to “>=”

  def goal_achieved?
    self.number_of_sales >= 500
  end

Keeping the tests tight around the boundary provides more confidence, as if the boundary changes then the test suite safety net will quickly let us know. It has also forced us to think about what happens when the number_of_sales is exactly 500, which may prompt a discussion with our Product Owner enabling us to get the necessary code in place without having to raise another PR for a single character change (“=”) in the near future.

Summary:

✅  Anytime there is a >, <, =, or any combination of them, have 3 tests. One just below, one on and one just above the boundary.

✅  Avoid data that is far from the boundary.