Test everything

TDD from starting from user stories – a top-down style

Posted in programming by exceedhl on August 2, 2007

A rule of TDD is writing tests first. In most real world projects, the software usually contains GUI, web, database and dependencies on external systems. Usually they are not as simple as examples in junit cookbook that can be test driven out by just doing unit testing. There are different kinds of tests in such kind of projects such as functional tests, unit tests, acceptance tests etc. Different projects have different styles of implementing TDD. For example, testers may take the responsibility of writing acceptance tests and developers are only writing unit tests. I would like to advocate a top-down TDD style starting from user stories and ending at production code for TDDers.

TDD from starting from user stories


Before we start talking, I would like to achieve a shared understanding of various types of tests with you. We probably have heard about unit test, functional test, acceptance test, module test, integration test etc. By unit test we mean tests that only test one class by mocking out every class it couples with. By functional test we mean tests that test if groups of clusters of classes meet external requirements and achieve goals. Module test and integration test are a kind of functional test that test logically divided module and the integration between modules, and we just treat acceptance test as one of kinds of functional tests which tries to cover a more complete scenario.

Based on our common understanding of these typical kinds of tests in our application, “top-down approach” says developers should write acceptance criteria for the user story first, run it and see the failure, then start to write unit tests, see the unit test fail and fix them. After finishing an amount of unit tests, go back to run the acceptance again, see it pass and then we can say this acceptance criteria has been fulfilled and we can move to next acceptance criteria and repeat this process.

There are several points in this process:
* There may be several more levels of functional tests(such as module tests, integration tests) you need to write between acceptance tests and unit tests depending on how do you organize your tests in your project.
* Always keep it in mind that doing simplest things to make your tests pass. Apply this rule when you try to pass the unit tests and functional tests.
* A prerequisite of this approach is a good user story. A good story is a INVEST story which is small and testable. If you don’t have clear acceptance criteria or the story is not testable , you may find it’s difficult to write acceptance tests first. If your story is too big, you may find there are too many acceptance tests you have to make them pass and the story will take you very long time.


This is not the only approach we have. As I said at the beginning, different projects and different people have their favorites. Here are some reasons why I like to use this approach.

TDD zealots like me want to test everything in the software and make everything automatic. We all know the rule “no tests, no code”, because tests represent the requirements. We think by expressing those requirements using tests first will make sure that we have understand the requirements before we start to write production code to meet those requirements. Tests also provide quality assurance for our production code, serving the documentation and ground for refactoring etc.

Things become a little complex and interesting when a story has different kinds of or levels of tests. When you are writing unit tests, you may be already clear about the functionality and interfaces of the class you want to test. But how does the need of this class come? How do you decide we need this class and we will design like this? The answer is you decide to create these classes to meet the functionality described in the story, you are writing unit tests according to your understanding of the functionality described in the story which usually comes by your talking with business analysts.

At this time we can say without the requirements of the story(functionality described in the story and usually represented as acceptance criteria), we are not clear how will our code look like and what unit tests to write. Since you write unit tests to test a unit piece of functionality(usually an public interface of an class) which is again determined by the functionality represented by the story, since you have to understand functionality first to write unit tests, why not first write functional tests to represent the requirement, see it fail then find out the solution and write the corresponding unit tests.

So starting from writing acceptance tests will make sure you have understood the requirements of the story which will help you to choose a simple solution to fulfill the requirement represented by the acceptance tests, then write unit tests to based on your clear understanding of each classes’ functionality in the solution and use simple code to make them pass. This is a very natural TDD approach.


As we said above, the first step is to turn acceptance criteria into tests, so one of the most important factors for us is to find tools that support us to achieve that goal. Functional testing tools are actually a very important part in our process. Given that there are already many unit testing tools available for nearly every programming language, a suitable functional testing tool seems more important for us.

While unit testing tools can be categorized by programming language, we can examine those functional testing tools by domains of problems. Due to differences of technical implementation nature of web application, windows desktop application and java GUI application etc., there are different functional testing tools for them.

For web applications, functional testing tools such as Selenium, Watir(Watin, Watij), Sahi are available. We have Abbot for java GUI applications, NUnitForm for Windows Form applications and Microsoft UIAutomation Framework for a wide range of windows application. More heavy tools like QTP could also be a choice. For other kinds of software such as library, console application we can either use xUnit framework to do functional testing or we can use varies of other approaches to achieve such purposes to some extent.

It seems that functional testing tools for some kinds of applications are more abundant and mature than those for other kinds, which seems make implementing the practice in this article easier in some applications than in others. It is probably true, but just remember the rationale behind this approach and once you understand and agree with it and try to apply the rules behind to the greatest possibility, you can always find other variations and substitutes in practice.

By continuously refactoring your testing code, you can actually make your tests easier to understand and more business natural(DSL). Some tools like rbehave and jbehave are already able to allow us to write our acceptance tests in a way more clearly reflecting business value and easier to understand by non-technical people. They help developers to think about the problem and their code from a perspective which focus more on business value of a story.

pros and cons


* writing acceptance tests first makes developers think more about value before starting to code
It’s really hard to write acceptance tests first without clearly understand what are you going to test. Strictly applying the rule of “writing tests first” forces people to think more about business requirement before they start to write any code. This approach encourages developers to understand the business first by either talking to business analysts or customers directly, by this way the misunderstanding of requirement can be reduced to minimum extent and the discrepancy between business people and development team can be decreased.

* test automation is easily achievable from the beginning of the project
TDD starting from acceptance tests implies automation of all tests in the project by itself, because it’s impossible for people to get fast feedback without the ability to automatically build the project and run tests. With the support of continuous integration tools such as CruiseControl, regression testing is also easily achievable.

* clear and simple design, testable code
As mentioned before, we should clearly understand the business value of the story and the acceptance criteria and then turn them into tests. The rest of our work is just to use simplest code to make those tests pass. If writing unit tests first can help us achieve simple design of each public interface of a class, writing functional tests first can also help us choose simplest design to implement the business value.

* tests as documents
One byproduct of TDD is a good set of tests which can serve as documentation of the project. Having a high quality collection of functional tests can help everyone understand the business requirement and the functionality of the application by just going through the functional tests, Meanwhile, unit tests are documents for people to understand the implementation of each class.

There are tools such as Testdox helping us translate our tests which can only be understood by technical people into a format more friendly to non-technical people. Testing frameworks like rspec already have built-in support of this kind of translation, and other frameworks like rbehave and jbehave make the tests themselves represented by business natural language. All of those efforts can make our tests more representative of business value, more self-explaining and more easy to understand.

* delay the implementing and design decision
Given the existence of the tests, developers can make implementation and design decisions at the last step, after tests being written.

* nurture the good habits of programming
Developers should never only focus on coding and technical problems. Good developers should have a sense of considering everything to make what they produced really valuable and with high quality. Agile software development methodology requires developers to be more versatile. Besides having capability of generating high quality code, developers should understand the value of what they are going to do and be able to question the requirement if they are not so valuable. Thinking about our code from the perspective of the value they produced helps us nurture a good habit of simple design. TDD from stories can help us to achieve this.

* testing team more effective
With functional tests written by developers and automated, testers can be more effective. Tester can spend most of their time on other testing such as performance testing, exploratory testing and manual testing etc. They can focus their energy on tests that can not achieved by automated tests. They can spend more time on acceptance criteria and providing their testing ideas rather than actually write tests code.

One practical problem is that some testers can not generate very good tests code. This problem become more serious when the tools need more programming thinking and the test suite become huge while the project grows bigger. The problem disappears if developers write most functional tests.

In agile planning, we usually have to estimate how much work a story will take. Developers writing functional tests first can make them more confident and easier to say that a story has been finished if all tests passed, which helps them estimate the work of each story more accurately. That will make the release planning easier.

* not mention fast feedback, iterative release


* sometimes it’s not easy
While we are talking about many benefits of this TDD style, we should realize that sometimes it is difficult. For people who have been used to it, it is very natural and happy to do this, but it might be difficult to be accepted by other people. For most people, TDD itself is a mind shift. They might have been used to writing code first and testing and fixing the problems. The common questions from them are such that how can I write tests without the code that I am going to test. Doing TDD and using this approach need people to understand the problem first, then you can write tests to represent the requirements. It’s not easy at the beginning, but it’s a good practice and worth spending time on it. We should make ourselves disciplined and think about the software from value perspective. Pairing with some experienced people can also help us quickly be accustomed to this approach.

TDD from stories need some prerequisites. Firstly we need good stories, good stories are independent, valuable, estimable, small and testable stories. Good stories make developers easy to understand the business value and raise good questions. They are small and testable so that we can finish them with writing tests first in reasonable time. Secondly the acceptance criteria of each story should be clear, so that developers can easily turn those into tests. A good story is the corner stone of a successful project.

* difficult in some areas like cpp, game, restricted by the availability of tool set
We have said that one important factor for this approach is having a good functional testing tool, which is not always the case. In practice sometimes it’s very difficult to find a good functional testing tool for your application because of the programming language you use, the target platform of your application or the framework your application depending on. For example, there are not very handy tools for testing windows desktop application. QTP is a good tool, but it’s too heavy for TDD. Another example is game. It is also very difficult to implement TDD in some languages such as C, Cpp.

Tools are evolving. Several years ago it’s also very difficult to do functional testing for web application, but nowadays there are a bunch of tools for web application functional testing. The requirement will drive people to make better tools.

* build will get slow if not being careful because functional tests number increases
By definition, functional tests test the real functionality of an software. Usually we need to hit the databases, transfer data through the network, writing and reading files etc. in our functional tests. One common problem is while a project becomes bigger, the time spent on running all those functional tests will be very long. That will make us unable to get fast feedback from functional tests and decrease the productivity of the team. It will make developers unwilling to run those tests and write functional tests.

Sometimes we have to face this reality, but in most situations we can find many solutions to solve this problem or at least we can bypass it. For example, we can write and run functional tests belong to the story that we are working on during development, and then we run all tests before checking in our code. In some cases we may divide tests into smaller groups and run them parallel. We may also be able to reorganize our tests and use stubs and mocks to test our application without talking to some external systems.

a sample

Let’s use some sample code to demonstrate the whole idea and process. Suppose that we are working on a web application, we have a story about login functionality, which is:

As a user, I want to login to the website, so that I can use registered user specific functionality.

For this story, we have those acceptance criteria:
* Happy path: successfully login
Given: user go to the login page
When: user input correct username and password and submit
Then: user logged in with a successful message

* username or password missing
Given: user goto the login page
When: user does not input username or password and submit
Then: user should see an error message with “username or password missing”

* username or password incorrect
Given: user goto the login page
When: user input wrong username or password and submit
Then: user should see an error message with “username or password is not correct”

Suppose we use Ruby on Rails to make this web application, with the help of selenium, we can easily write acceptance tests for this story:

Story "Login", %(
        As a user,
        I want to sign in the website,
        So that I can use registered user specific functionality) do

    @selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*iexplore", "http://localhost", 10000);

    Scenario "user successfully login" do
        Given "correct username and password" do
            @selenium.open "http://localhost/users/login"
            @selenium.type "username", "someone"
            @selenium.type "password", "password"

        When "login" do
            @selenium.click "submit"

        Then "user logged in successfully" do
            @selenium.is_text_present "Welcome, someone!"

    other scenarios...

Here we use rbehave, a framework for expressing business acceptance criteria using ruby code. After we got this tests, we can run them and see the failure. Now we can think about how to make this test pass. At this point, we probably need to create a UsersController and an action called login, and have User model. Once we made that decision, we can start to write tests for them:

def test_registered_user_should_able_to_sign_in_and_redirect_to_home_as_default
    post :login, :name => "someone@domain.com", :password => "password"
    assert_equal "Welcome, someone!", flash[:notice]

Here we test that the login action of Users control should perform the login and then set a flash notice, then we will come up with a controller action:

def signin
    user = User.authenticate(params[:name], params[:password])
    flash[:notice] = "Welcome, " + user.nickname + "!"
    redirect_to :controller => "home"

Obviously we delegate the authentication to the User model, so we write the tests for it:

def should_be_able_to_get_authenticated_user
    assert_equal users(:someone), User.authenticate("some user", "user's password")

After this we can easily write some simple code to implement this authenticate method. We don’t have to worry about too much exception handling because we are currently just focusing on the happy path. After we write more tests, we will make our code more robust gradually.

So far I have presented the process of implementing a successful scenario of a simple story. Notice that we solved a problem by dividing them into many vertical slices(here are different scenarios), every time we finish one slice, write the tests first and write the simplest code to make them pass. In this way we are very confident at each step, and we can clearly see the value we are creating.


* lack of tools support
Sometimes because of lack of testing tools, we can not write full scenario functional tests for the application. Under this situation we still want to achieve this goal to the greatest extent, so we write functional tests for modules instead of whole application(end to end). For example, if we can not find a very handy functional test tool for Windows application, we may organize our code to some pattern like passive view, so that without performing actions through the view, we still can write functional tests for our application (maybe through testing the controller). Another example is in a C/S rich client application, we can test the server module’s functionality to guarantee that the server works as we expected.

* dev unit test -> add functional tests

* extract writing functional tests from dev to tester
It is very attractive for QAs to write functional tests and for Developers to write unit tests and let them work parallel. It seems the productivity of the whole team will be greatest. Once the acceptance criteria has been clarified, dev can write unit tests, and tester may write functional tests. Ideally it’s true, but it’s difficult practically. The following are some potential problems related to this approach:

* Sometimes dev and testers can work parallel, which may increase productivity(but again it’s dubious, because this may need more communication and iteration between testers and devs).
* Depending on the tools the QA uses, sometimes it requires QA to have more advanced programming skill to write and maintain functional test suite. If QA does not have that level of skill, it’s difficult for QAs to catch up with devs to write functional tests for new developed functionalities. If developers deliver the stories very fast, QAs’ job will be more difficult.

If these problems happen, there are some ways to mitigate it. One way is to let developer write and maintain some (not all) functional tests (happy path tests, for example), integrate them into developers’ build and make sure them pass before developers’ commit.


Our stories? Their stories?

Posted in consulting by exceedhl on June 26, 2007

Working with customers as an agile consultant in these days, I found that using agile software development practices to manage this consulting project works very well. By turning training, coaching and implementation works into stories and plan with customers, utilizing various best practices such as standup, retrospective etc. that we have been always doing in our software projects, we have made our work transparent to the customers and our planning very adaptable and responsive. A group of customers from different projects are a part of our project team. They acts as customers and project team members as well.

One issue that I found recently is about some coaching and implementation stories. For those kinds of job we have to work with our customer people for a period of time, and they usually have their own daily job to do. Our customers have already used stories to represent the requirement so they have their stories to finish. At the beginning I felt that we are interrupt their working and sometimes they do not have time to work with us on our coaching or implementation stories. We are like sort of external forces not belonging to their projects. I am not saying our customers are unlike to work with us, on the contrary they are very willing to work with us. The problem is just that we were not involved into their projects though we have involved our customers into our consulting project.

Things started going well after we turn our stories into their stories.  For everything we have to work with them, we try to create a story in their project and the people who are assigned to that story will have dedicated time to work with us, which also improved communication between us and customers.

JavaScript functional testing using selenium

Posted in Javascript by exceedhl on July 25, 2006

Currently we found that current selenium can actually be used to test the functionality of javascript. Check the ajax testing demo: http://www.openqa.org/selenium-core/demos.html. You will find selenium can be used to test the drag and drop and other ajax effects.

The recently added css selector locator can be used to test the order of HTML elements by using the pseudo class selectors as you saw in the demo. The drag and drop command works well in the Firefox, IE and Opera.

Based on selenium, jsunit, we actually can develop javascript application in a TDD way. There are still some cross browser bugs and usability of some commands are not so good, but we are trying to improve it.

Tweak Emacs

Posted in Emacs by exceedhl on July 13, 2006

After I used Emacs, Eclipse and Intellij Idea for a period of time. I gradually found some of their best features that we use everyday. I found Intellij idea is quite good, but it’s heavy and not very suitable for editing some kind of files(ruby, for instance). I like Emacs but when I use it, I found some features and key bindings are not well designed to be very efficient (at least for me). So I decide to add those features and change the key bindings to make it more comfortable to use it.

Here are some features that I think a good editor or IDE should have:

  • Move(word, sentence, sexp, paragraph, page).
  • Transpose(line, that is to move one line upward or downward)
  • Mark (word, string, sexp, paragraph, page, whole document, C-w in idea)
  • Delete(word, space, empty lines, one line, zap to char)
  • Insert(split line, open line, duplicate one line)
  • Case change(word, region)
  • Live template. I like to invoke the template by pressing <tab>
  • Auto completion. Auto completion was bound to C-SPC in idea, alt-enter in eclipse. There should be only one key binding for all kinds of auto completion. If there are multiple choices, there should be some easy way to choose what you really want.
    • auto completion while typing. like pabbrev
    • for pairing tags like xml tags, finish typing the first part should invoke the insertion of the second part and the cursor should stay between them in the end.
    • library auto completion
    • tag auto completion
  • Comment and uncomment should use same key binding. Editor should be able to check if some text has been commented and execute the corresponding command. Generally, if a selected region containing some lines that are not commented, the command should comment the region, otherwise uncomment it. C-/ is a good key binding for this.
  • Navigation:
    • Go back and forward. In idea, they are bound to C-M-Left and C-M-Right.
    • Go to declaration. In emacs, the tag can support this to some extent. Need investigation.
    • Last edit position which is pretty much like the above one.
  • Document reference support
  • Spell checking
  • Tab bar. The tabbar design can referred to idea.
  • Should provide an unobtrusive way to popup a window displaying a directory structure or type hierarchy etc.

IDE feature:

  • Support unit test running, comipling and run the application etc.
  • Refactoring
  • More sophiscated ide can support intentional programming.

Some thoughts:

  • Moving is a very common operation. Usually people use mouse to move their cursor. But they can be done using pure keyboard too, emacs does that perfectly. But those moving key bindings should be simple and not be scattered, which means you don’t need to press many keys to move and the switch from moving by word to by sentence or sexp should be very comfortable.
  • C-w style marking implemented by IDEA is very nice.
  • Delete extra space or empty lines are nice features. They can also be done while formatting the code. And deleting word, sexp or sentence where the cursor is in are also very useful.

Frequent operations should be bound to key bindings that are easy to press. such as copy, paste, cut, delete one line. The key binding should also consider using mouse. Actually, mouse can be very useful. I think the key to be efficient is not that you don’t have to move your hands outside the keyboard to finish all tasks but be able to do what you want by very simple and few steps. If you can use mouse to select a region and your left hand can easily and comfortably perform cut and paste operation, I think that’s better than using C-N or C-P like key combination to select a region.

The design of idea is intentional programming. You do not have to browse the file structure to select the file you want to edit. You can navigate easily while you are thinking, to finish your job you don’t need many mouse clicks and key pressing, that’s partly why people says ecplise is not as good as idea. You just have to click more and press more to finish a job than using idea and Eclipse don’t support you to program smoothly.

The same idea can be applied to emacs. Do it simple. Frequent operations should be able to be done by very simple steps. More unusual operations can be bound to more complex key sequence. For example, moving by sexps in lisps should be simple to do. Moreover, if you don’t want to let your hands leave the keyboard, most moving operations should not need more than 1 key pressing to do it. Like moving by paragraph, the default key binding is C-S-Up or Down, which is good, while you are holding the C and S, you can move the cursor by paragraph as many times as you wish by just pressing up or down. Some other operations like compile, open html file in browser etc. are not very frequently used, so they can be bound to key sequence over 2 key pressing( such as C-c C-v ).

Don’t complain that emacs doesn’t support some ide feature like file structure pane very well. Anyway, we should provide them ,but the important thing is to make programming in emacs don’t need you to use them very often, like in idea.

How to make emacs more comfortable to use:

  • (done)Swap the caps lock and left control on your dell d610 keyboard, just because the design is bad.
  • Kill-buffer command (C-x k) is used very often(at least for me), which can be bound to a more convenient key binding.
  • Idea has a nice feature(maybe controversial): when it loses the focus, it saves all open files in its window.
  • (done)When a region is selected, pressing Backspace or DEL Should delete the region
  • Speedbar should be better too. Refer to the design of textmate.
  • Scroll bar. the vertical bar is not necessary, but there should be some convenient way to scroll horizontally.
  • Different major modes should be able to have different set of live templates.
  • Tabbar
  • Auto completion
  • Navigation
  • Unit test support
  • (done)Move one line upward or downward
  • (done)Duplicate one line
  • (done)Delete one line
  • (done)Comment and uncomment a region or one line

Here are some key bindings that easy to use:

  • <tab>, TAB
  • C-, M-, C-M-
  • C-c … C-x … C-h …

(M here means ‘alt’ key)

some features that are often forgotten in emacs:

  • keyboard macro
  • register
  • bookmark
  • rectangle

The following are some code of those above works that I have marked (done):

(defun tweakemacs-delete-region-or-char ()
  "Delete a region or a single character."
  (if mark-active
      (kill-region (region-beginning) (region-end))
    (delete-char 1)))
(global-set-key (kbd "C-d") 'tweakemacs-delete-region-or-char)

(defun tweakemacs-backward-delete-region-or-char ()
  "Backward delete a region or a single character."
  (if mark-active
      (kill-region (region-beginning) (region-end))
    (backward-delete-char 1)))
(global-set-key [backspace] 'tweakemacs-backward-delete-region-or-char)

(defun tweakemacs-duplicate-one-line ()
  "Duplicate the current line. There is a little bug: when current line is the last line of the buffer, this will not work as expected. Anyway, that's ok for me."
  (let ((start (progn (beginning-of-line) (point)))
	(end (progn (next-line 1) (beginning-of-line) (point))))
    (insert-buffer-substring (current-buffer) start end)
    (forward-line -1)))
(global-set-key (kbd "C-=") 'tweakemacs-duplicate-one-line)

(defun tweakemacs-delete-one-line ()
  "Delete current line."
(global-set-key "M-o" 'tweakemacs-delete-one-line)

(defun tweakemacs-move-one-line-downward ()
  "Move current line downward once."
  (transpose-lines 1)
  (forward-line -1))
(global-set-key [C-M-down] 'tweakemacs-move-one-line-downward)

(defun tweakemacs-move-one-line-upward ()
  "Move current line upward once."
  (transpose-lines 1)
  (forward-line -2))
(global-set-key [C-M-up] 'tweakemacs-move-one-line-upward)

;; There is a bug in the uncomment-region. When you select
;; the last line of the buffer and if that is a comment,
;; uncomment-region to that region will throw an error: Can't find the comment end.
;; Because I use uncomment-region here, so this command also has this bug.
(defun tweakemacs-comment-dwim-region-or-one-line (arg)
  "When a region exists, execute comment-dwim, or if comment or uncomment the current line according to if the current line is a comment."
  (interactive "*P")
  (if mark-active
      (comment-dwim arg)
      (let ((has-comment? (progn (beginning-of-line) (looking-at (concat "\s-*" (regexp-quote comment-start))))))
	(push-mark (point) nil t)
	(if has-comment?
	    (uncomment-region (mark) (point))
	  (comment-region (mark) (point)))))))
(global-set-key (kbd "C-/") 'tweakemacs-comment-dwim-region-or-one-line)