Dev and Test codebase — A part or Apart?
Updated: May 7, 2019
There has been a monumental shift of paradigm from manual to automation with respect to testing a software application. With that, there has been an implicit need for automation frameworks. Conventionally, the popular school of thought has been to maintain 2 separate code bases — development and automation. Developers maintain the former while SDETs look into the latter. Theoretically, it sounds quite apt. A clear demarcation of roles and responsibilities and separation of concerns.
I too was following this norm up till now. Over time I have realised that merging your automation framework with application code is quite beneficiary. To be more specific, all kinds of tests (Unit, Functional, Integration, Regression and even UI tests) can be, and should be, accommodated under the same project as separate modules.
What are the benefits, you ask? Well, here you go :
Shared Resources : Constants and properties of the application code are often duplicated in automation code. Under the same hood, the common resources can be pulled in a separate module and used at both places. I personally find it terrifically cumbersome to change my string-based assertions for every identifier change that the developers would make in the application. This generally leads to false failures and additional debug time. Also, if the project supports localisation, you might have multiple tests pertaining to multiple languages. In the same repository, you can easily test all languages supported without having to duplicate the constants in the automation code.
Test as you code : The biggest advantage of having both dev and test code bases in the same repository — developers don’t have to jump between projects to test their changes. Now every time a developer writes a new piece of code, he would deploy his changes and then run the automation code on the new deployment. This adds additional steps in your development phase. Not to mention the need to jump between 2 projects (or multiple if you have decided to go that way). With the same repo, dev can just run the test module against the latest code piece locally. With this, the responsibility of developer doesn’t just end at Unit tests. He is implicitly getting involved with regression and integration tests. This would also give early feedback about their code and help in catching bugs earlier.
Push the onus on developers : The usual practice has been that developers would write their unit tests and the rest of the tests would be handled by testers. Why the discrimination? The logic that dev would test his/her code with unit tests and the bigger picture would be handled by automation testers because they “specialise” in testing is absolutely bogus. If the developer is writing the UTs, why is he not involved in functional or regression? The framework has been laid down by the automation testers. All they have to do is write your tests on top of it. Also, when there is new code which breaks existing tests (not unit tests), a developer is less likely to fix it if it lies in a different project. In the same repository, a developer would be running these tests themselves and are more likely to fix them on their own. Also, you could enforce git hooks to prevent check-in of code with failing tests.
Better exposure to automation engineers : I have often seen SDETs stay away from application code base. “Not our domain”. “No point going there, we are bothered about automation only”. “I don’t understand the code / it is beyond my comprehension”. The idea is not to have a clear demarcation between these two roles. The idea should be to permeate through the boundaries and jump into the application code itself. It has multi-pronged benefits. It allows automation testers to help out with unit tests. Large picture though, it could be a good learning for SDETs to understand the application code design and eventually start contributing to the dev code itself.
Access to DAO layers : We all agree that UI tests are costly. Yet, we all start there and then gradually shift to lower levels of tests. Having a common project allows us to write tests for functions which might not have public-facing APIs. The DAO layer and business methods can be tested without hitting the UI. All frameworks today provide an in-built server. This enables to spawn an abstraction of your application as opposed to bringing the entire mammoth up. That leads to using your web services for faster and less brittle execution.
These are just some of the benefits I have found by having the automation project as a same entity. However, like any other design approach, this one has cons as well. It would be unfair to not list them. Here’s what you should be cautious about:
Improper separation of concerns : Code pushes could be tricky if you do not have proper separation of concerns in your project. Let’s take a trivial example. Say you merge your automation code with your application code. The way to separate a prod deployment from a QA deployment is some flag in some property file. How long until an engineer pushes QA flag to prod environment? It might sound quite trivial, but a potential pitfall to watch out for.
Change in thought process : Your way of writing tests could change dramatically with automation and application code under the same hood. So, your test cases should be derived from your business logic, and not from your code base. A test derived from code base would never fail. In fact, it could potentially be a false positive. You tend to look into the code to understand the flow and design test cases around it. This thought process could be detrimental to the sanity of your test cases itself. On the contrary, you should think of the business requirement and design test cases around that.
Despite the cons, I am glad to have challenged the age-old belief of separate projects and have got good results off it. I am sure it is not a 1:1 relation when it comes to deciding which approach to go by. But today, this is my go-to approach for building frameworks and you would have to really convince me that separate projects would be better.
If you have differing thoughts or extensions to these, we would love to hear about them. Leave comments to let us know what you think.