Tester - skills needed for successful software testing (2)

We continue exploring the traits needed for testers to be successful in testing software. The previous post looked at traits 1-3. In this post, let us look at three more Software Tester traits.

Tester Trait 4: Perseverance

Testers must keep at testing, exploring and trying to unearth issues. Bugs may appear intermittently or under certain conditions. Testers should not ignore or give up, but instead try different scenarios to reproduce the issue.

Software Testers must also realize that all products have bugs. If a product looks to be free of bugs, it just needs more testing to find issues that current tests haven't looked at. Testers should always be in the pursuit of bugs and view every defect found by a customer as a slip or gap in their tests which must be addressed immediately.

Tester Trait 5: Creativity

Software Testing is an art. It is not enough to test for the obvious. Finding bugs requires creativity and out-of-the-box thinking in testing. Software Testing must be amongst the most creative of professions. Lets make a fairly simplistic comparison between testing software and software development which is considered to be a creative endeavor. Which of these disciplines needs more creativity ? Is it to "introduce" defects or find defects ? While the example is a bit crude, the idea is that it is harder to find defects when you do not know what and how many defects exist. It requires a high degree of creativity to discover defects in Software.

Tester Trait 6: "Flexible" Perfectionists

Software Testing requires the pursuit of perfection. However, the pursuit of perfection must be tempered with flexibility. There are times when perfection may not be attainable or even be feasible. Testers whilst seeking perfection, should adapt a certain degree of flexibility when perfection is not an ideal goal to seek. As an example, when testers report bugs, they must also pursue a fix for the bug. Now, a fix need not just mean fixing the software. It could be a statement in the release notes or other documentation that highlights the defect as a known and open issue; it could be a case of marketing toning down its spiel or enlightening customers about the potential issue - in the real world, it may not be possible to fix every defect that testers want fixed. Being able to prioritize and pick your battles appropriately, knowing when to give in and when to stick to your guns is important.

In the next upcoming post, we look at the remaining traits of a successful tester. Stay tuned.



Tester - skills needed for successful software testing (1)

What are the qualities (a.k.a. attributes, traits, skills) that are required of Testers for testing software ?

In this post, let us look at three Software Tester traits. More to follow in the next post.

Software Tester Trait 1: Curiosity

Software Testing is about exploring and discovering. Software Testers should like exploring, be curious and inquisitive about everything and display keenness in understanding the why and how of products, their interactions, dependencies and ecosystem in totality. Testing software requires venturing beyond the realms of the tried and known, to discover what lies beyond. Installing new software builds, experimenting, seeking to better understand the software and breaking it are part of the skills needed in software testing.

Tester Trait 2: Detail oriented and thorough

Software Testing requires discipline and systematic approach. It is important for testers to be able to pay attention to details and be thorough. While testing involves exploration and experimentation, testing requires that testers not leave any gaps in test coverage. Ensuring that all requirements and areas are thoroughly tested is important. Having an eye for detail in testing would also mean looking out for oddities and incorrect behaviors in the software application being tested. What might seem like a small, insignificant or even irregular occurrence may be the harbinger of much larger issues. It pays to scrutinize each issue thoroughly.

Tester Trait 3: Trouble-shooter

Software Testing requires the ability to root-cause issues. Being good at finding out why something does not work is a useful attribute to possess in testing software. Testers should be able to narrow down the conditions that cause an issue and help identify or at least suggest causes for issues observed. A detailed bug report that lists the issue, narrowed-down steps to reproduce, as well as probable cause along with relevant details can help software development teams address issues faster and better. Also, being able to find out why something does not work can point to more issues lurking around or areas that may need more testing. A tester's job is not just about executing a standard set of software tests and reporting any failures.

We'll look at more traits in the next post.


Article: Software Tools. Build vs Buy



This is a fairly long article that consolidates the common reasons associated with either buying or building software tools. The article also lists the factors to be considered when estimating costs involved with either of the decisions - to build or buy.

Software Tools are of various types and can be used either by a specific function or cross-functionally across the organization. In the case of software testing, Tools is a broad term that encompasses automation and allied testing tools used as part of testing software. These could be defect tracking tools, reporting tools or software test management tools. When an organization or a group within a larger organization decides that it needs a testing tool, a likely question that can crop up would be – can we build it in-house or do we buy a commercial testing tool?

Not every group will have to make this choice. Sometimes, the decision's been made, probably by a central entity who has ordained that a specific tool be used. However, this is a common question encountered by most organizations when they need software tools. Both of the choices above have some common characteristics.
  • Require investment of time, effort and money
  • Have implications that are long lasting
  • Shape the way we do our jobs
While making choices on whether to build or buy software tools, we must weigh our options, not rely on assumptions and make well informed decisions. The repercussions of a choice can be long lasting. There is no such thing as an option that can easily be rolled back in case the choice turns out to be less than expected. In this article, let us examine both the build and buy choices and some of the popular reasons and assumptions around them.

Building a software tool may often seem to be the easier route to take. There does not seem to be a “steep” upfront cost involved as in buying a testing tool. Organizations reason that they can put something together quickly using their available resources. Here are some reasons put forth for building software tools in-house.

Reason: We just need something for the short term, a temporary solution, something quick for the time being. We are hard pressed for time and cannot spend time evaluating a lot of choices. Let us quickly put together something now and later explore more longer term solutions.

We must realize that when we decide to build something, even if it is supposed to be a temporary solution, we will be investing money, time and resources to create, use and maintain the testing tool. The software tools we employ to facilitate our work, will soon mold our work habits around them. Sooner  than later, the software tool will become a part of your regular job and exercise a certain degree of inertia towards any change. When the tool falls short in certain areas, you will (have to) come up with workarounds to continue using the software tool and over time it becomes harder to change established work patterns and software tools learnt.

Lets take the example of a software test automation tool / harness that your in-house resources put together. Once you have reached a certain level of automation using the software tool, how easy will it be to scrap the testing tool and opt for a new one ? If your software tool is lacking in certain features, would you spend the time and effort to enhance the existing tool or would you ditch it to go for an external software tool that offers those extra features ?

In my experience, I have come across software tools that were home-grown and met the basic requirements of the group. However, the software tool paled in comparison to some of the externally available software tools in terms of features and options available. Since the group had invested enough time and resources on the tool, it became hard to let go or even migrate to another software tool without incurring significant additional investments of money, time and resources – which they were reluctant to do.

In essence, you cannot afford to justify a decision based on it's perceived or assumed short-term need. Every decision carries a set of ramifications with it and it is rare that you can easily back out with minimal damage, if things do not work out as expected.

Reason: We have the talent (technical / functional) and can understand our needs better than any external vendor. We can develop a software tool that best matches our requirements,  better than what a commercial software tool vendor may be able to provide.

Whom are we kidding ? Unless we are talking of really simple software tools, the likelihood of internal groups being able to achieve the above and meeting requirements to the desired extent is when the particular internal group that is expected to be building the software tool – has software tools building as it's main responsibility, has demonstrated expertise in successfully building software tools, has the time / money / resources / skills to commit to building a software tool that matches all requirements and can maintain the same on an on-going basis.

While it is easy to commit to developing software tools, it is often hard to cater to all of the requirements and commit to sustain/maintain the software tool for the longer term. Most commercial software tool vendors have specialized groups dedicated to developing specific software tools and have made large investments towards building tools. These software tool vendors have built the expertise to address challenges and have the resources available to support the software tool on an on-going basis. Usually, when in-house software tools are developed, it is the case that a set of engineers are assigned to work on the software tool along with their regular tasks. Divided attention is hardly beneficial. If the software tool developers have pressure to work on other tasks, they can easily lose focus and attempt to cobble up a software tool that at best, barely meets requirements. When developers are assigned to work full time on creating software tools, unless they have the aptitude and skills required – they can give up soon or even look at cutting/watering down requirements to complete their assignment and move on to other projects. Internal groups often face the challenge of maintaining and enhancing the software tools that are developed. Another element to consider while choosing to develop software tools in house is software testing. Like any software product that is developed, the software tool too must go through the same standard development processes and test efforts to be useful, which can prove to be a significant added investment.

So, unless you are willing to commit the resource (time, money, effort) to develop, test and sustain a software tool, it might not be a great idea to build it in-house.

Reason: Building a software tool in-house saves a lot of money when compared to purchasing the tool externally. Since we are using our available resources, things would work out much cheaper than buying a software tool.

How easy is it to estimate costs accurately for a development project ? Try estimating the costs for your planned software tool development project. A fairly detailed list of items to consider when estimating the cost of building a software tool, is provided towards the end of this article.

Include the cost of resources involved in software development (gather requirements, analyze, design, implement, localize if needed, migrate data), documentation, software testing (unit testing, functional testing, performance testing, etc), training users, performing on-going maintenance and enhancements of the software tool. A common perception when developing in-house software tools is that users of the software tool will be more forgiving and adjusting of the tool's failings and they would have reduced expectations from the software tool. The fact is – users not only demand similar standards as external software tools, they generally have more (rather than less) expectations from the in-house developed software tool. After all, the developers are in-house and accessible. We can always request for customizations and changes whenever we want.

Cost calculations must also consider the opportunity cost involved with having both Developers & Management staff working on the software tool development project vs working on other revenue opportunities. We must not under-estimate the cost of software tool maintenance. In many cases, on-going maintenance costs (defect tracking, regular updates / patch releases, support) can turn out to be a significant recurring investment similar to the licensing costs involved when buying a software tool.

Basing the software tool build vs buy decision purely on cost is not really a feasible option. What may seem to be a less expensive choice could turn out to be a less than feasible alternative.

Reason:  We should build it since we just need a simple solution which our engineers should be able to quickly put together; our software testing requirements aren't too complex; we know what we need and figure it should not take much time or effort to build a  tool.

Organizations find it tempting to put together or build simple software tools to address a specific problem or area without considering, at the risk of making a cliched statement, the “larger picture”. If your software tool is not easily enhance-able/maintain-able and you are limited to the functionality you built-in initially, you can be sure that sooner than later, your needs will outgrow what your software tool offers. The tool will soon hinder than aid your efforts. I have seen different groups create such “simple” software tools to address specific pain points. Over time, various such “simple” software tools were created and put to use by different groups. Since little thought was given to integrating the various software tools or making them work together, the proliferation of these “simple & standalone” software tools posed a challenge to the organization's productivity.

From experience, such software tools tend to get discarded after some amount of usage and a suitable replacement is found. However, there are cases where these software tools continue to be used, since the organization has already invested significantly in the existing software tool and may not be willing to incur the additional costs associated with a migration to a new software tool.  In either case, the organization has already incurred costs on the simple software tool and will incur more costs to procure and move to a replacement tool. It would be better to consider the larger picture when making software tool decisions. Look beyond the confines of your group; examine (best) practices from other groups and software tools they use; explore any possible changes to your current processes/additional requirements for the software tool you intend to build and consider if your software tool can handle them.

It need not just be new or changed requirements that can make your software tool redundant. Simple scalability issues can cause your simple software tool to pose complex issues. If your tool is designed to handle the needs of your current group, consider if the same tool can handle the requirements of a larger group - will your software tool then scale or crawl ?

Does your software tool consider all audiences who might potentially use/want/need to use it? Some software tools are created to solve pain points that a section of engineers may encounter. Their usage, output and reporting are often put together to provide the basic information that those engineers would need. However, when the same software tool needs to provide information to a wider audience or someone in the business side of things, a different usage and reporting paradigm emerges. If the software tool cannot meet the needs of the different audiences, current as well as potential future ones, it can again be a drag. What if the products you are working on changes significantly so that the tool also needs to be changed to handle newer operational/functional needs?  Can your software tool adapt to changes?

Reason: Software Tools we build can be fully customized when compared to commercial software testing tools

This is one argument that's hard to beat. It is true that software tools you build will be more customizable than anything you can get externally.

However, that is just one part of the equation. We need to look at the emphasis on customization. Beyond a reasonable degree, the need for customization may be viewed as a factor emerging from an organization's fluid and yet-to-be-clearly-defined set of methods and processes that makes defining a set of requirements for a software tool, hard. The preference in such as situation is to go with a software tool that can theoretically be customized infinitely !

While the ability to customize is very good and useful, it must be viewed in the light of costs involved and time to deploy the software tool. Here's a very simplistic example – let us assume that your organization wants a GUI test automation tool. One possibility is to buy a commercial tool such as SilkTest or WinRunner. The other option is to build a software tool that is similar to these software tools but can be customized to any degree you would like.

Here, you need to consider the following.
  • The costs involved – you might end up spending huge sums of money (many times more than it would cost to buy licenses) to build such a software tool and much more to maintain it for as long as your organization needs to use it
  • The time to deploy – assuming your organization is willing to invest the money, an important factor to consider would be the time to deploy the tool. Building a software tool from scratch takes time, lots of it in fact. Factor this in while making your choice
Having looked at some of the common reasons stated in favor of building software tools, let us examine some of the common reasons stated in favor of buying software tools.

Reason: Buying will lead to faster deployment of the tool; a software tool that is purchased will be faster to deploy.

Yes, when you consider that buying a software tool cuts out the development costs involved in building it. Yet, the extent of quickness in deployment depends on your particular situation. 

There still needs to be due diligence towards selecting the right software tool. Once that is done and a software tool is procured, any needed customizations are to be performed. The software tool then needs to be setup and configured, following which a detailed migration exercise has to be performed to move away from the existing software tools and users need to be trained to work with the new tool. Time taken to accomplish the various tasks involved in deployment is also dependent on the extent of complexity. A factor to consider would be the human acceptance of the software tool. Just because a new software tool is procured and setup, users & others associated with the roll-out will not automatically accept it. When a new software tool replaces existing software tools that are being used widely, you can expect some degree of resistance to the new tool. Comparisons are inevitable and in the initial days there are bound to be comments on the new tool vis-a-vis the existing software tools. Despite any shortcomings of an existing tool, users can be reluctant to adopting a new software tool; more so, if they were involved in or had a stake in developing the existing tool. Customization, migration and acceptance are chief areas that can consume a lot of time.

I have come across teams that were reluctant (resistant would be more like it) to adopting a new software tool. The new software tool could potentially help overcome drawbacks which the existing software tool had. But folks had gotten used to the current software tool, warts and all of it and were pushing back wider implementation of the new tool. They would find ways to delay implementation and after several deadlines for deployment had passed, the organization settled for a dual systems approach (use the old software tool for some tasks and the newer one for some others). In some organizations, the transition to the new tool does happen but at a much later date than what was targeted.

The above factors will influence the amount of time it takes to deploy the software tool you buy. Will it be faster than building a software tool ? There is no definite answer. In some cases, it might be faster to build a software tool while in other situations, buying a software tool and deploying it will be quicker.

Reason: The software tool we buy will be maintenance-free – after all that's what the vendor claimed. All we need to do is to buy the software tool, pay up the annual maintenance fees and bingo, the vendor will take care of the rest.

Wishful thinking ! There's hardly any software tool of reasonable complexity that is maintenance-free. Maintenance includes the effort required from the organization's side towards setting up the tool, configuring it and regularly performing administrative activities. 

If customizations have been performed, we can expect additional maintenance requirements. This is more so when updates/patches/upgrades to the software tool are received. Applying an update or upgrading to a later version could involve significant work including backing-up and re-applying any customizations, testing and re-deploying the tool. Even when there are no customizations, moving to a later version can pose compatibility issues and requires testing before deploying widely. While the vendor personnel may offer support, the organization will also need its own resources to work with the vendor on software tool maintenance activities and support.

Summing up, there are situations where it makes more sense to build your software tool, while there are situations where it might be better to go out and buy a software tool and customize it if need be. Sometimes, it can be hard to find software tools externally that meet your specific requirements. With software tools you buy, you will need to adapt to the tool and the way it works. With a software tool that you build, you can integrate it with your current way of working.

Irrespective of the choice you make – to build or buy – there are aspects that you must consider while estimating costs involved with either decision. Given below, is a listing to help in calculating costs associated with building and buying software tools.

1. Costs to consider while deciding to build a software tool in-house.
  • Administrative costs involved in Defining the problem
    • Collecting requirements
    • Development environment and method selection
    • Developing and administering training for users
    • Managing the project and people involved
  • Production costs involved in
    • Planning the Tool development project
    • Architecture and HLD (High Level Design)
    • Detailed Design
    • Coding / Implementation
      • Client / Web
      • Server / Back-end infrastructure
      • Module integration
      • Migration
    • Software Testing
      • Unit
      • Integration
      • System
      • Performance
      • Acceptance
    • Debugging
    • Documentation
    • Deployment
  • Support Costs involved in
    • Ongoing maintenance activity
    • Developing and Delivering upgrades/patches
    • Defect tracking and reporting
    • Ongoing support activities, such as running a help desk or similar mechanism
In addition, the opportunity costs involved must be considered. The cost of time spent by employees (technical and business staff) along with any delays in schedules must be taken into account.

2. Costs to consider while deciding to buy a software tool.
  • Administrative costs involved in
    • Defining the problem
    • Collecting requirements
    • Evaluation of various tools, Proof-of-Concept and Selection
    • Managing Vendors
    • User training
  • Procurement costs involved in
    • Cost of licenses
    • Customizations
    • Cost of tool deployment – setup, configuration, integration
    • Tool testing – integration, compatibility, performance, acceptance
    • Cost of migration
  • Support Costs involved in
    • Annual Maintenance costs
    • Costs involved to upgrade across versions
    • Additional internal support costs
Also, as in the case of building a software tool, the opportunity costs involved must be considered. The cost of time spent by employees (technical and business staff) in the above activities along with any delays in schedules must be taken into account.

Stay tuned for more upcoming posts on various facets of Software Testing, QA and Development.

Costs involved in building vs buying tools

Continuing from our previous posts on the subject of build vs buy decision while choosing testing tools, listed here are the costs involved when deciding to either build or buy a tool in-house.

1. Costs to consider while deciding to build a tool in-house.
  • Administrative costs involved in
    • Defining the problem
    • Collecting requirements
    • Development environment and method selection
    • Developing and administering training for users
    • Managing the project and people involved
  • Production costs involved in
    • Planning the Tool development project
    • Architecting and HLD (High Level Design)
    • Detailed Design
    • Coding / Implementation
    • Client / Web
      • Server / Back-end infrastructure
      • Module integration
      • Migration
    • Software Testing
      • Unit
      • Integration
      • System
      • Performance
      • Acceptance
    • Debugging
    • Documentation
    • Deployment
  • Support Costs involved in
    • Ongoing maintenance activity
    • Developing and Delivering upgrades / patches
    • Defect tracking and reporting
    • Ongoing support activities – such as running a help desk or similar mechanism
We must also consider the opportunity costs involved. The cost of time spent by employees (technical and business staff) along with any delays in schedules must be taken into account.

2. Costs to consider while deciding to buy a tool.
  • Administrative costs involved in
    • Defining the problem
    • Collecting requirements
    • Evaluation of various tools, Proof-of-Concept and Selection
    • Managing Vendors
    • User training
  • Costs related to Procurement
    • Cost of licenses
    • Cost of Customizations
    • Cost of tool deployment – setup, configuration, integration
    • Tool testing – integration, compatibility, performance, acceptance
    • Cost of migration 
  • Support Costs involved in
    • Annual Maintenance costs
    • Costs involved to upgrade across versions
    • Additional internal support costs
Also, as in the case of building a tool, the opportunity costs involved must be considered. The cost of time spent by employees (technical and business staff) in the above activities along with any delays in schedules must be taken into account.

Next up is a detailed article covering the popular myths and reasons behind making the build vs buy choice in testing tool selection. Stay tuned.

Selecting a testing tool (6)


Having looked at some of the common reasons stated in favor of building tools in-house, let us now look at some of the reasons ascribed in favor of buying tools.

Reason – Buying will lead to faster deployment of the tool; a tool that is purchased will be faster to deploy.

Yes, when you consider that buying a tool cuts out the development costs involved in building it. Yet, the extent of quickness in deployment depends on your particular situation. 

There still needs to be due diligence towards selecting the right tool. Once that is done and a tool is procured, any needed customizations are to be performed. The tool then needs to be setup and configured, following which a detailed migration exercise has to be performed to move away from the existing tools and users need to be trained to work with the new tool. Time taken to accomplish the various tasks involved in deployment is also dependent on the extent of complexity. An important factor to consider would be the human acceptance of the tool. Just because a new tool is procured and setup, users & others associated with the roll-out will not automatically accept it. When a new tool replaces existing tools that are / were being used widely, you can expect some degree of resistance to the new tool. Comparisons are inevitable and in the initial days there are bound to be comments on the new tool vis-a-vis the existing tried and tested tools. Despite any shortcomings of an existing tool, users can be reluctant to adopting a new tool; more so, if they were involved / or had a stake in developing the existing tool. Customization, migration and acceptance are chief areas that can consume a lot of time.

I have come across teams that were reluctant (resistant would be more like it) to adopting a new tool. The new tool would technically help overcome drawbacks which the existing tool had – but folks had gotten used to the current tool, warts and all of it and were pushing back wider implementation of the new tool. They would find ways to delay implementation and after several deadlines for deployment had passed, the organization settled for a dual systems approach (use the old tool for some tasks and the newer one for some others). In some organizations, the transition to the new tool does happen but at a much later date than what was targeted.

Ultimately, the above elements will influence the amount of time it takes to deploy the tool you buy. Will it be faster than building a tool ? There is no definite answer. In some cases, it might be faster to build a tool while in other situations, buying a tool and deploying it will be quicker.


Image courtesy: Salvatore Vuono / FreeDigitalPhotos.net

Selecting a testing tool (5)

Here's the fifth part in this series of posts examining the buy vs build choice in test tool selection. Let us look at -
 

Reason 5 to build a tool in-house – Tools we build can be fully customized when compared to commercial software testing tools.

This is one argument that's hard to beat. It is true that tools you build will be more customizable than anything you can get externally.

However, that is just one part of the equation. We need to look at the emphasis on customization. Beyond a reasonable degree, the need for customization may be viewed as a factor emerging from an organization's fluid and yet-to-be-clearly-defined set of methods and processes that makes defining a set of requirements for a tool, hard. The preference in such as situation is to go with a tool that can theoretically be customized infinitely.

While the ability to customize is very good and useful, it must be viewed in the light of costs involved and time to deploy the tool. Here's a very simplistic example – let us assume that your software testing group wants a GUI automation tool. One possibility is to buy a commercial test tool such as SilkTest or WinRunner. The other option is to build a testing tool that is similar to these testing tools but can be customized to any degree you would like.

The points to consider would be - 1) the costs involved – you might end up spending huge sums of money (many times more than it would cost to buy licenses) to build such a test tool and much more to maintain it for as long as your organization needs to use it 2) The time to deploy – assuming (for some weird reason) your organization is willing to invest the money, the other important factor to consider would be the time to deploy the tool. Building a tool from scratch takes time, lots of it in fact. Factor this in while making your choice.

I must repeat that I am not taking sides here, nor am I suggesting you choose to either build or buy tools. The attempt here is to present various perspectives that can help you make an informed choice. There are situations where it makes better sense to build your own software testing tools, as there are situations where it makes sense to go out and buy a tool and if need be customize to your needs. In certain cases, it can be hard to find tools that meet your specific requirements. Also, with any commercial tool, you will have to adapt to the tool and the way it works. With a tool that you build, you can integrate it with your current way of working.

We continue our exploration of the build vs buy choice in upcoming posts. Stay tuned.


Image courtesy: Suat Eman / FreeDigitalPhotos.net

Selecting a testing tool (4)

Continuing from our earlier posts that looked at the common reasoning behind the question of whether to build or buy testing tools, let us now look at -

Reason 4 to build a testing tool in-house - We should build it since we just need a simple solution which our engineers should be able to quickly put together; our software testing requirements aren't too complex; we know what we need and figure it should not take much time or effort to build a testing tool


Organizations find it tempting to put together or build simple test tools to address a specific problem or area without considering, at the risk of making a cliched statement, the larger picture. If your test tool is not easily enhance-able / maintain-able and you are limited to the functionality you built-in initially, you can be sure that sooner than later, your needs will outgrow what your test tool offers - the test tool will soon hinder than aid your efforts. I have seen various groups create such “simple” tools to address specific pain points. Over time, various such “simple” tools were created and put to use by different groups. Since little thought was given to integrating the various tools or making them work together, the proliferation of these “simple & standalone” tools posed a significant handicap to the organization's software testing efforts.

From experience, such test tools tend to get discarded after some amount of usage and a suitable replacement is found. However, there are cases where these tools continue to be used – the organization has already invested significantly in the existing test tool and may not be willing to incur the additional costs associated with a migration to a new tool.  In either case, the organization has already incurred costs on the simple tool and will incur more costs to procure and move to a replacement tool. It would be better to consider the larger picture when making tool decisions. The test software (testing tools) should enable your organization to perform better software testing. Look beyond the confines of your group; examine (best) practices from other groups and tools they use; explore any possible changes to your current processes / additional requirements for the tool you intend to build and consider if the test tool can handle them.

It need not just be new or changed requirements that can make your tool redundant. Simple scalability issues can cause your simple tool to pose complex issues. If your tool is designed to handle the needs of your current group, consider if the same tool can handle the requirements of a larger group - will your tool scale or crawl ?

Does your tool consider all audiences who might potentially use / want / need to use it ? Some tools are created to solve pain points that a section of engineers may encounter. Their usage, output and reporting are often put together to provide the basic information that those engineers would need. However, when the same tool needs to provide information to a wider audience or someone in the business side of things, a different usage and reporting paradigm emerges. If the tool cannot meet the needs of the different audiences - current as well as potential future ones, it can again be a drag. What if the products you are testing changes significantly so that the tool also needs to be changed to handle newer operational / functional needs ?  Can your tool adapt to changes ?

We'll continue looking some more at the build versus buy choice for testing tools in this series of posts. Also, coming up is a comprehensive article on making testing tool decisions, listing some questions you need to ask and evaluate before making your decision of whether to build or buy a tool for your testing needs. Stay tuned. 

Image courtesy: Danilo Rizzuti / FreeDigitalPhotos.net

Selecting a Test Tool (3)

Continuing from our earlier posts that looked at the common reasons behind the question of whether to build or buy tools … lets now look at -

Reason 3 to build a tool in-house - Building a tool in-house saves a lot of money when compared to purchasing the tool externally. Since we are using our available resources, things would work out much cheaper than buying a tool


How easy is it to estimate costs accurately for a development project ? Try estimating the costs for your planned tool development project. Include the cost of resources involved in developing (gather requirements, analyze, design, implement, localize if needed, migrate data) the tool, documenting, testing (unit, functional, performance, etc), training users and performing on-going maintenance and enhancements of the tool. A common perception when developing in-house tools is that users of the tool will be more forgiving and adjusting of the tool's failings. Also, users might have reduced expectations from the tool. The fact is – users not only demand similar standards as external tools, they generally have more (rather than less) expectations from the in-house developed tool. After all, the developers are in-house and accessible. It is not hard to reach them and we can get them to add in a feature or customize something as we want it. Developer not willing to do it ? Lets go through the right channels / hierarchy and request for changes.

Cost calculations must also consider the opportunity cost involved with having both Developers & Management / Business staff working on the tool development project vs working on other revenue opportunities. We must not under-estimate the cost of tool maintenance. In many cases, on-going maintenance costs (defect tracking, regular updates / patch releases, support) can turn out to be a significant recurring investment similar to the licensing costs involved when buying a tool.

Basing tool decision purely on cost is not really a feasible option. What may seem to be a less expensive choice could turn out to be a less than feasible alternative. Focus on what provides maximum benefits and meets most requirements. 

CU3HFAVW6RFR
Image courtesy: graur razvan ionut / FreeDigitalPhotos.net
CU3HFAVW6RFR

Selecting a Test Tool (2)

Continuing our look at the factors contributing to the build vs buy decision making process, let us now look at -

Reason 2 to build a tool in-house – We have the talent (technical & functional) and understand our requirements / needs better than any external vendor. We can develop a tool that best matches our needs and better than what a commercial tool vendor may be able to provide.


Whom are we kidding ? Unless we are talking simple tools, the likelihood of internal groups being able to achieve the above and meeting requirements to a “satisfactory” extent is when the particular internal group that is expected to be building the tool – has tool building as it's main responsibility, has demonstrated expertise in successfully building tools, has the time / money / resources / skills to commit to building a tool that matches all requirements and can maintain the same on an on-going basis.

Why are these requirements needed of internal groups ? While it is easy to commit to developing tools, it is often hard to cater to all of the requirements and commit to sustain / maintain the tool for the longer term. Most commercial tool vendors have specialized groups dedicated to developing specific tools and have made large investments towards building tools. These tool vendors have built the expertise to address challenges and have the resources available to support the tool on an on-going basis.

Usually, when in-house tools are developed, it is the case that a set of engineers are assigned to work on the tool along with their regular tasks. Divided attention is hardly beneficial. If the tool developers have pressure to work on other tasks, they can easily lose focus and attempt to cobble up a tool that at best, barely meets requirements. When developers are assigned to work full time on creating tools, unless they have the aptitude and skills required – they can give up soon or even look at cutting / watering down requirements to complete their assignment and move on to other projects. Also, internal groups often face the challenge of maintaining and enhancing the tools that are developed. One other factor to consider while choosing to develop tools in house is testing. Like any product that is developed, the tool too must go through the same standard development processes and test efforts to be useful.

So, unless you are willing to commit the resources (time, money, effort) to develop, test and sustain a tool, it might not be a great idea to build it in-house. Now before you start thinking that i am trying to discourage any in-house development activity, let me state that the intent is to present some common reasoning from both sides of the fence (in-house vs external). I'll cover more reasons in subsequent posts. Stay tuned.


Image courtesy: Michal Marcol / FreeDigitalPhotos.net

Selecting a Test Tool


Test tool is a broad term that encompasses test automation tools and other tools used as part of testing. These could be defect tracking tools, reporting tools or test management tools. When an organization or a group within a larger organization decides that it needs a test tool, a likely question that can crop up would be – can we build it in-house or do we buy a commercial tool ?

For some groups, the decision's already made. There might be a centralized entity that has ordained that a particular tool or framework be used and all other groups simply toe the line. But what would you do if you are making a fresh choice ? Would you have to face the above question of build vs buy ?

Lets look at a few reasons why organizations might make either choices – build the tool in-house or buy the tool. First, why would organizations choose to build a tool.

Reason 1 – Short term solution

We just need something for the short term, a temporary solution, something quick for the time being. We are hard pressed for time and cannot spend time evaluating a lot of choices. Let us quickly put together something now and later explore more longer term solutions. Any of these sound familiar ? Looking for short term solutions ?

We must realize that when we decide to build something, even if it is supposed to be a temporary solution, we will be investing money, time and resources to create, use and maintain the tool.

The tools we employ to facilitate our testing, can soon mould our work habits around them. Sooner than later, the tool will become a part of your regular job and exercise a certain degree of inertia towards any change. When the tool falls short in certain areas, you will (have to) come up with workarounds to continue using the tool and over time it becomes harder to change established work patterns and tools learnt.

In most cases, once employed, a tool becomes hard to change. Lets take the example of an automation tool / harness that your in-house resources put together. Once you have reached a certain level of automation using the tool, how easy will it be to scrap the tool and opt for a new one ? If your tool is lacking in certain features, would you spend the time and effort to enhance the existing tool or would you ditch it to go for an external tool that offers those extra features ?

In my experience, I have used tools that were home-grown and met the basic requirements of our group. However, the tool paled in comparison to some of the externally available tools in terms of features and options available. Since we had invested enough time and resources on the tool, it became hard to let go or even migrate to another tool without incurring significant additional investments of money, time and resources – which we were reluctant to do.

In essence, you cannot afford to justify a decision based on it's perceived or assumed short-term need. Every decision carries a set of ramifications with it and it is rare that you can easily back out if things do not work out as expected. We'll explore more on the question of build vs buy in subsequent posts. Stay tuned and send in your feedback using the "Contact" link at the top of this blog.

Image courtesy: Simon Howden / FreeDigitalPhotos.net

Download a Free Copy of the Test Automation article here



I have put together the content from my posts related to the subject of "Pitfalls of Test Automation" into a handy PDF document.

Download it freely - here.

Who automates your tests ?

Whom do you assign to work on automating your tests ?

Ideally, a person who is chosen to automate tests should have sound development and testing skills. Often times, people are chosen who might have expertise in either of these areas. This can be worked upon. If your test automation engineer is predominantly from a testing background and lacks development skills, you need to check if this person can pick up the required development skills and demonstrate the necessary rigor and discipline.

Generally, test engineers who desire to be programmers, jump at the opportunity to automate tests. This provides them an opportunity to develop their programming skills. While doing this, one must make sure that these engineers are well grounded on the automation requirements and focus on delivering them rather than getting carried away with the programming task. Sometimes, you may not need to automate all your tests given the available time and resourcing. Here, the ability to focus on those areas that can provide maximum return on investment is important.

A reason for requiring development skills is the need for focus on aspects such as maintainability, re-usability, reliability and importantly, the ability of the test automation suite to easily adapt to changes to the system under test. Test automation needs to be approached with the same rigor and commitment as a regular software development project.

Sometimes, developers get assigned to do test automation. In this case, an evaluation needs to be done to ascertain if this person can gain understanding of the testing discipline and develop a healthy respect for the function. When in-house engineers aren't available, it may be the case that contractors are employed to perform test automation. The advantage here is that you are likely to get engineers with matching skill-sets. However, one must be aware of & take appropriate steps to mitigate any maintenance challenges once the contractors have departed. Having a sound automation framework in place and reviewing conformance to stated standards and guidelines can be useful.

In some groups, the new college graduates, interns or junior programmers are assigned to work on automating tests. The argument may be that these folks have spent a lot of time learning about programming in college and would be able to apply their “fresh” learnings to automating tests which is a type of programming activity. Also, these new comers often are glad to have the chance to demonstrate their competence and creativity and show interest. In some cases, it may be that the management wants to keep these junior folks engaged on some relatively lower risk activity and hence assigns automation to these engineers. If what they develop, works – good and if it doesn't it can be shelved with low cost incurred. The new comers often are not well trained on the testing function and often end up using their programming skills to creating complex test automation suites that can be hard to maintain without meeting testing requirements.

Worser still, is the attempt to assign automation to the “lower-performers” from development or testing. These folks have poor skills, commitment and motivation in either area. The chances of success in this case is very low. Test automation requires skilled, committed and motivated folks who can work effectively with both development and testing teams.

Test Automation Pitfalls (3)

Continuing from the previous posts, we look at another factor contributing to less than effective test automation.

3. Stability of the system being tested and the ability of the automation suite to adapt to changes to the system

There are two parts to this – one, the system or application that is being tested needs to be in a relatively stable state before you embark on an extensive automation exercise. If you try to build your automation suite when the system on which these automation tests are based upon keeps changing frequently, you are guaranteed to encounter – false test results being reported by automation runs and ever increasing maintenance activities on your suite. We must however, realize that despite calling for “relative stability”, there will be some changes and any test automation exercise must accept and be open to some amount of change. Having a stable product can help minimize these changes.

This is where we look at part two, which is the ability of the test automation suite to adapt to changes to the product it is testing. When test automation is not based on a sound framework that can minimize the impact of changes to the system under test, extensive maintenance activities need to be performed. This can have the added effect of affecting stakeholder perception of both the reliability and usefulness of your automation suite.

From a process stand-point, when major changes are planned to be made to the system under test, these must be reviewed to evaluate their impact on existing test suites. Any resulting test automation activity must be planned and communicated appropriately.

Test Automation Pitfalls (2)


Continuing from the earlier post on the subject, lets look at another factor that leads to less than effective test automation.

2. Automation Framework and Process.

For many groups, test automation begins with the acquisition of a tool and from thereon it proceeds towards a race to automate as many tests as quickly as possible. Everyone's excited about the tool and its features, so much so that each one lets loose their creative energies on automating their set of tests quickly. During this time, some attributes of test automation such as – reusability, maintainability, scalability, separating data from the automated tests and such others are often ignored. These attributes unfortunately, are required for a robust and maintainable test automation suite.

An oft repeated aspect of test automation is that you must approach test automation just as you would a product that your organization produces. This implies that the same principles and practices need to apply to test automation; the same level of detailed planning, designing, architecting, process definitions, reviews, etc. need to apply. Test automation is software that you are developing and a reason that automation can fail is because most organizations do not treat the test software they develop with the same level of care and commitment as they do to the products they sell.

Dis-organized test automation or automation done with little attention to the required attributes, results in a suite that can be a maintenance nightmare. Groups soon realize that they need more resources to both maintain the automated suites as well as to automate newer tests, resulting in ever increasing costs and little added benefit.

It will be useful to remember that all automation needs to be maintained or abandoned. Make sure your automation is maintainable easily at low cost.

We'll continue examining additional factors in the next post.


Image courtesy: Francesco Marino / FreeDigitalPhotos.net

Test Automation Pitfalls (1)

Organizations make significant investments in Test Automation. However, few of them realize corresponding benefits from their investments.

Often, automation suites turn out to be expensive to create and subsequently maintain without being  effective in unearthing defects. The natural question that gets asked is – Why ?

There are various factors, some of which include the following.

1.    Quality of tests being automated.

"Garbage in, Garbage out” applies well to test automation. If your test automation is based on tests that are themselves of poor quality, then you can rest assured that your automation results will hardly be spectacular.

While analyzing issues with test automation, we must examine the underlying tests and their design. For most organizations, their test automation goal is to automate 100% of their existing manual tests. While it may be a challenge to reach this goal, especially in larger and complex projects, the usefulness of such a goal is debatable. Unless you have a good set of tests to automate, merely trying to convert all of your manual tests to automated ones, does not offer much benefit. The exercise could very likely incur significant investments of time and resources while yielding little effective returns.

Quoting Mark Fewster - “It doesn’t matter how clever you are at automating a test or how well you do it, if the test itself achieves nothing then all you end up with is a test that achieves nothing faster.

We will look at more factors in upcoming posts. 


Enterprise Application Development - changes needed




While reading a recent Forrester research report on - The Top Five Changes For Application Development, I was attracted towards the suggestion that enterprises re-invent themselves to think and behave like startups.

In the changed business environment following the recessionary times we've been through, it becomes even more important for enterprises to learn from startups. The need to be nimble, quick and able to satisfy shifting customer requirements, rapidly respond to competition, quickly change tracks when things are not working, slow down when business is slow but move swiftly when the “tipping point” arrives – all attributes that enterprises, should pickup from their startup brethren.

Other significant attributes of startups that enterprises would do well to learn include the following.
  • For your developers to know the inner workings of your business and understanding what customers want, what competitors are doing, and what it will take to create a great user experience. Given that startups have very little resources, they have to be focussed on what will give the maximum return on investment, which in turn translates to doing exactly those activities that can be linked to their ability to generate revenue.
  • Discard wasteful, redundant and non-value-add process steps from your Development process. Continuing from the earlier point about startups being business focussed, we also observe that startups do not adopt elaborate development processes or expensive tools. Their focus is on having just-enough processes and tooling to deliver solutions asap. Enterprises can pick up a lesson or two from startups here and challenge the status quo in their organizations by questioning the need for heavy-weight processes vis-a-vis their value proposition.
  • Incorporate Agility into your DNA. Like startups who are glued to their customer requirements and competition, both of which have a common attribute – change, it is important for enterprises to align process, methods, resources and the organization itself towards agility. Being able to discard methods or even projects that no longer make business sense, being able to rapidly respond to changing customer requirements and market conditions while realizing that your organization's course is not set in stone, is key. It might make for sound business sense to explore Agile development practices for your organization.
  • Do more with less. It is easy to focus on doing one of them – either more or less. The challenge is doing both; do more with less … also known as being “Lean and Mean”. This isn't a totally new attribute but rather an extension of the previously mentioned startup attributes – focus on what is important and smartly organize all your resources towards projects that have the most business impact, discard waste and be focussed on value generation, ensure your developers know the business thoroughly and understand what matters most in a project – technical genius is useless if the user experience of your product or service is bad.
The quote by Niccolò Machiavelli: "Never waste the opportunities created by a good crisis", is very apt. The recessionary times have reinforced the theory of survival of the fittest. Organizations will need to adopt practices that will help deliver better, quicker and cheaper – while meeting customer requirements, reducing costs, innovating and  responding to market opportunities quicker than the competition.

Image courtesy: djcodrin / FreeDigitalPhotos.net

Hashtags in Twitter



Today's blog entry is about the use of hashtags in twitter. Hashtags are reported to be amongst the most confusing aspects of twitter. It is however, a simple concept and definitely worth understanding.

Twitter does not yet, have a way to classify messages into categories. For example, there isn't a way to mark a bunch of messages as belonging to a specific category or event. To workaround this, twitter users use hashtags. A hashtag is basically a term prefixed with the # symbol. Example - #tech, #news, #anything …

Software professionals call the '#' symbol as a hash. When the hash symbol is combined with a term which is used to tag twitter messages, it is called a hashtag. When an user wants to classify related messages around an item, say news about apple, the user might come up with a hash tag as #apple and put it in their message. Then other users add the same hashtag to their messages about the similar subject. You can search messages based on hashtags.

While hashtags help identify events, they can also be used to denote messages from specific groups of people. People of a particular community might want to tag their messages with a common hashtag so it is easily identified and search-able. Some hashtags can be used to cross-post messages to both twitter and non-twitter applications such as Facebook (using hashtag “#fb”), Linkedin (using hashtag “#in”), etc.

To find messages classified with a particular hashtag, visit  http://search.twitter.com and query for the hashtag. Hashtags can be used for various purposes. A few tips are listed below.
  • Use to collect ideas / crowd-sourcing – you can ask questions on twitter and give a hashtag that other twitter users can use to tag their answers to your question. That way you can quickly identify the responses
  • Use to have a group discussion – you can have folks located anywhere on the planet, participate in discussions. Put out a hashtag that folks participating in the discussion can use. Formal discussions may be arranged in advance to occur at a specific time.  There are third party tools that allow you to collect messages that use the particular hashtag and aggregate them in the form of a conversation
  • Use to share thoughts / experiences on any subject – unhappy about a product or service you received ? Use a hashtag along with your message. Generally, many people would be participating / using most common hashtags. You might want to check if a hashtag is being used on twitter search and also the conversations around those hashtags
Hopefully, this helps clarify the use of hashtags in Twitter.

Cloud computing - offline




With all the talk around cloud computing, a general assumption seems to be the availability of internet connectivity and uninterrupted availability of the cloud. Cut to the real world – where interrupted connectivity is a given and even the best of hosted clouds fall short of 100% availability. What would users of cloud based apps and services do in such situations ?

In my earlier post I was highlighting the usefulness of a cloud based suite of office productivity applications (such as Google Docs) and how they could either complement or in some cases replace traditional desktop software. There are many useful cloud based apps which can enable users to migrate from the traditional model of installing apps on their local machines and maintaining copies of their data at each location where they need to access it.

Any move to a cloud computing paradigm must consider the requirements for “offline” access to cloud resources – would users require offline access to their cloud applications and also understand the fact that not all applications lend themselves smoothly to offline access.

Looking back at the example of cloud based office applications (Google Docs) or even cloud based communications apps such as email and calendar, there are tools to enable offline access for these applications. Technologies such as Google Gears, facilitate this offline access. There are many real life use cases where such offline access is required – allowing users to, for example, download their emails & calendar events to a local cache while working offline and later uploading changes when connected back to the cloud.

However, there are applications where such offline access can pose significant challenges. Take the case of large distributed ERP applications that are hosted on a cloud wherein some users are working online (connected to the cloud) and some are working offline with a snapshot of the data downloaded to their devices. Issues around data integrity can quickly arise when folks who are offline make changes to data / create new entries and later try to sync back to the cloud while the folks who are already on the cloud have made other changes to the same set of data. Take a simple example of a salesman on the field who uses the offline data that shows stock available of a particular product and then commits to the customer to deliver it immediately. However, another sales person who is on the cloud has also made a sale of the same product to another customer and now there is no stock available. The sales person who is offline will later try to sync back to the cloud and realize that delivery as per commitment would not be possible. Again, looking at the same sales person who is offline and generates an invoice with the invoice number being auto generated based on the last invoice number. If folks on the cloud have also simultaneously been generating invoices, we can easily end up in a conflict situation where the same invoice number could be assigned to multiple customer orders. Such cases do not mean that we just say “No” to cloud based apps – there may be few cases when “No” might be a more plausible way to go – in most other cases, some amount of thought around re-design of applications to cater to the challenges of working on the cloud will be required.

Cloud computing - for everyone

Most of us produce and work with a variety of files (documents, spreadsheets and presentations) each day AND we do all of it using our traditional tried and tested Office software applications which we procure and install on each of our machines, be it at work, home or anywhere else. This post introduces web-based office applications such as the freely available Google Docs which could be leveraged in conjunction with with your existing tools to enhance productivity and collaboration.
I have been using the set of web-based applications that come bundled as part of Google Docs, in addition to the traditional office applications for a while now and find the web-based tools very useful in areas where i need more folks to collaborate (including in real time) on a document i am creating or to share a document to a wider audience and have the latest revision always be available to all of them without me having to resend a copy of the file to everyone or to have some of my files stored centrally and accessible from any where and continue working on them using just a simple web browser (no additional software required) or not having to worry if i have the latest version of a file or even about regularly backing up files! 

While there are very few data privacy concerns using Google Docs, i still retain confidential files on my local setup while using the web-based tool set to work on other documents that could benefit from being shared to folks i choose to view / edit them. This collaborative model may be used for a variety of applications, including tracking statuses, working jointly on a single document in real time and publishing reports that are automatically updated and available to everyone as when changes occur. You could find several other uses and scenarios where these tools may be applied to enhance productivity. If you haven't already been using these simple web-based tools, i suggest you give it a shot. You just might find them a nice complement or even in some cases a good-enough replacement for your present tools.

Back from a short hiatus from the holiday season

Today marks the last day of the holiday season extending from Christmas until just after New Year's day. Things have been fairly slow these few days and am hoping to get back to a faster pace starting tomorrow. I am looking forward to be in office with all of the great colleagues at work and also, resume blogging more often in the days ahead.