I sent a questionnaire out to a bunch of people the other day. I thought that the community might find it interesting so here is what I can share.
Here is the questionnaire, unaltered, typos and all.
The term testability in these questions is about the test subject. How easy is it to test? It is not about how well or poorly the unit test is written.
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
What benefits if any do you see in testability beyond making testing easier?
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
How does testability effect design?
How does design effect testability?
What do you think is the best way to get testability into a product?
Before I show responses let me say that I refined the questions and posted them to the TDD Yahoo Group. I got NO responses on the group, that is right none, not a one! I was very disappointed as I think that the industry as a whole needs to take testability more seriously. I only got 7 responses from those that I emailed directly and I can share only 5 of them. I really was hoping for more of a response.
Here are the refined questions and then the responses:
The term testability in these questions are about the test subject. How easy is it to test? How easy is it to control? How easy is it to observe? It is not about how well or poorly the unit test is written.
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design? Is testability a good enough reason to create an abstraction to allow for mocking?
What benefits if any do you see in testability beyond making testing easier ( e.g. maintainability…)?
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication to testing and testability?
What, if any, design characteristics and forces do you see in opposition with testability ( e.g. encapsulation)? Is the tension beneficial?
How does testability affect design?
How does design affect testability?
What do you think is the best way to get testability into a new product?
What do you think is the best way to get testability into an existing product?
Ayende Rahien’s Response:
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
I test to a lot of thinking about a design, but I rarely commit it to any form without tests.
I tend to use tests to drive the design, but I sometimes slip and add YAGNI stuff there.
What benefits if any do you see in testability beyond making testing easier?
SRP mainly. I found that I can cut my way across the code much easier when I have a testable design. Usually by adding cross-cutting features at the seams of the system.
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
No, good software tests are an order of magnitude higher than that to be any use of all. I think that you mean that 3-5% dedicated to making testing possible. If that is the case, certainly. Take into account all those hard to test APIs. They are hard to test because I have no way to hook into them.
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
One thing that I found is that I get issues as the amount of work a certain piece does.
It needs several collaborators to make it work, when it gets too big (usually because of added requirements), it becomes hard to test, although the design is sound.
One thing that is opposing force is encapsulation vs. test hooks that may be an issue.
How does testability effect design?
Usually by making it simple and locallized
How does design effect testability?
Design that is highly closed (.Net’s cache API, for instance) is very hard to test…
What do you think is the best way to get testability into a product?From the start J Wouldn’t want to start doing it from scratch.
Morten Mertner’s Response:
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
YAGNI to me is more of a “what”, whereas testability is more of a “how”.
Testability is clearly something to be considered, along with security,
scalability and other potential goals, during the design phase.
What benefits if any do you see in testability beyond making testing easier?
Testability forces developers to consider testing aspects of a
component/class/method. This usually leads to more specific contracts
(e.g. for method parameters and return values), because the developer
has spent time considering how a method could be called (as opposed to
only considering how he intended it to be called).
Designing for testability also helps reduce the entanglement of UI and
business code. Visual design tools usually make no effort at decoupling
code and having testability as a design goal forces the developer to
deal with such auto-generated code.
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
No, I think that is comparing apples and oranges.
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
I certainly see encapsulation as a part-time opponent to testability,
because it forces you to rely only on black-box testing. Black-box
testing is actually preferred (as the test is decoupled from the
implementation details), but sometimes the abstracted component is of
sufficient complexity to warrant additional white-box testing.
Encapsulation can make this difficult.
I also think it is sad that visual design tools rarely pay attention to
the organization of code (which is a key factor in achieving testability).
How does testability effect design?
How does design effect testability?
Neither effects the other, but they might affect each other
Testability is primarily something that affects design, in that it is
something you should consider before writing code. In this way
testability is much like other design goals: scalability, reliability,
availability, security, etc.
What do you think is the best way to get testability into a product?
To deal with it during the design phase. And being specific about what
to test and expect.
Phil Haack’s Response:
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
I think testability complements Yagni in that you don’t want to write a test for code you don’t immediately need. I find it keeps my ambitions in check.
What benefits if any do you see in testability beyond making testing easier?
Gee, isn’t that enough? Not to get into a long spiel, but our industry does not do very well with project overruns. Various studies show that some huge percentage of projects fail. I think a fair share of that is due to inflexible designs and code written under unrealistic schedules that end up being buggy. Considering that it only takes a second to introduce a bug, but days to debug that bug, a little more investment in up-front testing makes for more pliable designs and less buggy code.
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
Probably. Not too intimately acquainted with the hardware industry. But they’ve realized the importance of making sure it just works.
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
Hmmm, have to think about that one.
How does testability effect design?
When designing code meant to be used by another (class libraries, apis, etc..) it forces the developer to think about how the user of the code will interact with it. This makes for more usable code design.
How does design effect testability?
Interesting question. I always think about how testability affects design. I think one can certainly subvert testability with poor design.
What do you think is the best way to get testability into a product?
Test first development.
Marc Stober’s Response:
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
YAGNI and testing are odds; you add things for testability that you don’t need otherwise. For example, a colleague recently broke our build by making a certain method (that I had written a unit test for) private; his argument was this we didn’t need to make this method public since it’s only there to support other methods in that class. On the other hand, I think the principle of “show your work” is also important so you don’t just end up building black boxes that work because you say so, on the principle that all people really “need” is the end result.
What benefits if any do you see in testability beyond making testing easier?
Encourages the “show your work” principle mentioned above.
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
Maybe but it’s something that needs to be researched scientifically; I wouldn’t want to see bloggers just start throwing around “3-5%” as best practices.
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
Well just the idea that everything you add is something else that can break. Probably the hardest part is that different people will make different trade-offs so you have the challenge of getting people on the same page as far as how much testability is required.
How does testability effect design?
It improves it.
How does design effect testability?
Things that are designed with testability in mind will require less work and risk to make testable.
What do you think is the best way to get testability into a product?
Use the latest and greatest development tools so that you can refactor things as needed. For example I think .NET 2.0 actually has a way (I’ve seen it in Microsoft’s Enterprise Library code) to make an assemblies internal methods accessible to a test assembly, which is a way to deal with some of the issues above if they really bother you.
Jeff Brown’s Response:
How do YAGNI and testability relate? Is testability a first class citizen, or force, in design?
YAGNI reduces the number of states in the system that must be tested. Correspondingly, the system may become more testable. However one must be careful with YAGNI. Certain affordances within the system for testing purposes alone may be greatly beneficial for reason of testability while not providing any additional functional benefits. Consequently testability may constitute a force in the design process by introduction of design artifacts that would not be present or required but for the needs of testing.
What benefits if any do you see in testability beyond making testing easier?
Designing for testability tends to encourage decomposition of a program into cohesive units that are more decoupled from another than otherwise required. As an example, it is often useful to introduce new interfaces into the system even for components that are expected to only have a single implementation ever but for reason of testing. A system designed without testability in mind might not clearly define such components and in the worst case might scatter the state and behavior of those components everywhere with tight coupling, poor cohesion, poor encapsulation, and many internal interfaces. Thus designing for testability may produce better component architectures with attendent non-functional benefits such as understandability, maintainability and extensibility.
Do you think that the software development industry has any thing to learn from the hardware industry’s dedication of 3-5% of circuits to testing and testability?
Yes. In the development of both hardware and software, more time is spent in the process of verification than in the process of design. As with additional hardware circuits, it is reasonable to provide additional software instrumentation to reduce the cost of verification by making it easier to observe and/or simulate deep aspects of system behavior.
What, if any, design characteristics and forces do you see in opposition with testability? Is the tension beneficial?
Testability is occasionally at odds with performance. It may happen that instrumentation or design artifacts introduced purely for reason of testability may negatively influence the performance of the system. For most software, however, this is not a concern as long as one is careful to separate functional and testing concerns as much as possible. In many cases in software, design for testability introduces a few new additional virtual functions calls at negligible cost where none were required, some moderate storage overhead for objects that were created to make the model accessible for testing (and understandability) rather than to achieve any particular functional objective, and a few quick checks at locations where hooks or instrumentation might be injected by a test harness.
In addition, design for testability may require opening up more aspects of a system to extension. In this case, fewer assumptions can be made about certain components because they may be replaced by implementations with somewhat different characteristics for reason of testing. On the other hand, exposing these components, clearly defining their contracts, and permitting them to be replaced can yield other benefits.
I believe the tension is beneficial in the end. Without explicit effort to make a system testable, an overabundance of time may be spent performing software verification that yields an outcome of dubious certainty. Likewise if insufficient effort is placed on testability, the resulting system may perform extremely well but yield results of dubious accurary, or may support one particular deployment scenario very well but prove to be difficult to extend, to maintain or to scale in the long run.
How does testability effect design?
Testability is a cross-cutting concern. It may impact design by introducing new artifacts that would not otherwise be demanded by functional requirements. More importantly, it may impact the functional requirements themselves by introducing new features that must be supported by a system in order to make it easy to verify according to the processes adopted by the software development group or the customer.
For reason of testability one might introduce whole new screens into a user interface that reveal diagnostic information that would otherwise be hidden from view. Likewise one may spend more effort to provide symmetric operations with overvable outcomes so that test harnesses can perform some operation, verify it, and revert it even though querying the detailed results of an operation or reverting it might not be a part of normal system operation. This situation occurs frequently when designing systems intended to be tested by non-programmers, particularly when it becomes difficult to simulate certain inputs.
Occasionally the needs of testability mirror those of demonstration to a given client. It is sometimes necessary to permit a non-technical external stakeholder to demonstate the correct operation of a system under specific conditions. For example, voting machines must by law support a mode of operation that can simulate an election. Accreditation of the software depends upon satisfactory behavior during the simulated election. This demonstration of the system amounts to an independent validation of its basic functions. Thus designing for testability by internal parties may well simplify the implementation of the external validation functional requirements demanded by external parties.
How does design effect testability?
A given design may arbitrarily effect testability. Designs that are well defined, well documented, cohesive, easy to observe, configurable, symmetric, deterministic and open to extension tend to be easier to test because it is easy to identify the components to test, it is easy to build tests to exercise them and it is easy to build test harnesses that simulate portions of the system that are beyond the scope of a particular test. Conversely, designs that are poorly specified, poorly understood, tightly coupled, obscure, hard-wired, asymmetric, non-deterministic or closed to extension tend to be harder to test because it is difficult to identify what must be tested, difficult to establish the functional requirements of the components that are to be tested, or difficult/impossible to alter the configuration of the system in such fashion as to produce deterministic inputs or to observe results as required by a test harness.
What do you think is the best way to get testability into a product?
Make testing a priority of your design and development teams.
Test first or test as you go. Decompose your system into easily identified components and clearly specify their contracts. Ensure that any important state change can be observed. If you have separate development and testing teams, ensure they work together and ensure that the testing team is sufficiently empowered and sufficiently skilled to demand testability on the part of the developers. Get product managers on board to ensure that the requirements are clearly defined and to allot sufficient time for testing. Adopt Service Oriented Architecture or equivalent approaches to design that encourage a good decomposition of the system into well-defined, cohesive, and replaceable modules.
[…] It seems that I misunderstood Kelly Anderson and he wished to be included in the posting. Thanks again for straightening me out Kelly. Here is his reply: […]
September 27th, 2006, at 8:13 pm #