Friday, October 31, 2008

Suggestions for software testing and quality assurance discussions

The software testing and quality assurance is a wide area and I'll try to provide the best guidelines for understanding these processes, but you can also help me to understand what type of guidelines you need.
I'll try to cover in my next posts and to provide useful information about:
- fundamental test processes (test planning and control, test analysis, test design, test implementation, test execution, exit criteria, reporting)
- software development models (V-model, testing within a life cycle model)
- test levels (component testing, integration testing, system and acceptance)
- test types (functional and non-functional
testing, structural, confirmation, regression testing)
- test techniques (black box technique: equivalence partitioning, boundary value decision table testing, state transition; white box techniques: statement testing and coverage, decision testing and coverage; exploratory testing)
- test planning and estimation (test planning, exit criteria, test estimation, test strategies) and test progress monitoring and control
- determinng the project risks, product risks
- test tools and which is the best choice for specific application areas
- support information (recommended books, standards, courses)
- software testing tips & tricks
- glossary of terms
- and more

So please add your comments in this post and specify what areas of software testing and quality assurance you want to discuss here.

Recommended Testing Books - 3rd list

...extending the recommended books with the 3rd list (use the Books label to find easily and read the previous 2 lists) :

16. Jeff Tian - Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement (use link)
Product Description:
The one resource needed to create reliable software
This text offers a comprehensive and integrated approach to software quality engineering. By following the author's clear guidance, readers learn how to master the techniques to produce high-quality, reliable software, regardless of the software system's level of complexity.
The first part of the publication introduces major topics in software quality engineering and presents quality planning as an integral part of the process. Providing readers with a solid foundation in key concepts and practices, the book moves on to offer in-depth coverage of software testing as a primary means to ensure software quality; alternatives for quality assurance, including defect prevention, process improvement, inspection, formal verification, fault tolerance, safety assurance, and damage control; and measurement and analysis to close the feedback loop for quality assessment and quantifiable improvement.
This is a superior textbook for software engineering, computer science, information systems, and electrical engineering students, and a dependable reference for software and computer professionals and engineers.

17. Paul Hamill - Unit Test Frameworks (use link)
Product Description:
Unit test frameworks are a key element of popular development methodologies such as eXtreme Programming (XP) and Agile Development. But unit testing has moved far beyond eXtreme Programming; it is now common in many different types of application development. Unit tests help ensure low-level code correctness, reduce software development cycle time, improve developer productivity, and produce more robust software. Until now, there was little documentation available on unit testing, and most sources addressed specific frameworks and specific languages, rather than explaining the use of unit testing as a language-independent, standalone development methodology. This invaluable new book covers the theory and background of unit test frameworks, offers step-by-step instruction in basic unit test development, provides useful code examples in both Java and C++, and includes details on some of the most commonly used frameworks today from the XUnit family, including JUnit for Java, CppUnit for C++, and NUnit for .NET. Unit Test Frameworks includes clear, concise, and detailed descriptions of: The theory and design of unit test frameworks, Examples of unit tests and frameworks, Different types of unit tests, Popular unit test frameworks. It also includes the complete source code for CppUnit for C++, and NUnit for .NET

18. William E. Lewis - Software Testing and Continuous Quality Improvement (use link)
Product Description
Software Testing and Continuous Quality Improvement, Second Edition, illustrates a quality framework for software testing in traditional structured and unstructured environments. It explains how a continuous quality improvement approach promotes effective testing, and it analyzes the various testing tools and techniques that you can choose. Section I explains the role of QA principles and best practices in software testing. It provides a detailed overview of basic software testing techniques and an introduction of Deming's concept of quality through a continuous improvement process. This section explores the Plan, Do, Check, Act (PDCA) process, which is applied to all aspects of software testing. Section II reviews the software development life cycle and describes how testing and continuous quality improvement are incorporated into each phase of development. Section III details continuous quality improvement as part of the testing process. It breaks down software testing into a series of tasks that apply Deming's PDCA cycle. Section IV discusses fundamental challenges of managing testing projects, whether they are on-site or offshore. You learn how to establish effective test estimations to ensure that testing projects are on track. It also covers strategies for monitoring and managing software defects. Section V contains a brief history of software testing, previews advanced futuristic testing tools, and provides guidance for choosing the proper tool for various environments. It provides examples of some of the most popular products and offers a detailed methodology for evaluating them

19. Jonathan Jacky - Model-Based Software Testing and Analysis with C#(use link)
Product Description:
This book teaches new methods for specifying, analyzing, and testing software; essentials for creating high-quality software. These methods increase the automation in each of these steps, making them more timely, more thorough, and more effective. The authors work through several realistic case studies in-depth and detail, using a toolkit built on the C# language and the .NET framework. Readers can also apply the methods in analyzing and testing systems in many other languages and frameworks. Intended for professional software developers including testers, and for university students, this book is suitable for courses on software engineering, testing, specification, or applications of formal methods.

20. Derk-Jan de Grood - TestGoal: Result-Driven Testing (use link)
Product Description:
Software testing is traditionally seen as a difficult and time consuming activity that is hard to embed in the software development process. This book provides a different view. It explains to stakeholders how testing can add value to software development and doing business, and provides the tester with practical information. TestGoal is not just another methodology. Several good testing methodologies exist. But, like any other profession, also testing encompasses more than the simple application of a methodology. After all, strict adherence to a specific methodology is no guarantee for success. Success stems from the mindset, enthusiasm, knowledge and skill of the tester. These factors determine whether a methodology is applied successfully and whether testing takes on a result-driven character. And that’s what TestGoal is about: a result-driven attitude, testing principles and expertise as fundament, and a hands-on six step plan to enable result driven testing

21. Marcus S. Fisher - Software Verification and Validation: An Engineering and Scientific Approach (use link)
Product Description:
The world needs an in-depth technical book describing the methods and techniques used to provide confidence in our system software. In today’s market, not only are governments more focused on software safety, but private industry and academia are as well. The methods and techniques used to provide such confidence is commonly called software verification and validation.
Software Verification and Validation: An Engineering and Scientific Approach, a technical professional book, performs software verification and validation on high-fidelity systems. This book includes crucial information for methods and techniques that not only develop the system, but ensure that the system was developed correctly.
Software Verification and Validation: An Engineering and Scientific Approach is structured for research scientists and practitioners in industry. This book is also suitable as a secondary textbook for advanced-level students in computer science and engineering.

22. Roland Groz - Testing of Communicating Systems (use link)
Product Description:
This book constitutes the refereed proceedings of the 16th IFIP TC6/WG6.1 International Conference on Testing of Communicating Systems, TestCom 2004, held in Oxford, UK in March 2004. The 14 revised full papers presented were carefully reviewed and selected for inclusion in the book. All Current issues in testing communications system are addressed, ranging from classical telecommunications issues to the testing of reactive systems and testing from UML.

Thursday, October 30, 2008

Top testing mistakes

Making mistakes is a natural thing and by testing we try to find them. But what if we do mistakes when trying to find mistakes. Well, I hope we all will learn from them and don't repeat them in the future. Until then, I'll try to enumerate some of those mistakes which I find the most important:
- to think that the only purpose of testing is to find bugs.
- to not find the most important bugs (and to be satisfied with finding a large amount of non-significant bugs)
- to fail to correctly identify risky areas
- to start the testing too late
- poor bug reporting
- to concentrate more on running tests than to design them

I recommend you to read the
"Classic Testing Mistakes" article, written by Brian Marick (use the link) if you want to know more about other testing mistakes.

Wednesday, October 29, 2008

Testing standards (IEEE, BS, CMMI, RUP, etc.)

Now is time to specify several good "testing" related standards, which you'll probably find somehow interesting :

1. IEEE 1012 Software Verification and Validation
2. IEEE 829 Software test documentation
3. BS 7925-2 British Std for Software Component testing (recommended)
4. US FDA General principles of software validation
5. US NIST Reference Information for the Software Verification and Validation Process
6. RUP Test discipline
7. CMMI v1.2 Verification and Validation

I'll try to comment some of these standards in one of the next posts, so you can decide if its can be important or not for you as QA/tester

Testing courses

Since I've observed your interest for testing courses, not only for books, I've also put together a collection of very good testing courses which I recommend you to read:

1. Cen Kaner - BlackBox testing
2. Cen Kaner - Metrics and Measurement
3. Stale Amland - Exploratory testing
4. James Bach - Rapid Software Testing
5. Sita Ramakrishnan - lectures on testing
6. Brett Pettichord - Test Automation Architectures
7. Scott Barber - User experience, not metrics
8. Scott Barber - Beyond performance testing

Tuesday, October 28, 2008

Recommended Testing Books - 2nd list

...extending the recommended books with the 2nd list:

8. Glenford J. Myers - The Art of Software Testing, Second Edition (use link)
Product Description:
This long-awaited revision of a bestseller provides a practical discussion of the nature and aims of software testing. You'll find the latest methodologies for the design of effective test cases, including information on psychological and economic principles, managerial aspects, test tools, high-order testing, code inspections, and debugging. Accessible, comprehensive, and always practical, this edition provides the key information you need to test successfully, whether a novice or a working programmer. Buy your copy today and end up with fewer bugs tomorrow.

9. Rick D. Craig (Author), Stefan P. Jaskiel - Systematic Software Testing (use link)
Product Description:
This textbook sets out to provide professionals with an in-depth understanding of the software-testing people and process issues that are critical for delivering high-quality software on time and within budget. The authors aim to give those involved in building and maintaining complex, mission-critical software systems a flexible, risk-based process to improve their software-testing capabilities. Whether an organization currently has a well-defined testing process or almost no process, this resource provides insights into better ways to test software. This guide is written for: software-test managers; testers; developers; quality-assurance managers; and software configuration managers

10. Elfriede Dustin - Effective Software Testing: 50 Specific Ways to Improve Your Testing (use link)
Product Description:
The knowledge of what constitutes a successful, end-to-end software testing effort is typically gained through experience. In this new book, noted testing expert Elfriede Dustin imparts the best of her collected wisdom. She presents fifty specific tips for a better testing program. These fifty tips are divided into ten sections, and presented so as to mirror the chronology of a software project. Using this book as a guide and reference, quality assurance professionals will be better able to insure the quality of their next application. While testing has historically been viewed as an afterthought in the grand scheme of software engineering, the success of an application, and possibly an organization, can rest on the shoulders of the testing team. That's because the testing program functions as the final "quality gate" for an application. Testing allows or denies the transition of an application into the market. There are a multitude of seemingly minor tasks that must be performed and managed by the testing team.

11. Ron Patton - Software Testing (2nd Edition) (use link)
Product Description:
Software Testing, Second Edition provides practical insight into the world of software testing and quality assurance. Learn how to find problems in any computer program, how to plan an effective test approach and how to tell when software is ready for release. Updated from the previous edition in 2000 to include a chapter that specifically deals with testing software for security bugs, the processes and techniques used throughout the book are timeless. This book is an excellent investment if you want to better understand what your Software Test team does or you want to write better software.

12. Paul C. Jorgensen - Software Testing: A Craftsman's Approach (use link)
Product Description:
Since the last publication of this international bestseller, software testing has seen a renaissance of renewed interest and technology. The biggest change comes in the growing prominence and acceptance of Agile Programming. Software Testing—A Craftsman’s Approach, Third Edition extends the combination of theory and practicality of the first two editions to include agile programming development and discusses the serious effect this emerging area is having on software testing

13. Lydia Ash - The Web Testing Companion: The Insider's Guide to Efficient and Effective Tests (use link)
Product Description:
* Written by one of Microsoft's key testing trainers, this hands-on tutorial and reference explains why, when, what, and how to test
* Teaches new and experienced testers how to analyze and properly test Web applications
* Filled with practical advice that can be immediately applied to any Web-testing task, on any browser running on any platform
* Concentrates on proven solutions and presents the material in a way that will help develop a professional skill set in novice testers and will improve the productivity of all testers
* Companion Web site includes dozens of valuable templates and test patterns that will allow readers to rapidly conduct tests in multiple languages against all browser and operating-system combinations

14. David Astels - Test-Driven Development: A Practical Guide (use link)
Product Description:
This book teaches programmers how to make Test Driven Development (TDD) work in their organization. TDD is unique because it forces the programmer to write tests for code before the code is actually written. This process is the reverse of how software testing has traditionally been conducted, but TDD ensures that software is produced more efficiently. A test-first mentality allows the programmer to define, specify, illustrate, limit, and drive the code, resulting in documented, tested, code that is as simple and lean as possible. The book also presents tools and techniques, and all major points are supported by numerous examples (including an entire project, end-to-end) and exercises.

15. Kent Beck - Test Driven Development: By Example (use link)
Product Description:
Quite simply, test-driven development is meant to eliminate fear in application development. While some fear is healthy (often viewed as a conscience that tells programmers to "be careful!"), the author believes that byproducts of fear include tentative, grumpy, and uncommunicative programmers who are unable to absorb constructive criticism. When programming teams buy into TDD, they immediately see positive results. They eliminate the fear involved in their jobs, and are better equipped to tackle the difficult challenges that face them. TDD eliminates tentative traits, it teaches programmers to communicate, and it encourages team members to seek out criticism However, even the author admits that grumpiness must be worked out individually! In short, the premise behind TDD is that code should be continually tested and refactored. Kent Beck teaches programmers by example, so they can painlessly and dramatically increase the quality of their work be continued

Sunday, October 26, 2008

Recommended Testing Books - 1st list

I've put together a collection of my favorite testing books which I recommend you to read ( I'll also try to specify a direct link for each of these recommended books, especially to the appropriate page on
I'll add additional books in other lists order to extend the recommended books.

1. Rex Black - Critical Testing Processes: Plan, Prepare, Perform, Perfect (use link)
Product description:
Text presents twelve testing processes that are critical to success. After each process is introduced, a case study demonstrates its use in various organizational, operational, and technological contexts. For computer software test professionals. Softcover. DLC: Computer software--Testing

2. Lee Copeland - A practitioner's guide to software test design (use link)
Product description:
Here’s a comprehensive, up-to-date and practical introduction to software test design. This invaluable book presents all the important test design techniques in a single place and in a consistent, and easy-to-digest format. An immediately useful handbook for test engineers, developers, quality assurance professionals, and requirements and systems analysts, it enables you to: choose the best test case design, find software defects in less time and with fewer resources, and develop optimal strategies that help reduce the likelihood of costly errors. It also assists you in estimating the effort, time and cost of good testing.

3. Yu k Kuen Wong - Modern software review: Techniques and Technologies (use link)
Product description:
Software quality is vitally important to the success of a business, as a single undetected error or defect during the software development process could have disastrous consequences during a business operation. Software review is one of the methods used to detect defects. This process maintains the quality of the product by reviewing interim deliverables during development. Modern Software Review: Techniques and Technologies provides an understanding of the critical factors affecting software review performance and gives practical guidelines for software reviews...

4. Jerry Zeyu Gao - Testing and Quality Assurance for Component-Based Software (use link)
Product description:
Presenting the state-of-the-art in component-based software testing, this new, cutting-edge resource offers you an in-depth understanding of the current issues, challenges, needs, and solutions in this critical area. The book discusses the very latest advances in component-based testing and quality assurance in an accessible tutorial format, making the material easy to comprehend and benefit from no matter what your professional level.

5. Cem Kaner, Jack Falk, Hung Quoc Nguyen - Testing Computer Software (use link)
Product description:
The Testing Computer Software set the standard for the emerging field of test engineering with a full tour of the state of the art in managing the testing process. The reissued text makes this classic out-of-print text available once again. Though it relies heavily on older examples (including MS-DOS) and has not been updated, this text is still a worthwhile resource for practical-minded advice on the realities of testing. The best thing about Testing Computer Software is its practical point-by-point guide to everyday software testing, from creating a test plan, to writing effective bug reports, to working with programming staff and management to fix bugs...

6. Ilene Burstein - Practical Software Testing (use link)
Product Description:
Software testing is rapidly evolving as a critical sub-discipline of software engineering. The industry needs professionals educated in the modern methods and tools of software testing and quality assurance. Based on the needs of software professionals, "Practical Software Testing" takes a unique approach to teaching readers how to effectively plan for testing, design test cases, test at multiple levels, organize a testing team, and optimize use of testing tools. It introduces testing concepts that are managerial-, technical-, and process-oriented, using the Testing Maturity Model (TMM) as a framework....

7. Rex Black - Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional (use link)
Product Description:
A hands-on guide to testing techniques that deliver reliable software and systems
Testing even a simple system can quickly turn into a potentially infinite task. Faced with tight costs and schedules, testers need to have a toolkit of practical techniques combined with hands-on experience and the right strategies in order to complete a successful project. World-renowned testing expert Rex Black provides you with the proven methods and concepts that test professionals must know. He presents you with the fundamental techniques for testing and clearly shows you how to select and apply successful strategies to test a system with budget and time constraints
... be continued

Saturday, October 25, 2008

Testing terminology - Defect ( Bug, Fault or Problem)

definitions (same with Bug, Fault or Problem, but with some "nuances" ) :
- an imperfection that causes inadequacy or failure
- the lack of something necessary or desirable for completion or perfection
- an imperfection or weakness; fault; flaw; blemish
- any variance from a desired attribute, state, or functionality
- anything that either adversely impacts the expected functionality to the user, or otherwise reduces the quality of a work product
- (bug) fault or defect in a computer program, system, or machine
- (bug) a defect or difficulty, as in a system or design
- (bug) a defect in the code or routine of a program
- (problem) a state of difficulty that needs to be resolved
- (problem) a source of difficulty, etc..

The defect is a "flaw" in a component, that can cause it to fail to perform its requirements and it may cause the failure of the component while using it in real-time.
I've specified that the meaning of defect - bug - fault - problem is about the same, but with certain nuances; "Bug" implies that there is a fault made in a software, by developers, while the "Defect" is perceived as being more severe.
Also, we must not get confused by the sense of the "defect" and the "error" terms.
E.g. :
- for a calculation component of an application is indicated ( in its specifications) that x - x = 10 x;
this means that the one who wrote it in the specifications would have made an Error and by doing this he would have created a Defect.

... please comment if you have other suitable definitions

Testing terminology - Error

"Error" definitions:

- a mistake
- a wrong action attributable to bad judgment or ignorance or inattention
a misconception resulting from incorrect information
- a human action that produces an incorrect result
- the difference between a computed or measured value and a true or theoretically correct value
- the act or an instance of deviating from an accepted code of behavior, etc.

The errors are not restricted only to the code wrote by the developer (errors can occur also in the technical, functional specifications or design stages, so anywhere in the software development life cycle)

... please comment if you have other suitable definitions

Testing - definitions

So, what is testing? We can find and give various of simple or complete definitions for this question:
- "...the process of executing a program with the intent of finding errors"
- "... testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test, with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding "
- "... testing is finding out how well something works. In terms of human beings, testing tells what level of knowledge or skill has been acquired. In computer hardware and software development, testing is used at key "
checkpoints in the overall process to determine whether objectives are being met"
- "... the process of executing a program with the intent to certify its Quality"
But testing is mainly the process which consist of the following activities:
- review requirements
- planning
- preparation and evaluation
- defect detection,
in order to satisfy the specified product requirements (it involves operation of a system or application under controlled conditions and evaluating the results).

Friday, October 24, 2008

Testing - a necessity testing is a need. Why? Because the softwares built today are so complex and released with a stunning speed, due to the tough market competition. This implicitly leads to failures/mistakes/oversights, which will end up with incomplete user requirements (missed out somewhere during the development process) or not functional areas.
So, by software
testing we reduce the failures occurance and provide to the user a product which meet all the specified user requirements.
But in the main the answer should be - because of the cost of error.
Each failure involves a variable cost, depending on how important it is, the area of activity, etc. Imagine an important failure in a safety critical software, for example a control system used by an airplane or by a medical device. Exactly, besides millions it can
also cost lives.
Very important on the cost estimation is to understand that the cost of defects increases with the passing of the development process stages (in the first stages - e.g. requirements review stage - it may cost very little, but in a post-implementation stage it will cost a lot, for sure).
The benefits of software testing can be resumed at the followings: to reduce the number of defects (contained in the released version), to improve the quality of the software and to increase the reliability of the product.
We won't be able to test everything, because of the time and resources constraints on the level of testing performed, but we can determine the specific project risk factors based on the testing performed during the software development process.