Category: BLACK BOX Testing

What is Beta Testing

In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers. Beta testing comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. View the advantages , Types of beta testing

The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.

Beta testing otherwise referred to as user testing is done by the end users at the end users site in order to authenticate the functionality, compatibility, usability, and reliability testing. It enables the real customer the chance of offering inputs into the product’s design, usability and functionality. The inputs are part of what contribute to the product’s success. The inputs also act as future products investment when they efficiently manage the data that are gathered.
 

Types of beta testing

There are basically two types of beta testing that a product has to go through: Open and Closed beta.

• Open beta: This is from a larger group to the general public and interested individuals. The testers usually make a report of the bugs detected. In addition, they make suggestions regarding extra features they feel should be included in the final version.
• Closed beta: These are done on “invitation only” basis where the versions are released to a group of selected people for a user test.

Features of beta testing

The following are the features of beta testing:

• Beta testing is performed by customers or end users who are not employed by the company.
• The test takes place at the site of the user
• Black box technique is normally used.
• It only needs a few weeks of execution
• It involves the testing of the software in real conditions and no testing environment is needed.
• It checks for security and reliability
• The testing pays attention to product quality and makes sure that the product is all set for the real users based on the gathered opinions.
• The suggestions made by the customers are applied in the product’s future versions. 

 

Advantages of beta testing

Beta testing has numerous advantages and they are listed below:

• The actual users are given the opportunity to test the application before it is released to the general public.
• Customers can install and test the application as well as provide you feedbacks.
• It is an opportunity for the beta testers to detect bugs that you may not have spotted yourself.
• The feedback derived from the users assist in taking care of the problems before the application is released to the general public.
• Taking care of more issues that solve customers’ problems leads to an application with higher quality when finally released to the public.
• Early users help to create a thrill about your application.
• Beta testing is more cost effective compared to other methods of gathering data.

Disadvantages of beta testing

• Exercising control over the testing is difficult since it is not done under lab environment but in real environment.
• It is difficult to find the correct testers and manage their participation.

Conclusion

Beta testing is the final test carried out by the end users in real environment. The feedback gathered from it serves as the basis for improving the future versions of the application.

What is ALPHA TESTING

In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.

Alpha testing is done before the beta testing and after the acceptance testing. Mostly its done by the in-house members from developers and qa teams. I simple words its the testing by developed team just before launching the live beta version of that software.

Alpha testing is the preliminary software field test that is done by a team of users with the purpose of discovering bugs that had escaped being detected in previous tests. Alpha test is carried out in order to detect and fix bugs that exist in a system, though they might not have been found in the tests previously done. The test is usually done at the developer’s site.

Alpha testing gives room for minor changes to be effected before the development of the application is completed. It is usually handled by a group that is not a part of the design team, though they still work within the company. Examples include software QA engineers or in-house software test engineers. They can also comprise of potential customers/users. Both white box and black box techniques are used for the test.

Stages of alpha testing

Alpha testing passes through two different stages of testing most of the times – first phase and second phase.

• First phase: The software is tested by in-house developers. The main purpose is to quickly detect bugs. The software used includes hardware-assisted debuggers or debugger software.
• Second phase: This involves the handing over of the software to the QA staff so that it can be further tested in an environment that is comparable to where it would be finally used.

Advantages of alpha testing

The advantages of alpha testing include the following

• It gives you an opportunity to have a clear view of how reliable the software is, right from the early stage.
• It assists in discovering bugs that could pose a serious threat so that you can easily fix them.
• It helps to mimic the real-time behavior of the users and the environment.
• The test helps to know the real quality of the software before it released into the market.

Disadvantages of Alpha testing

Alpha testing has its own disadvantages. Some of the disadvantages include the following

• It does not involve security and reliability testing. This can to some extent limit the trust intended users will have on it.
• It requires a testing environment such as a lab. This can sometimes be hard to come by, thereby delaying the test.
• It needs long execution of cycles. This can be tedious sometimes.
• It is difficult to test the in-depth functionality of the software since it is still under development.

Conclusion

Alpha testing is necessary for helping to identify bugs while software is still being developed. This aids the developers in fixing the bugs before moving on to the next stage of development.

What is User Acceptance Testing

In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to. In software development, user acceptance testing (UAT) – also called beta testing, application testing, and end user testing – is a phase of software development in which the software is tested in the “real world” by the intended audience. View the pdf tutorials about UAT

User Acceptance Testing can be done by in-house testing in which volunteers or paid test subjects use the software or, more typically for widely-distributed software, by making the test version available for downloading and free trial over the Web. The experiences of the early users are forwarded back to the developers who make final changes before releasing the software commercially.

The popular technology encyclopedia, Technopedia: “User acceptance testing (UAT) is the last phase of the software testing process. During UAT, actual software users test the software to make sure it can handle required tasks in real-world scenarios, according to specifications. UAT is one of the final and critical software project procedures that must occur before newly developed software is rolled out to the market.”

User acceptance testing also goes by other nomenclature such as End-user testing, Application, or Beta and is usually the last stage during the process of designing a website before the software is finally distributed or installed on the system of the client.

This is done by selecting some people from the potential users of the software and asking them to use the software before it is finally released for use. While using the software, these selected users will take note of the flaws in the software for further modification and correction before the final release. This affords the users the opportunity to interact well with the software and see what the features are and which of the features are absent or malfunctioning. They also pay attention to sections of the software that are disconnected from the others and may lead to a lack of meaningful communication between the different sections of the software.

Types of User Acceptance Testing

User Acceptance Testing is divided into the following types:

• Black Box Testing

In this testing, the tester doesn’t bother himself with the features of the application but tests whether the software meets its purpose. In essence, some functionalities of the software will be tested.

• Alpha & Beta Testing.

This is also called Field Testing and takes place in the customer’s environment. This is specifically testing the software extensively by a couple of customers who use the software in their environment. After the testing, they will provide feedback that will be used to improve the product.

• Regulation Acceptance Testing

The software is checked against some regulations such as legal and governmental regulation to determine whether it meets those regulations. This is also known as Compliance Acceptance Testing.

• Contract Acceptance Testing

This testing is carried out to determine whether the designed software meets some specifications and criteria which are specified and agreed upon before the development of the software.

• Operational Acceptance Testing

This testing is used to determine the operational readiness or pre-release of a system, service, or product as an important aspect of a quality management system.

How to perform in-house User Acceptance Testing

Sometimes, there may be the need for in-house testing of a software program. These are the simple steps to take for testing:

• Planning: While planning, the UAT strategy is explicitly outlined.
• Designing test cases: In order to cover all the areas where the software will be used in real life, test cases are designed
• Selection of the testing team. This team will be made up of real users.

Why is User Acceptance Testing Necessary?

You may be wondering why a company will go through the process of repeatedly testing a product before it is distributed for general consumption. Here are a few reasons why the testing is necessary:

• User requirements are captured in a verifiable way.
• They help the developers identify the potential problems the software will have in the future.
• It is a good assessment of how ready for use the system is.

If you have a system designed for public consumption, you will not want to avoid the user acceptance testing. It is the only way to ensure the efficiency of your system.

PDF Tutorials

• User Acceptance Testing introduction – pdf tutorial
• Article about User Acceptance Testing

What is Regression Testing

Regression testing is a style of testing that focuses on retesting after changes are made. In traditional regression testing, we reuse the same tests (the regression tests). In risk-oriented regression testing, we test the same areas as before, but we use different (increasingly complex) tests. Traditional regression tests are often partially automated. These note focus on traditional regression. You can also view more advanced guides about Regression testing in pdf at the end of this tutorial.

Regression testing attempts to mitigate two risks:

• A change that was intended to fix a bug failed.
• Some change had a side effect, unfixing an old bug or introducing a new bug

Regression testing approaches differ in their focus. Common examples include:

Bug regression: We retest a specific bug that has been allegedly fixed.

Old fix regression testing: We retest several old bugs that were fixed, to see if they are back. (This is the classical notion of regression: the program has regressed to a bad state.)

General functional regression: We retest the product broadly, including areas that worked before, to see whether more recent changes have destabilized working code. (This is the typical scope of automated regression testing.)

Conversion or port testing: The program is ported to a new platform and a subset of the regression test suite is run to determine whether the port was successful. (Here, the main changes of interest might be in the new platform, rather than the modified old code.)

Configuration testing: The program is run with a new device or on a new version of the operating system or in conjunction with a new application. This is like port testing except that the underlying code hasn’t been changed–only the external components that the software under test must interact with.

Localization testing: The program is modified to present its user interface in a different language and/or following a different set of cultural rules. Localization testing may involve several old tests (some of which have been modified to take into account the new language) along with several new (non-regression) tests.

Smoke testing also known as build verification testing:A relatively small suite of tests is used to qualify a new build. Normally, the tester is asking whether any components are so obviously or badly broken that the build is not worth testing or some components are broken in obvious ways that suggest a corrupt build or some critical fixes that are the primary intent of the new build didn’t work. The typical result of a failed smoke test is rejection of the build (testing of the build stops) not just a new set of bug reports.

 
PDF Tutorials

• Regression Test Selection Techniques
• Regression Test Selection for Java Software
• Analysis of Different Regression Testing Approaches

Scenario Testing in Software Development

This is an aspect of software development that involves testing particular software during development by using some scenarios. It is a realistic way of finding out the potential issues in software when it is fully used for the intended purpose.

The purpose of scenario testing is to make sure that the software is functioning flawlessly or that the business process of the software is not working below the expected standard.

What is Scenario Testing

Scenario tests are realistic, credible and motivating to stakeholders, challenging for the program and easy to evaluate for the tester. They provide meaningful combination of functions and variables rather than the more artificial combination you get with domain testing or combinatorial test design. A detailed tutorial about scenario testing in pdf can be read at the end of this page.

This test find out the issues in our software against the practical usage. The end users creates the scenario here. Now we can consider an example to get more idea. Supposed We have developed a billing software for shop. We have completed many testing and there is no bugs in coding and all features are working, thats good. Now we are discussing with our customer and starting and regression test. He is telling a scenario, that I have entered a processed bill for one order, then my customer require to change the quantity of material he purchased. I need to give its as same bill. Then we will try this scenario in our software, and we found that our software not able to edit the generated bill because there is no option for that. So we need to add that facility too. Its only a general example. In simple words it doing the test against practical situation , and that stories can be given by end customers.

How does it work?

In scenario testing, the testers assume the roles of the end users and stimulate the real world scenario where the software will be used. They can also make use of some cases that the end users can perform on the software. Others that assist the testers in creating the scenario to be used include the stakeholders, clients, and the developers.

Since the goal of doing scenario testing is to test the business aspect of the software, it is more efficient in spotting some defects that other testing techniques are not competent enough to identify.

To carry out scenario testing, test scenarios that mimic the usage of the end user is created. The scenario may be in independent or a range of test cases that are done one after the other.

Qualities of a good scenario testing

Due to the use of scenario testing to assess the functionality of a software program, a good scenario testing should have the following qualities:

• It should be motivating: The scenario used must be motivating otherwise it may be rejected by the stakeholders.
• Credibility is a must: The story must be considered as credible by the stakeholders. Apart from the feeling that the scenario may happen, they want to believe that it will happen and the system should be able to perfectly handle it.
• Complex use: The hypothetical story must involve subjecting the software to a complex use. On the other hand, the testing environment can be complex or the testers may use a set of data that is complex.
• An easy to evaluate result: the result of the testing should be very easy to evaluate. This will ease the understanding of the problem or error detected, making the job of fixing the bug easy for the developers too.

Any testing scenario should have all these qualities before it will gain the status of a qualified scenario for testing a software program. This is a typical scenario:

If a patient comes to a hospital to complain about chest pain, his personal information will be taken and he will be recorded as an outpatient and all his records in the hospital will reflect that. If after a couple of visits the hospital decides to hospitalize him after complaining of another more serious ailment, his status will change from an outpatient to an inpatient. The current doctor treating him for the new illness will need his medical history as part of the effort towards treating him. If the doctor cannot find his record, it is an error. His medical history is supposed to have been automatically transferred from the Outpatient section to the Inpatient section. In this scenario, the HMS system that is responsible for handling the records fails. This is a bug that can cause a lot of damage in the hospital if such a huge error is not immediately corrected. A suffering patient cannot be urgently attended to due to a bug in the program.

This hypothetical scenario underscores the importance of scenario testing when developing a software system. The efficiency of the software will be known and any other insidious errors that are elusive to other testing techniques will be detected.

What is Domain Testing

Domain testing is the most frequently described test technique. Some authors write only about domain testing when they write about test design. Also read pdf tutorials about domain based testing. The basic notion is that you take the huge space of possible tests of an individual variable and subdivide it into subsets that are (in some way) equivalent. Then you test a representative from each subset. This type of testing also known as equivalence testing or boundary analysis.

Domain testing is a technique for testing software in which a minimum number of inputs are used to test the output of a system, to be sure that the system does not accept invalid and input values that are out of range.

It is one of the essential white box testing methods. Its major objective is to confirm if the system accepts input within the acceptable range and delivers the output that is needed. It equally confirms that inputs, indices, and conditions that are outside the valid or specified range are not accepted by the system.

Is domain knowledge needed for domain testing?

Some may say this is not necessary. But it is quite difficult for anyone to perform effectively or efficiently in a terrain he/she is not conversant with. If one is to do well, therefore, as a domain tester, there is the need to have basic domain knowledge. Now, let’s look at a few reasons why this is important.

• Online banking: Some of the basic things that a tester needs to test as far as online banking is concerned are login, bill payment, transfers, and lots more. A tester might require essential testing knowledge in order to login. However, doing bill payments and transfers require that the tester has to be an expert in the subject matter. He has to be conversant with the business logic on money flow.
• Retail: Examples of retail domains are Enterprise management, In-store solutions, and Warehouse management. In order to successfully run a domain test, the tester has to have an understanding of how things flow at each level. For instance, you need to know some things regarding a POS before you can successfully run a test on it.
• Healthcare system: how would someone who has no knowledge of the system handle it? He will certainly be a big risk to someone’s life.

Where is domain knowledge needed?

Domain knowledge for effective domain testing is needed in the following key areas:

• Mobile application testing
• Protocol testing
• Network testing
• VoIP applications
• Wireless application testing
• Banking applications

Skills needed for domain testing

If your dream is to be a good domain tester, you need to acquire some important skills. These skills include

• Testing skill
• Domain knowledge
• Automation skill
• Ability to work under pressure
• Bug hunting skill
• Programming skill
• Communication skill
• Technical skill
• Quick grasping

Conclusion

Domain testing requires that testers should have basic knowledge in the domain they are to test in order to be able to perform effectively.

PDF Tutorials

• Domain Testing Based on Character String Predicate
• Domain Testing Techniques with sample
• Use of Domain-Specific Modeling with Model-Based Testing

Volume testing is another type of non-functional part that is used to test a software application with a volume of data to determine its capacity to handle such data. The objective of subjecting a software application to volume testing is to determine the performance of the system with different volumes of data from the database.

Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system. Read a pdf tutorial about Experiments with High Volume Test Automation after this article.

Volume Testing, as its name implies, is testing that purposely subjects a system (both hardware and software) to a series of tests where the volume of data being processed is the subject of the test. Such systems can be transactions processing systems capturing real time sales or could be database updates and or data retrieval.

Volume testing will seek to verify the physical and logical limits to a system’s capacity and ascertain whether such limits are acceptable to meet the projected capacity of the organization’s business processing.

The checklist for volume testing

To verify the ability of the system to handle the volume of data fed into it, compare the outcome of the test with this checklist.

• Verify if any data is lost during the process.
• Pay attention to the system’s response
• Check the correctness of the data storage method.
• Check the data’s integrity whether it is overwritten during the testing without prior notice or not.

Paying close attention to these will offer a helping hand in identifying any loopholes that need to be blocked to promote the overall performance of the software.  Why is volume testing in software development important? A discussion of the benefits of conducting volume testing will answer this question.

The objectives of conducting volume testing in software development

The main objectives of performing volume testing on software applications are:

• To determine the load or volume at data at which the stability of the system will be compromised.
• To identify the issues that can prevent a system from reaching the desired volumetric targets or service level agreements.

Ensuring that a system is tested with these objectives in mind will assist the developers to have a system that meets the required benchmark.

Volume testing characteristics

The basic characteristics of volume testing are these:

• A small quantity of the data is taken for a test during the development stage of the software.
• As the volume of data used for the testing is increased, the performance of the program will nosedive.
• Design documents form the basis for generating test data.
• The data may be incorrect since it is only used to have an insight into the performance of the system.
• When you are through with the testing, the tester will log and track the results as the final step towards the completion of the test.

Volume testing examples

To drive home what volume testing really means, some typical examples are given below:

• If you are designing a system to be used on 1,000 computers, you should simulate the use of the system on the same number of computers. That is volume testing. All the real-life activities that will be performed by the users should be simulated and performed. This will include opening files, creating files, and processing data or any use the end user will likely use the system for.
• If you want to test the application with a database of a specific size, the database of the system should be expanded by adding more data to the system database until it meets the target.

Performing either of these activities or a related one will serve as an effective volume testing technique that will possibly give the best result.

A comprehensive and effective volume testing involves the following processes:

• Test management.
• Building test labs.
• Test plan development.
• Preparation of test strategy
• Preparation of the automation testing frameworks.
• Test scenario development.
• Creating test script.
• Executing test cases.
• Test scenario development.
• Providing measurable test reports.

When everything is put in place, a very simple and effective volume testing will be done to add more to the credibility of a software application to meet the objective it is designed for

What is Recovery Testing

Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications. It is basically testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.Recovery testing is an effective way to check the recovery rate of an application when it has suffered a huge blow of hardware failure or crash. It is a way of manually stimulating the failure of the software to test the possibility and the rate of its recovery.

Recovery Testing in Software

There is an array of software testing to ensure its efficiency. One of these testing techniques is non-functional testing. A non-functional testing is a term used for the part of the software that is not connected to a specific user action or function such as security. This is further sub-divided into many testing methods among which is Recovery testing.

How to perform recovery testing

Recovery testing can be performed in a number of ways. Some practical examples of stimulating software failure to test its recovery ability are:

• While a system is receiving some data from a network for processing, you can stimulate software failure by unplugging the system. After some minutes, you can plug the system again and test the ability of the software to continue receiving the data after the brief interruption.
• When a browser is working on some sessions, you can restart the system to stimulate software failure. After restarting the system, check the browser whether it can automatically recover the opened sessions.
• If you are downloading music to your mobile device by using your data, move to a place where there is no network. The downloading will be interrupted. Move back to where there is network and see how it goes. If the downloading continues from where it stops, the system has a good recovery rate.

These are typical example of what recovery testing is and the recovery rate is determined by some factors such as:

• The volume of the tested application.
• The amount of the restart points.
• The skills of the recovery testers.
• The type of tools available to promote the recovery.

Recovery process cycle

The recovery process is expected to have five different steps:

• Normal operation
• Disaster occurrence
• Disruption of operation
• Clearing disaster through the recovery process
• Pulling together all the processes and information necessary to restore the software to normal.

A good software system is expected to seamlessly shift from one stage to the other without hitches to ensure that it has an acceptable recovery.

Instructions

Some precautionary measures should be taken before carrying out recovery testing on a software program. A couple of what should be considered are:

• A condition that is nearly identical to the real life situation should be created. Caution should be taken to ensure that the hardware, protocol, software, and firmware should be very close to the condition that is imitated.
• The complete checking of the system should be carried out as well as ensure that there is an identical configuration despite how expensive the process will be.
• The size of the backup is expected to be the same as the size of the original source of the information in some backup systems.
• An online backup may be a good backup idea to prevent total loss of information during recovery testing. However, it is important that you test the system’s restore ability before carrying out the recovery testing on a particular software program.

Advantages of recovery testing

1. It eliminates risks: Recovery testing exposes potential flaws that can lead to the failure of the product when used by the end users.
2. It improves the quality of the system: After detecting a flaw, it will be corrected to ensure the system is working as expected. This leads to improved system quality.

This is a necessity for the efficiency of a software program.

Pdf tutorials

• Recovery Testing – detailed tutorial in pdf

What is Smoke Testing

This type of testing is also called sanity testing. But there are some difference between Smoke and Sanity testing. and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level. Read pdf tutorials about Smoke test at the end of this page. A test of new or repaired equipment by turning it on. If it smokes… guess what… it doesn’t work! The term also refers to testing the basic functions of software. The term was originally coined in the manufacture of containers and pipes, where smoke was introduced to determine if there were any leaks.

The advantages of smoke testing are shown below.

A common practice at Microsoft and some other shrink-wrap software companies is the “daily build and smoke test” process. Every file is compiled, linked, and combined into an executable program every day, and the program is then put through a “smoke test,” a relatively simple check to see whether the product “smokes” when it runs.

 
View the difference between Smoke vs Sanity Test in the following diagram

 
PDF Tutorials

• Details about Smoke testing in PDF

What is Usability Testing

This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user. You can also read pdf tutorials about usability tests after this description.

Usability testing is the process of working with end-users directly and indirectly to assess how the user perceives a software package and how they interact with it. This process will uncover areas of difficulty for users as well as areas of strength.

The goal of usability testing should be to limit and remove difficulties for users and to leverage areas of strength for maximum usability. This testing should ideally involve direct user feedback, indirect feedback (observed behavior), and when possible computer supported feedback. Computer supported feedback is often (if not always) left out of this process. Computer supported feedback can be as simple as a timer on a dialog to monitor how long it takes users to use the dialog and counters to determine how often certain conditions occur (ie. error messages, help messages, etc). Often, this involves trivial modifications to existing software, but can result in tremendous return on investment.

Ultimately, usability testing should result in changes to the delivered product in line with the discoveries made regarding usability. These changes should be directly related to real-world usability by average users. As much as possible, documentation should be written supporting changes so that in the future, similar situations can be handled with ease.

 
PDF Tutorials

• Usability testing in usability engineering – PDF guide
• The science of usability testing