Skip to main content

False-Positive & False-Negative in automation


False-Negative is a case where your test result comes out to be positive, but in actuality, an issue is present in the system and the functionality is not working as it should. And Vice-versa is False Positive, which is a case where the test execution results show an error, even though everything is working as intended. Fundamentally, both these have always posed a challenge for web automated testing but I guess it’s ok to say that a False-negative is more hurting than a False-Positive, as the former creates a false sense of security (a lie) and would definitely end up costing much more to us.

I agree that a False-Positive too consume a lot of our's time and worth. On average 70% of automated test case failures are False-Positives due to which testers spend an average of 1/3rd of their time analyzing, correcting and reporting results that actually should NOT need any attention at all. In fact, with CI/CD implementation, running automated tests every night or after every commit to the master branch, time wasted by testers to triage these false positives easily outweighs the quicker test results provided by our automation suite.

Do not trust on every tool that you use blindly, a few ways through which we can handle False-Positives/False-Negatives:

  • Put solid assertions, cover both +ve as well as -ve assertions. While the creation of your tests, make sure your tests fail when they should.
  • Revisit your tests regularly. Peer-Review them regularly, not only at the time of the creation of your tests suite.
  • Remove tests from your suite if they no longer serve a purpose. I know it's debatable.
  • Review your Test report properly.
  • Go for both manual and automation testing, so that you get more confidence that there are no False-Positives/False-Negatives in your system.
  • Dynamic synchronization of objects.
  • Use correct test data and revisit it regularly. Don't always go with independent or pre-set test data- your application test data should be in line with your test strategy.


What best practices do you follow to prevent and detect false positives and false negatives in your test automation?


Labels:

Comments

Post a Comment

Popular posts from this blog

How to Unzip files in Selenium (Java)?

1) Using Java (Lengthy way) : Create a utility and use it:>> import java.io.BufferedOutputStream; import org.openqa.selenium.io.Zip; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.util.zip.ZipEntry; import java.util.zip.ZipInputStream;   public class UnzipUtil {     private static final int BUFFER_SIZE = 4096;     public void unzip (String zipFilePath, String destDirectory) throws IOException {         File destDir = new File(destDirectory);         if (!destDir.exists()) {             destDir.mkdir();         }         ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));         ZipEntry entry = zipIn.getNextEntry();         // to iterates over entries in the zip folder         while (entry != null) {             String filePath = destDirectory + File.separator + entry.getName();             if (!entry.isDirectory()) {                 extractFile (zipIn, filePath);            
Post a Comment
Read more

Encode/Decode the variable/response using Postman itself

We get a lot of use cases where we may have to implement Base64 encoding and/or decoding while building our APIs. And, if you are wondering if it is possible to encode/decode the variable/response using Postman itself or how to encode/decode the token or password in postman and save it in a variable? To Base64 encode/decode, the quickest way is to use JavaScript methods btoa, atob: atob - It turns base64-encoded ASCII data back to binary. btoa - It turns binary data to base64-encoded ASCII. Sample code : var responseBody = pm.response.json(); var parsedPwd = JSON.parse(atob(responseBody.password)); // presuming password is in the payload pm.collectionVariables.set("password", parsedPwd);
Post a Comment
Read more

The use of Verbose attribute in testNG or POM.xml (maven-surefire-plugin)

At times, we see some weird behavior in your testNG execution and feel that the information displayed is insufficient and would like to see more details. At other times, the output on the console is too verbose and we may want to only see the errors. This is where a verbose attribute can help you- it is used to define the amount of logging to be performed on the console. The verbosity level is 0 to 10, where 10 is most detailed. Once you set it to 10, you'll see that console output will contain information regarding the tests, methods, and listeners, etc. <suite name="Suite" thread-count="5" verbose="10"> Note* You can specify -1 and this will put TestNG in debug mode. The default level is 0. Alternatively, you can set the verbose level through attribute in "maven-surefire-plugin" in pom.xml, as shown in the image. #testNG #automationTesting #verbose # #testAutomation
Post a Comment
Read more