Skip to main content

How to integrate TestNG Test Automation Results (Selenium/RestAssured/Appium) with TestRail?

 


To integrate TestRail with any automation suite, we can use the TestRail APIs with the following basic flow:

  • Create test cases in TestRail. 
  • Provide those unique Test IDs from TestRail to your TestNG tests.
  • Run automation:
    • Create a test suite in TestRail before test execution using the TestRail API.
    • And then post the run results according to test automation results using the TestRail API.
  1. Create test cases in TestRail:



       2. Creating a custom annotation and calling it as TestRails:






     3. After that, we associate our test cases with actual Test Rails ID’s like:




    4. These ids 1, 2, 3, and 4 are test case ID that we got from TestRail test cases, you can hover over your test case and see that ID in the URL (as shown below):






    5. Create a Test Run in TestRail through code,  as you can look at that we have not created any Test Run manually (see below):




Snapshot of TestRail before execution of automation suite:







6. Retrieve Test Case ID from Annotation:




7. Run the test cases and update Test Rail Status:





Let’s run the test cases and see if it is working:







As you can see that 2 test cases are passed and the other 2 are failed as expected based on the assertions we added, let’s see the status in TestRail:

Voila… Worked perfectly fine. (The test run is created; plus, test statuses are updated.)

Snapshot of TestRail after execution of automation suite:






And for the failed case, we can see the error message also as we added that comment in the code:





Points to remember:


There are 2 ways in which we can update our TestRail results after a test case execution:

    • Create a TestNG listener that will listen for test results and call the TestRail API to update it with any results it finds
    • Call the TestRail API inside the test method to update results (I have used this one for now)

There are 2 methods to update test results:

  • add_result
  • add_result_for_case


The difference between these two is that “add_result” takes the Test ID, "add_result_for_case" takes the Case ID and Run ID. We can use any method depending on the automation approach selected but "add_result_for_case" comes with an advantage over "add_result" i.e., while creating a test run as per the iteration, the test ID changes corresponding to the test run ID, but the test case IDs remains the same. This makes it easy to maintain the test case ID and run the automation suite repeatedly.



Do comment if you would like to have the entire code of this integration piece.


To Integrate Automated TestNG @Test with @DataProvider Results in TestRail, please check this blog:









Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

The use of Verbose attribute in testNG or POM.xml (maven-surefire-plugin)

At times, we see some weird behavior in your testNG execution and feel that the information displayed is insufficient and would like to see more details. At other times, the output on the console is too verbose and we may want to only see the errors. This is where a verbose attribute can help you- it is used to define the amount of logging to be performed on the console. The verbosity level is 0 to 10, where 10 is most detailed. Once you set it to 10, you'll see that console output will contain information regarding the tests, methods, and listeners, etc. <suite name="Suite" thread-count="5" verbose="10"> Note* You can specify -1 and this will put TestNG in debug mode. The default level is 0. Alternatively, you can set the verbose level through attribute in "maven-surefire-plugin" in pom.xml, as shown in the image. #testNG #automationTesting #verbose # #testAutomation
Post a Comment
Read more

How to Unzip files in Selenium (Java)?

1) Using Java (Lengthy way) : Create a utility and use it:>> import java.io.BufferedOutputStream; import org.openqa.selenium.io.Zip; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.util.zip.ZipEntry; import java.util.zip.ZipInputStream;   public class UnzipUtil {     private static final int BUFFER_SIZE = 4096;     public void unzip (String zipFilePath, String destDirectory) throws IOException {         File destDir = new File(destDirectory);         if (!destDir.exists()) {             destDir.mkdir();         }         ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));         ZipEntry entry = zipIn.getNextEntry();         // to iterates over entries in the zip folder         while (en...
Post a Comment
Read more

Stop Overengineering: Why Test IDs Beat AI-Powered Locator Intelligence for UI Automation

  We have all read the blogs. We have all seen the charts showing how Generative AI can "revolutionize" test automation by magically resolving locators, self-healing broken selectors, and interpreting UI changes on the fly. There are many articles that paints a compelling picture of a future where tests maintain themselves. Cool story. But let’s take a step back. Why are we bending over backward to make tests smart enough to deal with ever-changing DOMs when there's a simpler, far more sustainable answer staring us in the face? -             Just use Test IDs. That’s it. That’s the post. But since blogs are supposed to be more than one sentence, let’s unpack this a bit. 1. Test IDs Never Lie (or Change) Good automation is about reliability and stability. Test IDs—like data-testid ="submit-button"—are predictable. They don’t break when a developer changes the CSS class, updates the layout, or renames an element. You know...
Post a Comment
Read more