How to run your postman (Newman) automation collection in a Continuous Integration environment like Azure DevOps build Pipeline?

 

How to run your postman (Newman) automation collection in a Continuous Integration environment like Azure DevOps build Pipeline?

Postman Introduction:

Postman is a tool that simplifies each step of building and testing APIs. If you've ever used it in your development or testing phase, you must already know what an incredible tool it is.

It presents the possibility to automate your API tests and integrate them into your CI/CD pipeline to make certain that any code changes won’t introduce any new regression issues in upper environments like QA, Stage, Production, etc.

Newman Introduction:

Newman is a tool, using which one can effortlessly run and test Postman Collections directly from the command line.

Azure DevOps CI/CD Introduction:

Azure DevOps by Microsoft Azure is one of the leading tools that automate CI/CD's process. Using continuous integration and continuous deployment with Azure DevOps, these pipelines are used to construct build-deploy-test workflows used mainly in continuous testing (CT).

Here are the steps to configure your CI/CD pipeline for postman automation collection execution:

Step 1: First thing first, let’s export these things via our postman tool:

• Postman automation Collection that has automated test added that we would like to configure as part of CI/CD pipeline
• Environment Collection  
• If you have any external test data CSV file to read data then copy/collect that too as we need to push all these into code repo.

Step 2: Before creating the pipeline we must push our code (collection file(s) and test data file(s)) in Git repository hosting services like GitHub, Bitbucket, Azure repo, etc.

For this example, we have created a new Azure DevOps project and committed all code. If you already have your own repo, then please use it.

Step 3: Create Pipeline in Azure DevOps: 

Click on Pipeline >> New Pipeline

Name your pipeline, let’s say “PostmanDemo-CI” and Choose Agent (This is where all the execution will happen; in case your APIs are restricted to execute through certain IPs only then make sure you create a Private Hosted Agent and use that to run your build pipeline.

Click on the Get Sources and confirm the following points:
Repository: It should point to the correct repository where your postman test collection and environment collection are located.
Default branch: Choose the suitable branch (in general this will be the master branch)   

Modify Agent job
Display Name: Add a name that you like (ex: API Automation)    
Agent pool: Choose the correct Agent pool from the drop-down or use <inherit from pipeline>.

Create the following 4 tasks in your build pipeline:

I. Add a task to install Newman in your build agent

• Click on the + button in your agent panel
• Search for task “Command line” (Or you can create an npm task)
• Click on Add

Now let’s configure this task::

• Display name: Mention task name as “Install Newman in your build agent”
• Script: npm install -g Newman
• Click on the Advanced section and make sure the Working directory is set to $(System.DefaultWorkingDirectory) which is an inbuilt Azure DevOps variable that will point to your source code. 

II. Add a task to install Newman-reporter-htmlextra in your build agent (This report is very comprehensive and is very useful to publish results in Azure DevOps. You will see that soon.)

• Click on the + button in your agent panel
• Search for task “npm” (Or you can create a Command-line task)
• Click on Add

Now let’s configure this task::

• Display name: Mention task name as “Install Newman-reporter-htmlextra'”
• Command: custom

Command and arguments: install -g newman-reporter-htmlextra

III. Add a task to run postman collection tests through Newman

• Click on the + button in your agent panel
• Search for task “Command line”
• Click on Add

Now let’s configure this task::

• Display name: Run API Collection
• Script :newman run {yourCollection. json} -e {yourenvironment.json} -r htmlextra --reporters cli,junit,htmlextra --reporter-htmlextra-export TestReport.html --reporter-htmlextra-browserTitle "Test Summary Report"
• Working Directory: Choose a directory where your tests are in the git repository.
• Control Options:  Check these checkboxes: Enabled and Continue on error 

Note: If you are using any external test data csv file to read data, you can use:

newman run {yourCollection. json} -e {yourenvironment.json} -d {yourTestData.csv} -r htmlextra --reporters cli,junit,htmlextra --reporter-htmlextra-export {yourTestReportname.html} --reporter-htmlextra-browserTitle "{yourTestSummaryReport"

IV. Add a task to upload Postman Html Report 

• Click on the + button in your agent panel
• Search for task “Upload Postman Html Report”. This provides tasks for Publishing Postman / Newman HTML Reports into built into Azure Storage. Reports can be viewed as a tab on the Build and Release result page. Each Tab contains embedded reports as well as direct download links.

(Note* If you don’t see this extension then you must add it through the MarketPlace option provided in the last tab of the “Add tasks” creation section).

• Click on Add

Now let’s configure this task::

• Display name: Upload Postman Html Report
• Report Path : $(System.DefaultWorkingDirectory)
• Report tab name: Postman
• Control Options:  Check: Enabled and Uncheck: Continue on error 

Step 4: Click on Save & Queue so the build runs now. You can enable continuous integration or Schedule the build using the Triggers tab while creating/editing this pipeline.

Verifying the results after execution. If everything is configured properly, all the tasks should pass and will look like below:

Viewing the Test results > Click on the pipeline and access the Postman tab to view or download the Test Report.

Once your build pipeline is configured properly and is working fine, then you can add this into your Release pipeline.

Regardless of the CI tool that you are using (Azure pipeline, Jenkins, Hudson, or any other one), the steps above remain almost the same. We hope that this step-by-step guide to demonstrate the integration of Postman/Newman automated tests in a Microsoft Azure Pipeline helps you to deliver your application more efficiently.

Enjoyed reading this article? Please share the knowledge with your friends and colleagues.