.svg)
.svg)
In the last few years, we have an exponential increase in the development and use of APIs. We are in the era of API-first companies like Stripe, Twilio, Mailgun etc. where the entire product or service is exposed via REST APIs. Web applications also today are powered by REST-based Web Services. APIs today encapsulate critical business logic with high SLAs. Hence it is important to test APIs as part of the continuous integration process to reduce errors, improve predictability and catch nasty bugs.
In the context of API development, Postman is great REST client to test APIs. Although Postman is not just a REST Client, it contains a full-featured testing sandbox that lets you write and execute Javascript based tests for your API.
Postman comes with a nifty CLI tool - Newman. Newman is the Postman’s Collection Runner engine that sends API requests, receives the response and then runs your tests against the response. Newman lets developments easily integrate Postman into continuous integration systems like Jenkins. Some of the important features of Postman & Newman include:-
We are going to look at all these features, some are intuitive and some not so much unless you've been using Postman for a while.
You can install Postman either as a Chrome extension or as a native application
Later, can then look it up in your installed apps and open it. You can choose to Sign Up & create an account if you want, this is important especially for saving your API collections and accessing them anytime on any machine. However, for this article, we can skip this. There's a button for that towards the bottom when you first launch the app.
Postman Collections in simple words is a collection of tests. It is essentially a test suite of related tests. These tests can be scenario-based tests or sequence/workflow-based tests.
There's a Collections tab on the top left of Postman, with an example Postman Echo collection. You can open and go through it.
Just like in the above screenshot, select a API request and click on the Tests. Check the first line:
| tests["response code is 200"] = responseCode.code === 200; |
The above line is a simple test to check if the response code for the API is 200. This is the pattern for writing Assertions/Tests in Postman (using JavaScript), and this is actually how you are going to write the tests for API’s need to be tested.You can open the other API requests in the POSTMAN Echo collection to get a sense of how requests are made.
To make your own collection, click on the 'Add Collection' button on the top left of Postman and call it “Test API”
You will be prompted to give details about the collection, I've added a name Github API and given it a description.
Clicking on Create should add the collection to the left pane, above, or below the example "POSTMAN Echo" collection.
If you need a hierarchy for maintaining relevance between multiple API’s inside a collection, APIs can further be added to a folder inside a collection. Folders are a great way of separating different parts of your API workflow. You can be added folders through the “3 dot” button beside Collection Name:
Eg.: name the folder “Get Calls” and give a description once again.
Now that we have the folder, the next task is to add an API call that is related to the TEST_API_COLLECTION to that folder. That API call is to https://api.github.com/.
If you still have one of the TEST_API_COLLECTION collections open, you can close it the same way you close tabs in a browser, or just click on the plus button to add a new tab on the right pane where we make requests.
Type in or paste in https://api.github.com/ and press Send to see the response.
Once you get the response, you can click on the arrow next to the Save button on the far right, and select Save As, a pop up will be displayed asking where to save the API call.
Give a name, it can be the request URL, or a name like “GET Github Basic”, and a description, then choose the collection and folder, in this case, TEST_API_COLLECTION> GET CALLS, then click on Save. The API call will be added to the Github Root API folder on the left pane.
Whenever you click on this request from the collection, it will open in the center pane.
We've seen that the GET Github Basic request has a JSON response, which is usually the case for most of the APIs.This response has properties such as current_user_url, emails_url, followers_url and following_url to pick a few. The current_user_url has a value of https://api.github.com/user. Let's add a test, for this URL. Click on the 'GET Github Basic' and click on the test tab in the section just below where the URL is put.
You will notice on the right pane, we have some snippets which Postman creates when you click so that you don't have to write a lot of code. Let's add Response Body: JSON value check. Clicking on it produces the following snippet.
| var jsonData = JSON.parse(responseBody); | |
| tests["Your test name"] = jsonData.value === 100; |
From these two lines, it is apparent that Postman stores the response in a global object called responseBody, and we can use this to access response and assert values in tests as required.
Postman also has another global variable object called tests, which is an object you can use to name your tests, and equate it to a boolean expression. If the boolean expression returns true, then the test passes.
| tests['some random test'] = x === y |
If you click on Send to make the request, you will see one of the tests failing.
Lets create a test that relevant to our usecase.
| var jsonData = JSON.parse(responseBody); | |
| var usersURL = "https://api.github.com/user" | |
| tests["Gets the correct users url"] = jsonData.current_user_url === usersURL; |
Clicking on 'Send', you'll see the test passing.
Let's modify the test further to test some of the properties we want to check
Ideally the things to be tested in an API Response Body should be:
| tests["Status code is 200"] = responseCode.code === 200; | |
| tests["Response time is less than 200ms"] = responseTime < 200; | |
| tests["Response time is acceptable"] = _.inRange(responseTime, 0, 500); | |
| tests["Body is not empty"] = (responseBody!==null || responseBody.length!==0); |
Once you've set up all your collections and written tests for them, it may be tedious to go through them one by one and clicking send to see if a given collection test passes. This is where Newman comes in. Newman is a command-line collection runner for Postman.
All you need to do is export your collection and the environment variables, then use Newman to run the tests from your terminal.
NOTE: Make sure you've clicked on 'Save' to save your collection first before exporting.
So the first step is to export your collection and environment variables. Click on the Menu icon for Github API collection, and select export.
Select version 2, and click on "Export"
Save the JSON file in a location you can access with your terminal. I created a local directory/folder called "postman" and saved it there.
Install Newman CLI globally, then navigate to the where you saved the collection.
| npm install -g newman | |
| cd postman |
Using Newman is quite straight-forward, and the documentation is extensive. You can even require it as a Node.js module and run the tests there. However, we will use the CLI.
Once you are in the directory, run newman run <collection_name.json>, </collection_name.json>replacing the collection_name with the name you used to save the collection.
| newman run TEST_API_COLLECTION.postman_collection.json |
Newman provides a rich set of options to customize a run. A list of options can be retrieved by running it with the -h flag.
| $ newman run -h | |
| Options - Additional args: | |
| Utility: | |
| -h, --help output usage information | |
| -v, --version output the version number | |
| Basic setup: | |
| --folder [folderName] Specify a single folder to run from a collection. | |
| -e, --environment [file|URL] Specify a Postman environment as a JSON [file] | |
| -d, --data [file] Specify a data file to use either json or csv | |
| -g, --global [file] Specify a Postman globals file as JSON [file] | |
| -n, --iteration-count [number] Define the number of iterations to run | |
| Request options: | |
| --delay-request [number] Specify a delay (in ms) between requests [number] --timeout-request [number] Specify a request timeout (in ms) for a request | |
| Misc.: | |
| --bail Stops the runner when a test case fails | |
| --silent Disable terminal output --no-color Disable colored output | |
| -k, --insecure Disable strict ssl | |
| -x, --suppress-exit-code Continue running tests even after a failure, but exit with code=0 | |
| --ignore-redirects Disable automatic following of 3XX responses |
Lets try out of some of the options.
Lets use the -n option to set the number of iterations to run the collection.
| $ newman run mycollection.json -n 10 # runs the collection 10 times |
To provide a different set of data, i.e. variables for each iteration, you can use the -d to specify a JSON or CSV file. For example, a data file such as the one shown below will run 2 iterations, with each iteration using a set of variables.
| [{ | |
| "url": "http://127.0.0.1:5000", | |
| "user_id": "1", | |
| "id": "1", | |
| "token_id": "123123", | |
| },{ | |
| "url": "http://postman-echo.com", | |
| "user_id": "2", | |
| "id": "2", | |
| "token_id": "899899", | |
| }]$ newman run mycollection.json -d data.json |
Alternately, the CSV file for the above set of variables would look like:
| url, user_id, id, token_id | |
| http://127.0.0.1:5000, 1, 1, 123123123 | |
| http://postman-echo.com, 2, 2, 899899 |
Each environment is a set of key-value pairs, with the key as the variable name. These Environment configurations can be used to differentiate between configurations specific to your execution environments eg. Dev, Test & Production.
To provide a different execution environment, you can use the -e to specify a JSON or CSV file. For example, a environment file such as the one shown below will provide the environment variables globally to all tests during execution.
| postman_dev_env.json | |
| { | |
| "id": "b5c617ad-7aaf-6cdf-25c8-fc0711f8941b", | |
| "name": "dev env", | |
| "values": [ | |
| { | |
| "enabled": true, | |
| "key": "env", | |
| "value": "dev.example.com", | |
| "type": "text" | |
| } | |
| ], | |
| "timestamp": 1507210123364, | |
| "_postman_variable_scope": "environment", | |
| "_postman_exported_at": "2017-10-05T13:28:45.041Z", | |
| "_postman_exported_using": "Postman/5.2.1" | |
| } |
Newman, by default, exits with a status code of 0 if everything runs well i.e. without any exceptions. Continuous integration tools respond to these exit codes and correspondingly pass or fail a build. You can use the –bail flag to tell Newman to halt on a test case error with a status code of 1 which can then be picked up by a CI tool or build system.
| $ newman run PostmanCollection.json -e environment.json --bail newman |
Postman and Newman can be used for a number of test cases, including creating usage scenarios, Suites, Packs for your API Test Cases. Further NEWMAN / POSTMAN can be very well Integrated with CI/CD Tools such as Jenkins, Travis etc.
Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.
We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.
In the last few years, we have an exponential increase in the development and use of APIs. We are in the era of API-first companies like Stripe, Twilio, Mailgun etc. where the entire product or service is exposed via REST APIs. Web applications also today are powered by REST-based Web Services. APIs today encapsulate critical business logic with high SLAs. Hence it is important to test APIs as part of the continuous integration process to reduce errors, improve predictability and catch nasty bugs.
In the context of API development, Postman is great REST client to test APIs. Although Postman is not just a REST Client, it contains a full-featured testing sandbox that lets you write and execute Javascript based tests for your API.
Postman comes with a nifty CLI tool - Newman. Newman is the Postman’s Collection Runner engine that sends API requests, receives the response and then runs your tests against the response. Newman lets developments easily integrate Postman into continuous integration systems like Jenkins. Some of the important features of Postman & Newman include:-
We are going to look at all these features, some are intuitive and some not so much unless you've been using Postman for a while.
You can install Postman either as a Chrome extension or as a native application
Later, can then look it up in your installed apps and open it. You can choose to Sign Up & create an account if you want, this is important especially for saving your API collections and accessing them anytime on any machine. However, for this article, we can skip this. There's a button for that towards the bottom when you first launch the app.
Postman Collections in simple words is a collection of tests. It is essentially a test suite of related tests. These tests can be scenario-based tests or sequence/workflow-based tests.
There's a Collections tab on the top left of Postman, with an example Postman Echo collection. You can open and go through it.
Just like in the above screenshot, select a API request and click on the Tests. Check the first line:
| tests["response code is 200"] = responseCode.code === 200; |
The above line is a simple test to check if the response code for the API is 200. This is the pattern for writing Assertions/Tests in Postman (using JavaScript), and this is actually how you are going to write the tests for API’s need to be tested.You can open the other API requests in the POSTMAN Echo collection to get a sense of how requests are made.
To make your own collection, click on the 'Add Collection' button on the top left of Postman and call it “Test API”
You will be prompted to give details about the collection, I've added a name Github API and given it a description.
Clicking on Create should add the collection to the left pane, above, or below the example "POSTMAN Echo" collection.
If you need a hierarchy for maintaining relevance between multiple API’s inside a collection, APIs can further be added to a folder inside a collection. Folders are a great way of separating different parts of your API workflow. You can be added folders through the “3 dot” button beside Collection Name:
Eg.: name the folder “Get Calls” and give a description once again.
Now that we have the folder, the next task is to add an API call that is related to the TEST_API_COLLECTION to that folder. That API call is to https://api.github.com/.
If you still have one of the TEST_API_COLLECTION collections open, you can close it the same way you close tabs in a browser, or just click on the plus button to add a new tab on the right pane where we make requests.
Type in or paste in https://api.github.com/ and press Send to see the response.
Once you get the response, you can click on the arrow next to the Save button on the far right, and select Save As, a pop up will be displayed asking where to save the API call.
Give a name, it can be the request URL, or a name like “GET Github Basic”, and a description, then choose the collection and folder, in this case, TEST_API_COLLECTION> GET CALLS, then click on Save. The API call will be added to the Github Root API folder on the left pane.
Whenever you click on this request from the collection, it will open in the center pane.
We've seen that the GET Github Basic request has a JSON response, which is usually the case for most of the APIs.This response has properties such as current_user_url, emails_url, followers_url and following_url to pick a few. The current_user_url has a value of https://api.github.com/user. Let's add a test, for this URL. Click on the 'GET Github Basic' and click on the test tab in the section just below where the URL is put.
You will notice on the right pane, we have some snippets which Postman creates when you click so that you don't have to write a lot of code. Let's add Response Body: JSON value check. Clicking on it produces the following snippet.
| var jsonData = JSON.parse(responseBody); | |
| tests["Your test name"] = jsonData.value === 100; |
From these two lines, it is apparent that Postman stores the response in a global object called responseBody, and we can use this to access response and assert values in tests as required.
Postman also has another global variable object called tests, which is an object you can use to name your tests, and equate it to a boolean expression. If the boolean expression returns true, then the test passes.
| tests['some random test'] = x === y |
If you click on Send to make the request, you will see one of the tests failing.
Lets create a test that relevant to our usecase.
| var jsonData = JSON.parse(responseBody); | |
| var usersURL = "https://api.github.com/user" | |
| tests["Gets the correct users url"] = jsonData.current_user_url === usersURL; |
Clicking on 'Send', you'll see the test passing.
Let's modify the test further to test some of the properties we want to check
Ideally the things to be tested in an API Response Body should be:
| tests["Status code is 200"] = responseCode.code === 200; | |
| tests["Response time is less than 200ms"] = responseTime < 200; | |
| tests["Response time is acceptable"] = _.inRange(responseTime, 0, 500); | |
| tests["Body is not empty"] = (responseBody!==null || responseBody.length!==0); |
Once you've set up all your collections and written tests for them, it may be tedious to go through them one by one and clicking send to see if a given collection test passes. This is where Newman comes in. Newman is a command-line collection runner for Postman.
All you need to do is export your collection and the environment variables, then use Newman to run the tests from your terminal.
NOTE: Make sure you've clicked on 'Save' to save your collection first before exporting.
So the first step is to export your collection and environment variables. Click on the Menu icon for Github API collection, and select export.
Select version 2, and click on "Export"
Save the JSON file in a location you can access with your terminal. I created a local directory/folder called "postman" and saved it there.
Install Newman CLI globally, then navigate to the where you saved the collection.
| npm install -g newman | |
| cd postman |
Using Newman is quite straight-forward, and the documentation is extensive. You can even require it as a Node.js module and run the tests there. However, we will use the CLI.
Once you are in the directory, run newman run <collection_name.json>, </collection_name.json>replacing the collection_name with the name you used to save the collection.
| newman run TEST_API_COLLECTION.postman_collection.json |
Newman provides a rich set of options to customize a run. A list of options can be retrieved by running it with the -h flag.
| $ newman run -h | |
| Options - Additional args: | |
| Utility: | |
| -h, --help output usage information | |
| -v, --version output the version number | |
| Basic setup: | |
| --folder [folderName] Specify a single folder to run from a collection. | |
| -e, --environment [file|URL] Specify a Postman environment as a JSON [file] | |
| -d, --data [file] Specify a data file to use either json or csv | |
| -g, --global [file] Specify a Postman globals file as JSON [file] | |
| -n, --iteration-count [number] Define the number of iterations to run | |
| Request options: | |
| --delay-request [number] Specify a delay (in ms) between requests [number] --timeout-request [number] Specify a request timeout (in ms) for a request | |
| Misc.: | |
| --bail Stops the runner when a test case fails | |
| --silent Disable terminal output --no-color Disable colored output | |
| -k, --insecure Disable strict ssl | |
| -x, --suppress-exit-code Continue running tests even after a failure, but exit with code=0 | |
| --ignore-redirects Disable automatic following of 3XX responses |
Lets try out of some of the options.
Lets use the -n option to set the number of iterations to run the collection.
| $ newman run mycollection.json -n 10 # runs the collection 10 times |
To provide a different set of data, i.e. variables for each iteration, you can use the -d to specify a JSON or CSV file. For example, a data file such as the one shown below will run 2 iterations, with each iteration using a set of variables.
| [{ | |
| "url": "http://127.0.0.1:5000", | |
| "user_id": "1", | |
| "id": "1", | |
| "token_id": "123123", | |
| },{ | |
| "url": "http://postman-echo.com", | |
| "user_id": "2", | |
| "id": "2", | |
| "token_id": "899899", | |
| }]$ newman run mycollection.json -d data.json |
Alternately, the CSV file for the above set of variables would look like:
| url, user_id, id, token_id | |
| http://127.0.0.1:5000, 1, 1, 123123123 | |
| http://postman-echo.com, 2, 2, 899899 |
Each environment is a set of key-value pairs, with the key as the variable name. These Environment configurations can be used to differentiate between configurations specific to your execution environments eg. Dev, Test & Production.
To provide a different execution environment, you can use the -e to specify a JSON or CSV file. For example, a environment file such as the one shown below will provide the environment variables globally to all tests during execution.
| postman_dev_env.json | |
| { | |
| "id": "b5c617ad-7aaf-6cdf-25c8-fc0711f8941b", | |
| "name": "dev env", | |
| "values": [ | |
| { | |
| "enabled": true, | |
| "key": "env", | |
| "value": "dev.example.com", | |
| "type": "text" | |
| } | |
| ], | |
| "timestamp": 1507210123364, | |
| "_postman_variable_scope": "environment", | |
| "_postman_exported_at": "2017-10-05T13:28:45.041Z", | |
| "_postman_exported_using": "Postman/5.2.1" | |
| } |
Newman, by default, exits with a status code of 0 if everything runs well i.e. without any exceptions. Continuous integration tools respond to these exit codes and correspondingly pass or fail a build. You can use the –bail flag to tell Newman to halt on a test case error with a status code of 1 which can then be picked up by a CI tool or build system.
| $ newman run PostmanCollection.json -e environment.json --bail newman |
Postman and Newman can be used for a number of test cases, including creating usage scenarios, Suites, Packs for your API Test Cases. Further NEWMAN / POSTMAN can be very well Integrated with CI/CD Tools such as Jenkins, Travis etc.
.svg)
.svg)
