[qase-newman-reporter] Handle newman iteration data as test parameters
complete
T
Taber Hust
Postman/newman has a concept of "Iteration data" which allows running the postman collection multiple times, with different data sets. For instance, you may want to run your collection with different configuration, different user, etc (see https://learning.postman.com/docs/collections/using-newman-cli/newman-options/#data-file-example for an example).
==============================
Current qase-newman-reporter behavior
==============================
Each iteration is treated as a "retry" of each request
This leads to false-positives in the test run. For instance, we iterate over our collection to execute tests against 3 different versions of our database, and it should pass in ALL versions. However, because the iterations are treated as retries, a success in 1 database will hide the failure in the other two.
In the attached screenshot, "search > single > exact" request ran 3 times (3 iterations), and 2 out of the 3 iterations failed, but the test was still marked as passed. Our expectation is that ALL iterations must pass.
==================================
Requested behavior for qase-newman-reporter
==================================
Iteration data should be treated as "parameters" for each test case so they can be logged separately.
Because iteration data can be complex JSON structures, it would be beneficial to have the option to explicitly define the key(s) from the structure that should be treated as the parameter. For instance, we have data files with a "kb_version" property (which is the unique identifier) then each object has a set of keys that are used by tests to verify the responses from our API depending on the "kb_version" passed in the request. So we would like to specify "kb_version" as a parameter for our tests and that reflect in our test cases automatically created in Qase.
Furthermore, because iteration data in newman is at a collection level, we DO NOT want to have to specify the parameter on each and every request, we would prefer to pass in a single parameter key at the config level that applies automatically to every request in the collection.
Ilya Volkov
Hi @Taber Hust!
The request from your latest comment has also been implemented - bump up to the version 2.0.3 :)
Here are the examples in the documentation: https://github.com/qase-tms/qase-javascript/blob/main/qase-newman/docs/usage.md#example-tests
T
Taber Hust
Ilya Volkov it works like a charm! Thank you for the very quick update! This will save us loads of time, and result in much more accurate test run results!
I really appreciate the team adopting this change so quickly!!
Ilya Volkov
Taber Hust Great to hear that, Taber! Passing along your thanks to the team responsible :)
Ilya Volkov
complete
Hi everyone!
We've added the support for utilizing parameters from data files with Newman reporter in the latest version.
Please, update the reporter to version 2.0.2, and refer to this documentation for instruction on usage:
Thank you!
T
Taber Hust
Thanks Ilya Volkov!
I've done some substantial testing, and if I follow the steps by adding the required comment in each request, it works as expected!
It's worth noting, you need to have "Update" tests via automation enabled in your settings to make it work; otherwise, you end up with some incomplete parameter definition, because the 2+ iterations are unable to update the test to add additional parameters.
I do have one concern though - we are forced to add this comment to every single request? An iteration data file in Postman applies to the whole collection (every request). So it seems unnecessary that I need to specify on every request the qase parameter comment. I tried to use the auto flag in the config, and it worked ok but its far from ideal, our iteration data files include huge datasets used for validating responses in that iteration, so it results in very verbose and unnecessary parameters, so we really want to look at only a few key properties.
Postman supports "Tests" at a folder and collection level. When done this way, the test script applies to every single request in that folder or collection.
I tried to add the qase.parameters specification once at the collection level, but it didn't seem to work. It appears the reporter is only looking within the scope of that request.
Can we modify the logic slightly to also look in "test" events in the parent (folder & collection) rather than just at that item level?
This would simplify it quite a bit, allowing us to specify the parameter at the entire collection level, and avoid issues arising if we forget to add that comment to one of our requests.
I've verified that the Newman Item findInParents('events') method (https://www.postmanlabs.com/postman-collection/Item.html#findInParents) would allow you to look recursively up the tree to folders and collections to find test events that would have parameters, and apply them to the test.
const matchingParentEvents = item.findInParents('events', (parent) => {
return parent?.events?.find(event => event?.listen === 'test' && eventHasParameterMatch(event));
});
The above snippet would return any events in the parent where eventHasParameterMatch is true (in which case you do a simple regex test)
Let me know your thoughts