Comparing Xray and Zephyr: Which JIRA test management tool is for you?

Zephyr has been the 'go-to’ plugin for users looking to run tests on information stored in Jira. Powerful, versatile and supremely capable, Zephyr is popular with Jira users. Xray is the new kid on the block and it’s making a lot of noise and seems to be catching on among users. Would one be better than the other in certain situations or are they similar with only marginal differences? Being new, how does Xray stand out? We decided to find out and narrowed our discussion to three main points.

Automated Testing
In larger enterprises with a mature approach to Agile development, automated testing becomes important. If you need to execute over 10,000 tests for your weekly releases, then Xray has native cucumber automation built in. This means the testing can take hours instead of days with no need for an add-on or a work around. Xray also has Pre-conditions, which allows for entry criteria to be added into testing (e.g. server is turned on). Some in our team really like the Pre-conditions, others not so much, it’s worth noting as an extra inbuilt feature that could be very useful.

Zephyr does not do automated testing from the Jira add-on. Having said that, out of the box it is a straightforward tool for manual testing and great for those just starting the journey of testing. Alternatively, Zephyr provides a robust software agent called a “ZBot” linking Zephyr to a target automation tool (additional add-on you must purchase), to empower the developers and testers to write their scripts and automate testing.

Reporting and Search

Zephyr can do traceability reports, and comes with its own project and dashboard reports. Zephyr also allows you to add comments and attachments to test executions at both the overall test level, and individual step level. Xray also offers a handful of logical reports including Latest Test Execution and Requirement Traceability. Both do an OK job in this department.

Drilling down further into details is where they differ. Xray uses 4 issue types - Test, Test Set, Test Execution and Pre-condition. Xray uses native JQL for searching for tests / test sets which give you extended search capabilities. You can search the Tests or Test Sets within a Test Execution - this enables you to get meaningful reporting from the test level to groups of tests.

Grouping large amounts of tests in a meaningful way in Zephyr within Jira is challenging. One suggestion to overcome this is to use eazyBI (add-on) with Zephyr. Zephyr has its own ZQL which is a bit light in terms of functionality within Jira. In Zephyr you can still search the tests themselves using JQL as a test is just another issue type. Where Zephyr is limited is that it cannot show you the latest execution of any given test regardless of which test cycle it belongs to, you can only see the latest execution for any given Cycle. This causes users to start cloning tests which mean you may end up with lots of redundant tests and nobody knows which is the latest.

Cost
Both add-ons are priced on a per-user basis and the pricing model is divided into tiers. For smaller organisations, there is little to choose between the two add-ons in terms of price. Up to ten users can work with Xray or Zephyr for $10. However, it is for larger organisations that the advantages of Xray pricing become apparent. Assuming you are starting from scratch and that the prices do not change as the product increases its popularity; currently, Xray works out cheaper than the Zephyr add-on when applied to medium and larger-sized organisations. Also, as mentioned above, Xray provides functionality as standard that Zephyr would require further expenditure to handle.

Xray VS Zephyr - pricing comparison*

Server Pricing as at 10 June 2016

^{*Prices are taken from the Xray and Zephyr website as of 10 June 2016 and may change at any time from this stated price.}