What’s up Guys! Welcome to automationcalling.com

Introduction

In this article, we are going to take a look at Xray Setup, Configuration for existing and New Project (Xray project), how to create a test plan, test set, test, and test execution, generate Xray Report and Custom Report, how to import automation test and test execution results, and finally automation  best practices etc.,

What is Xray?

Xray is a test management plugin which runs on top of JIRA. Xray is the fastest growing App for Jira and a Success Story in the Atlassian Marketplace.

• Enterprise Test Management in Jira
  • More than 4000 customers in 70 countries
  • Uses Jira issues, custom fields, screens, workflows
  • Traceability between requirements, tests, defects, executions
• Testing
  • Define reusable preconditions and associate to tests
  • Organize tests in test sets
  • Test plans for tracking progress
  • Test environments
• Automated tests
  • Write Cucumber scenarios in Jira
  • Integrates with test automation frameworks, including Selenium, JUnit, Nunit
• Reporting
  • Built-in reports
  • Possible to Generate JIRA Custom Report
• Integration
  • REST API
  • CI (Bamboo, Jenkins)
  • Import tests from CSV and JSON
• Focus on requirements coverage
  • Charts and reports (For eg., Traceability etc.,)

Preconditions and Configuration Setup

There are multiple ways we can enable Xray project in Jira, which are:

• Enable Xray in Existing Requirement Project
• Create New Xray Project and map to Requirement Project

Xray Proposed multiple ways to integrate Xray Project with Requirement projects. We are going to take a look at two approaches which are:

Keep All in One Project (Requirement and Xray):

Note: A disadvantage of this approach is: you can see all test related tickets in backlog which most of the scrum masters or product team or product owner may not be happy or interested in that.

Keep Requirement and Xray in Separate Project (Don’t mix with Requirement and Xray)

Note: For this approach, nobody sees test related tickets in requirement backlog. We can manage and maintain Test project is dedicated for QA team.

Before setting up Xray, you must come up with an approach of how Xray to be integrated with the requirement.

Here is the basic workflow of how Xray artifacts associates and map to requirement.

Finally, we expect to look at the following format while hitting create button in JIRA

Enable Xray in Existing Requirement Project

For existing projects, in the project Administration page (remember you must have administrator access), go to Actions at the top right-hand side, enabled below-highlighted items.
Note: This approach will be applicable for Keep All in one project.

Create New Xray Project and map to Requirement Project

By default, Xray offer “Xray Test Project” which you can manage as Test Project.

1. Log in as an Administrator
2. Click on Projects
3. Click on “Xray Test Project”
4. 
5. Make sure all Xray related artifacts available in Default Scheme, usually it’s.
6. 
7. Enter Project Details and click Submit button.
8. 

Create New Xray Project and map to Requirement Project

In order to calculate the overall requirement coverage by versions, you must guarantee that versions from the Requirement project and Test project are the same meaning The version available in your requirement project need to be copied to your Test Project so that all requirement coverage, the calculation and other types of report works fine.

Go to  Administration-> Projects  -> newly created Xray project -> Actions -> Copy Project versions

After copied, for eg., the version display in Test Project as below snapshot.

Xray Recommend JIRA Workflow

Xray recommended default workflow for the following issues types:

• Pre-condition
• Test
• Test Set
• Test Plan
• Test Execution

Please refer this link and make sure to set up JIRA workflow and apply in your existing workflow scheme

Xray Test Management Activity

How to Create/Maintain Repository

Xray 3.0 introduces the Test Repository concepts which helps and enables the hierarchical organization of tests at the project level by allowing you to organize tests in the folder.

To set up this in part of your project, please follow the below steps:

1. Log in to JIRA Dashboard and Go to your specific project
2. At the left pane, Click
3. Create Folder/SubFolder like below snapshot to organize your test
4. 
5. To organize test cases in a better way Test Repository is a solution, some companies maintain “Test Set” or “Test Plan” instead of a test repository. It’s all liberty of team requirements and expectation, but we can still maintain all (test plan, test set, and test repository), meaning organize the test in folder wise in the test repository, from there you still can generate test plan, test set and test execution like below snapshot.
6. 

Note: By Default, already added test cases or not specified path in your Test while creating will be listed under Orphans. If you add/move your test in a specific folder, the path will display like this

How to Create a Test Set

A Test Set is a collection of Tests. It often contains detailed instructions or goals as well as information on the configuration to be used during testing.

The main purpose of the test set we create is, categorize your test cases in a logical way or categorize your test cases in types of testing for eg., regression, integration, unit, current feature testing. You may have more than one test set to associate with the test plan, this will help in a future release testing, you can reuse the test set.

To Create Test Set in JIRA:

1. Log into JIRA board
2. Click on Create button
3. Add required test to your Test Set as below snapshot
4. 
5. Click on Add Selected button
6. 

How to Create a Test

In XRay, The test is categorized as two options, We need to select which option is required while creating a test.

• Manual
• Automation (Cucumber and Generic)

How to Create Manual Test from Story

1. Log into JIRA and choose any story
2. Create from Story
3. Go to Test Details tab and Select Test Type
4. Enter Step and its expected results
5. Click on ‘Create’ button
6. Test Cases created and display on top of the story. Requirement status says Not Run as show in the below snapshot.
7. 

How to Create Automation Test from Story

There are multiple possible ways to create automation test, in this section, we are going to take a look on how to create Automation Test from Story.

1. Log into JIRA and choose any story
2. Create  from the story
3. Go to Test Details tab and Select Type
4. The appearance of the screen gets changed appropriately like the below snapshot.
5. 
6. Select either Scenario or Scenario outline and write Gherkin syntax, finally save it
7. Gherkin syntax gets highlighted with blue font color as mentioned in the below snapshot.
8. 

How to Create a Pre-Condition from Story

A precondition specifies the condition that needs to be fulfilled in order to execute a test. Precondition can be reusable to many tests in case it fulfill the exact requirements, for eg., you have to configure certain settings in DB before run your test (or) Disable CSS in your browser before execute your test etc.,

There are different ways to create Pre-condition:

• Use Create Button at top of the screen in JIRA and later associate with Test
• Open Existing Test, Click on

How to create “Pre-Condition” in Existing Story

1. Open Existing Test Ticket
2. Click on
3. Select “Pre-Condition Type” and add conditions and associate Test ID in Test Tab (You can associate multiple Test ids)
4. 

Note: The best practice is, you can create generic pre-conditions that can be reused for multiple tests. The precondition can be closed after all tests were done, there is no validation check between test gets closed and precondition gets closed.

How to Create a Test Execution

Xray provides the Test Execution issue, which acts as an assignable task for executing a group of tests (either manual or automated) and collects the respective test results for a specific project version and environment.

A Sub-Test Execution is similar to a Test Execution, but created as a sub-task of a requirement.

There are number of ways you can create Test Execution:

• Create Test Execution from Story
• Using Create button on top, create Test Execution and Associate Test and Test Plan
• In Test Plan, you can create test execution.

The best practice is, create Test Executions from Test Plans where we already associate Test Set which may be listed necessary tests.

We can create test execution either all test cases or specific test cases in status wise, this will help us to track what is all done and not done in Test Plan.

I select “With Status” to create Test Execution, in this case, I have 4 ToDo Cases need to be execution part of Version 1.0 Test Plan. Click on the ‘Create’ button.

While creating test execution, please make sure to add details for below-highlighted fields. This will ensure to track our test execution progress in terms of Manual test.

Note: To verify test with different environment/configuration for eg., Android, iOS or Browser like Chrome, Firefox or Safari, need to create separate test execution.

Test Plan Board Usage

1. Go to the Existing Test Plan.
2. Click on button
3. It takes you to Test Plan Board
4. By default, it list number of tests that were associated in the test plan with one folder
5. We can customize the test by creating more sub-folder and categorize tests as below snapshot
6. 
7. This will help us to track test execution status in folder wise as well as overall execution.There are possibilities to create a test, test set and test execution from Test Board as below snapshot.
8. 

Enable Requirement Coverage for Both (Test and Requirement Projects)

In case you manage Test and Requirement as a separate project, please follow the below steps:

1. Log in as Administrator.
2. Click on Project Settings.
3. Click on Actions Menu at the right pane.
4. Click on “Enable Xray Requirement Coverage”
5. The option gets an enabled and prompt message to reindex

Note: If any changes in Project Settings and prompt you to re-index then you should perform re-index and acknowledge the same.

Test Coverage Screen View

Once followed all sections described above, enabled Requirement Status in your Sprint, Associate all Tests and test execution is done. The screens look like below snapshots.

Sprint Board View

Backlog View:

Note: In case, you don’t see Requirement status, there are two options:

1. Please visit the link and make sure you set up correctly on requirement coverage
2. In case status not seen in Agile Board or Backlog, please do the following setup
   • Log into JIRA as Administrator
   • Click on Configure under Board
   • Click on Card Layout and select “Requirement Status” in Active Sprints
   • Click the “Add” button
   • Go back to your Active Sprint board, you would see the requirement status.
   • 

Story View

Xray Integration with JIRA using REST API

Current Challenges and Issues

This topic is applicable for the team who adopt cucumber framework. At this moment, there is some redundant work by QA tester in the following ways:

• Not able to use Same Gherkin syntax format in other test management tool because it doesn’t support Feature file implementation
• Possibilities to write duplicate effort meaning generic test cases during manual testing and gherkin syntax during automation test, not the same format of test cases in all phases.
• Product Team is not able to see how we write a feature file in an earlier stage (Before Development)

To Address all this issue, Xray offers an awesome feature to import your feature file from automation project. I’m going to explain step by step in the following section.

Xray Proposed Workflow to Collaborate between Automation Project and JIRA

There are multiple workflow Xray proposed to us which can be referred here Xray_Proposed_Workflow but I feel the below workflow is beneficial to integrate between Automation project and JIRA.

• In this approach, there are possibilities to edit the cucumber test in VCS (bitbucket of GitHub) as well as JIRA
• The approach must be starting from Tool (Intellij or Eclipse) → JIRA.
• In case of an edit, we use VCS and re-import using REST API. Xray will not create duplicate test rather it check scenario if it’s duplicate then will get updated.
• Create Cucumber tests (e.g. Scenario/Scenario Outlines) and their respective step code in some other tool.
  1. add a unique tag to each Scenario/Scenario outline, in the form of “@id:xxx”, in which xxx is a number (e.g. 1, 2, 3, …). so this is the way Xray identified unique scenarios and create tests accordingly in JIRA.
• Implement Code (Step definition) for feature file and modify steps if necessary. Re-import to JIRA to make sure both your automation project and JIRA are in sync.
• Execute the test in CI
• Report result back to Xray using REST API

How to Import Cucumber Test to JIRA

Rest API Endpoints:

Cucumber “.feature” file	/rest/raven/1.0/import/feature

There are multiple ways to import your feature file using Rest API

• ProjectKey
• ProjectId
• Source format

I’m going to show how to import feature file using Project Key using Rest API in Rest Client

No	Description	Examples
1	Request Type	GET, PUT, POST
2	Host	https://sandbox.xpand-it.com
3	Resource Path	/rest/raven/1.0/import/feature
4	Query Param	?projectKey=POC
5	Authorization	Your User Credential to be generated in basic authorization format
6	Content-Type	multipart/form-data
7	Body	Need to upload File as you content-type is multipart/form-data
8	file	upload appropriate feature file or .zip file in Rest Client

Finally, Click on the Send button. Scenarios imported successfully in JIRA

Note:

• In case, if you want to upload multiple feature files that can be zipped and upload a .zip file instead of a feature file.

multipart/form-data: “file” : a MultipartFormParam containing a “.feature” file or a ZIP file to import

• This can also be automated kind of batch script runs on a daily basis. The reason is, often we change our gherkin syntax in VCS (Bitbucket or GitHub), there may be situations not in sync between Automation VCS and JIRA. When running this batch it will sync on a daily basis no manual intervention is required.

Similarly for Generic Automation, please refer to this page: https://confluence.xpand-it.com/display/public/XRAY/Using+Generic+Tests+for+Automation

Import Automation Execution Results using REST API on JIRA

This section is mainly focused on how to import automation test results using the REST API.

Xray Support different framework output such as JSON or XML which can be seen in the below tabular column.

Xray JSON format	/rest/raven/1.0/import/execution
Xray JSON format multipart	/rest/raven/1.0/import/execution/multipart
Cucumber JSON output format	/rest/raven/1.0/import/execution/cucumber
Cucumber JSON output format multipart	/rest/raven/1.0/import/execution/cucumber/multipart
Behave JSON output format	/rest/raven/1.0/import/execution/behave
Behave JSON output format multipart	/rest/raven/1.0/import/execution/behave/multipart
JUnit XML output format	/rest/raven/1.0/import/execution/junit
JUnit XML output format multipart	/rest/raven/1.0/import/execution/junit/multipart
TestNG XML output format	/rest/raven/1.0/import/execution/testng
TestNG XML output format multipart	/rest/raven/1.0/import/execution/testng/multipart
NUnit XML output format	/rest/raven/1.0/import/execution/nunit
NUnit XML output format multipart	/rest/raven/1.0/import/execution/nunit/multipart
Robot Framework XML output format	/rest/raven/1.0/import/execution/robot
Robot Framework XML output format multipart	/rest/raven/1.0/import/execution/robot/multipart
Compressed .zip file (e.g., Calabash execution results)	/rest/raven/1.0/import/execution/bundle

How to Import Cucumber Execution Results in Test

To import Cucumber execution results, Xray does support Cucumber JSON format. In order to generate cucumber JSON file after test execution, you must have test runner configuration like below.

@RunWith(Cucumber.class)
@CucumberOptions(
        features = {"src/test/resources/Factorial_SimpleTest.feature"},
        glue = {"bddStepDefinition"},
        plugin = {"json:target/cucumber-report.json"},
        monochrome = true
)

Similarly for other languages, setup test runner configuration accordingly.

1.  Use Attached basic factorial feature file
2. Implement step definition and factorial code.
3. Import the feature like below format to JIRA by following above link “How to Import Cucumber Test to JIRA“
4. Execute the test in IDE, which generates json file like this
5. 
6. It’s time to import Test Results to Specific test. Note: the JSON file may have more than 1 tests but Xray detect the story based on tag provided in feature file and create test execution and mark as pass.

Rest API Request:

Request: Request Type Post Resource Path rest/raven/1.0/import/execution/cucumber Header Basic Authorization, Content-Type: application/json Body import_cucumber_results.json Response: { “testExecIssue“: { “id“: “19719“ “key“: “POC-23“ “self“: “https://sandbox.xpand-it.com/rest/api/2/issue/19719“ } “message“: “Could not make transition from workflow status <b>To Do</b> to workflow status <b>Resolved</b>. “ }

After invocation, Test Execution Created successfully and marked automation results in JIRA as seen in the below screenshot.

The same can be tried out with the different framework, To play with more please visit official Xray Documentation: https://confluence.xpand-it.com/display/public/XRAY/Import+Execution+Results+-+REST

Automation Best Practice Collaboration with Xray

To manage Test automation and collaboration with JIRA. Xray Suggested workflow for Cucumber and Generic Automation which are:

Cucumber: https://confluence.xpand-it.com/display/public/XRAY/Testing+with+Cucumber

Generic: https://confluence.xpand-it.com/display/public/XRAY/Using+Generic+Tests+for+Automation

How Can We Track Automation Activities in Xray

In Xray basically, we have to create test execution whenever we wanted to run a set of tests. There are multiple ways we can plan and manage Automation Activities in Xray, which are:

• Create dedicated Test Execution and Associate All tests for Automation Project.
• Run Automation Test and Publish test results which will create new Test Execution and associate all your test results by automatically or you can use existing test execution and update test run status alone.

Create Dedicated Test Execution and Associate All Test

This process is quite helpful in term of updating automation results on a daily basis in Xray test artifacts, meaning In CI/CD world, we have the practice to run automation results in daily basis or often based new build deployment on QA environment. This will ensure no issue or identify bugs at the earlier stage than the later.

Currently, all team of automation projects have their own or custom automation test report to track failures, some of them even open a defect in JIRA through REST API in case any failures in the test script.

So, the question is how we can integrate our daily automation results in Xray/JIRA? There are possibilities to maintain and integrate test results in Xray irrespective of Sprint related Activities or Regression Cases.

Advantage:

The great advantage of this approach is, we use a dedicated automation test plan and test set and update the only status in Test Execution. You can keep this Test always Resolved or open but we can pull automation status using the Custom report for eg., Pie chart, Bar chart, etc., Using this approach, even we can possible to ignore the custom report in your automaton project rather update the status in Xray and integrate results in confluence page using JQL Query. The page gets reflected in daily basis based on automation results.

To Play with how to update specific test execution using REST API, please visit the following pages:

• https://confluence.xpand-it.com/display/public/XRAY/Test+Runs+-+REST
• https://confluence.xpand-it.com/display/public/XRAY/Test+Examples+-+REST
• https://confluence.xpand-it.com/display/public/XRAY/Test+Step+Results+-+REST

Run Automation Test and Publish test results in Xray

This is another excellent approach to update your test results in Xray. In the previous section, we use only to update the results using RESt API but this one, we can use the following ways:

For Cucumber Project: Use JSON file to import all your test results using REST API in Xray. This will create an additional test execution and associate all tests (Regression or Sprint).

For TestNG or JUnit Project: After Test Suites or Post Automation test execution use Rest API to import test results in Xray.

To play with its Rest API

https://confluence.xpand-it.com/display/public/XRAY/Import+Execution+Results+-+REST

Also, recommend going through step by step in the above section: “How to Import Cucumber Execution Results in Test”

Note: The import can be done either REST API or Jenkins Xray Plugin for Import Test and Test Execution Results

Xray Reports

Test reports play a vital role in terms of day to day testing activities regarding how we plan, execute and achieve related to sprint, releases or milestone. In this section, we are going to take a look at a very high-level look regarding how we can leverage the Xray Report (In-build) and Custom Report (based on JQL query).

The detail report is already available in Xray Documentation:

• https://confluence.xpand-it.com/display/public/XRAY/Examples+of+Reports+Built+with+Xporter
• https://confluence.xpand-it.com/display/public/XRAY/Reporting+using+Gadgets
• https://confluence.xpand-it.com/display/public/XRAY/Built-in+Reports

Xray Report

In JIRA, at the left pane, there is a tab called Xray Report. At the right pane, when you click on “Down Arrow”, the following report shows up.

This will help you to track report on daily basis and project to Management regarding how we did perform part of planning.

Overall Requirement Coverage

https://confluence.xpand-it.com/display/public/XRAY/Overall+Requirement+Coverage+Report

Historical Requirement Coverage

https://confluence.xpand-it.com/display/public/XRAY/Historical+Requirement+Coverage+Report

Traceability Report

https://confluence.xpand-it.com/display/public/XRAY/Traceability+Report

Test Plan Report

https://confluence.xpand-it.com/display/public/XRAY/Test+Plans+Report

Test Execution Report

https://confluence.xpand-it.com/display/public/XRAY/Test+Executions+Report

Custom Report

For Custom report, you can generate with your own customization using existing JIRA report feature. For eg., Pie Chart, Bar chart based on JQL Query.

This custom report you can either create a dashboard in JIRA or use Confluence inbuild JIRA plugin.

for eg., Component level breakup

Manual and Automation break-up:

Another customized Report

Xray support integration with third-party tools like confluence, Xporter, eazyBI etc., For leveraging this usage, you need to import Xray Plugin in Confluence or enable Xporter etc., Also, Xray support REST API capabilities to get required data in JSON format.

Conclusion

From the perspective of Manual, Automation, Shift Left Approach (Development, QA and DevOps) collaboration and compared with other plugin or testing tools, Xray is awesome tool which provides built-in REST API support, more flexible, Customizable report, Multiple Framework support (Including Cucumber Native), great efforts for collaboration and available with low Price. Taking all into consideration, I strongly recommend Xray should be best-fit for Integration with JIRA  and it’s best Test Management tool for Agile project.