|
| Task | Missing functionalities | User number | Expert review |
|
| Task 1 | Criteria of >49 years were selected but appear as ≥49 years | User 01 | Not important; mistake in specification of query not tool |
| No ability to execute results for all countries, only for UK/no response when clicking on all countries | User 19 | Medium importance; probably a problem with available sites not with the tool |
| The query in eclectic format did not show up and looked similar to screen shot in training manual | User 06 | Low importance; feature only used for testing probably be removed for “real world” version of tool |
|
| Task 2 | Than & less selections appear transformed into more & less than OR EQUAL to | User 01 | Not important; mistake in specification of query not tool |
|
| Task 3 | The sequence of building the query is not clear; system seems to require it in reverse (i.e., the parameters of time to be entered before the diagnosis) | User 18, user 19 | Important; comment on usability though unspecific |
| Entering exclusion criteria (e.g., 3.3.4 EC02) is cumbersome | User 06 | Important; comment on usability though unspecific |
| No visible option how to add a range of 5–30 months; the range always began at 0 months | User 01 | Medium importance; option is there when user selects “between” rather than “more than” or “less than,” user interface issue, or poor documentation |
| No way to clear just one component from the query; “clear” clears all components/if you want to change particular part of the inclusion or exclusion criteria, you have to delete the whole; it would be better to delete parts | User 06, user 21 | Medium/low; true but the individual inclusion sections are never hugely complex so deleting all is not too bad |
| The “before now” button did not work several times | User 19 | Important; was not seen this reproduced though |
| The run function and eclectic format were not possible; computer crashed when running the query or doing it in eclectic format/“does not compute” message appeared, when trying to generate the eclectic format | User 21, user 01 | Important; a “crash” reproduced elsewhere was not seen |
| System feedback was that query had been saved, but it does not appear to have been | User 01 | Important; true in terms of lack of feedback, but it is always saved |
| If you want to check a specific value of a criterion (e.g., if left ventricular ejection fraction was correctly entered and you want to check it later) you are not able to see it by clicking on the symbols | User 21 | Important; this information appears at the bottom of the screen and not where user would originally see it and may need to scroll, user interface issue |
|
