h1. Introduction
What does this feature do?
Enable pattern match question authors to maximise the grading reliability
of the questions they create by determining how reliably a question grades 
answers compared to a set of human markers.

AMATI simplifies the process of grading free text answers by focusing on 
fewer more efficient rules. It achieves this by providing:
1) statistics on the accuracy of a question and its rules
2) tools to help develop better rules

Managing pmatch questions becomes easier because the author:
* has fewer rules to manage
* has more information to make informed decisions
** can see how accurate their question rules actually are.
** can see which rules are useful and which are redundant
* can try rules and to see the updated statistics before saving the rule 

Background
To automatically mark a free text sentence or multiple sentences the pattern
match question question was developed.

The Moodle pattern match question type is very powerful but equally difficult
to author. Requiring carefully creating a long list of complex rules to
generate a question that consistently marks student answers as well as human
markers. With Pattern match the rules are generated by assuming the answers
students may give.

AMATI is a separate OU developed approach providing a workflow which develops
grading rules based on human mark responses from to the given question.

This feature adds additional functionality to the Moodle quiz pattern match
authoring  workflow to incorporate the AMATI methodology and support
development of new grading rules using existing marked responses.

The research paper Using NLP to Support Scalable Assessment of Short Free Text
Responses shares the questions used to develop AMATI
http://aclweb.org/anthology/W/W15/W15-0628.pdf

Use cases
What situations will this feature help?
* Author has finished creating a question without rules and wants to upload
  sample responses to help create the rules.
* Author has finished creating a question with rules and want to upload
  sample responses to test/refine the rules.
* Author has previously created a question, rules and uploaded sample
  answers. The question has been used by students and the author wants to
  check/refine the rules based on additional sets of responses because the
  question didn’t perform as expected.

h1. Functionality
What can the feature actually do?

Test this question 
When previewing a pattern match question you will now see a "Test this 
question" link in the top right corner. Clicking this link leads to a screen
listing the sample responses attached to the question. Each response includes:
* the response
* the expected score
* the actual score

This highlights any mismatches between the score given by experienced human
markers and the score given by the question.

Test selected questions
When responses have been uploaded to a question select the checkbox next to
each response you wish to grade or click "select all".

Click "Test selected responses" and each selected response will be marked by
the question with the grade being recorded in the "Actual mark" column.

There are 4 categories that response grading statistics fall into:
1) Matched response: the computer gives a mark a human would give
2) Missed positive: The human marked it correct, the computer marked it incorrect
3) Missed neagtive: The human marked it incorrect, the computer marked it correct
4) Ungraded: The computed mark is not available because the response has not been tested.

Upload responses
Responses can be added to the question in the upload response screen.
To import responses create a simple CSV file with:
* a header row containing mark,response
* a series of rows with the human graded mark (expected mark) and response
separated by a comma
** e.g. 0,jane is friends with harry

Uploaded responses are associated with the specific question they were uploaded
against and stored in the database for use later.

Delete responses
Responses can be deleted in the "Test this question" screen. select the
checkbox next to each response you wish to delete or click "select all".

Click "Delete" and a confirmation box will appear asking you  to
confirm the delete request. Click yes to delete the selected responses.

h1. FAQ
Are duplicates stored?
To prevent duplicates only unique responses for the given question will be
saved. Before an uploaded response is saved a check is made to ensure the
same response is not already stored for the question. The upload feedback
message reports which duplicate responses were not saved. 

Can an author remove test responses?
Initially we have provided a standard interface allowing full editing (create,
read, update, delete functionality). Later on we may restrict this
functionality through capabilities within Moodle according to requirements.

Does this feature change or remove users attempts?
No, The test responses used are not part of the quiz or question data. Student
attempts are not affected.

The test responses are separate data uploaded to a separate table added specifically
for this feature. They do not represent student attempts. These responses represent
test data on which to verify the accurcay and reliability of a given question.

There it is not possible to edit student attempts or grades held in the quiz
results or the gradebook.

h1. Administration

Capabilities
Create and read capability are clearly required by anyone using the new feature.
Consensus has not been reached on which users would delete or update capability
or whether this should be possible at all. The considered use case is where
data is incorrect or no longer required. In this case the capability to update
or remove data may be an important consideration.

moodle/question View : View Question
moodle/question Edit : Edit Question
mod/quiz:regrade Regrade : Regrade Quiz

Supported Browsers and Operating systems
Use is expected to be on desktop browsers and potentially tablets.

No significant user interface challenges are expected and standard moodle form
elements should be sufficient to deliver the required functionality. Thus in
theory all browsers that Moodle currently supports using the OU theme should
be supported.

Automated Testing
Automated testing is crucial to validating the reliability of delivered
functionality on a regular basis. In depth unit and Behat tests have been 
created.
