Abstract
This study was aimed at developing an automatic exam system as an intelligent and high-quality higher math examination solution for Department of Computer Engineering at Pai Chai University (PCU) based on S/W Engineering. As a research and development (R&D) project, the study used the rational unified process (RUP) method for software development. RUP describes how to effectively use commercial and reliable methods to develop and deploy software system. It is a heavyweight process; therefore, it is particularly suitable for large teams to develop large projects. This paper describes the RUP process of this R&D project what we named it as Automatic Higher Mathematically Exam System (AHMES). AHMES provides a new way to automatically generate exam questions. In the study, our team and the requesting department collected the requirements and selected methods and tools. Then, our team designed and implemented the framework of the system, completed the development of some main functions, tested and summarized the system, and planned the future work.
1. Introduction
A system named “Automatic Higher Mathematic Examination System (AHMES)” was developed based on RUP process in this study. The linear algebra subject of AHMES has been equipped with 11 question-generating engines and 181 mathematical models. Random questions can be generated automatically by the system based on the mathematical model. AHMES could collect the participants’ answers and adjust the difficulty level of the mathematical model based on the correct rate of these answers. The linear algebra function produced by AHMES has been tested by teachers and students. We will improve the linear algebra subject in the subsequent phase and gradually develop functions for other mathematics subjects.
2. Background
The role of exam question-generating software in the current educational environment is becoming more and more important. It can save a lot of time and energy for educators, reduce the artificial uncertain factors in exam paper questions, and prevent cheating.
The previous examination system had a question database consisting of more than 100 fixed questions. When users use this system to generate an exam paper, the system will randomly select a certain number of questions from the question database to form the exam paper. In this mode, although mathematics teachers can add, delete, and modify questions in the question database at will, it will undoubtedly require a lot of efforts, and the exact same question may appear repeatedly in each exam paper generated. This system is undoubtedly unacceptable for the scientific discipline of mathematics. Therefore, a new examination system that is different from the old one, a smarter, scientific, and controllable software system, needs to be developed.
3. Methods
First, the requirements of the new exam system need to be collated before choosing the appropriate approach and tools.
3.1. Requirement Collation
The most difficult part of building a software system is to determine what to build [1]. The work clarifying requirements laid the foundation for construction of the system. Management of requirements will run through the entire development life cycle of RUP so that the analysis of requirements in each phase of RUP is very important.
In the preliminary communication with the department that proposed the requirements, the problems that need to be solved were collected and listed as follows:(1)The system needs to create exam papers based on the difficulty level and number of questions specified by teachers.(2)The system needs to turn mathematical problems into software models (mathematical models) and to create exam questions based on the mathematical models.(3)Each model has an attribute of the difficulty level.(4)In the initial phase, the system is designed to handle the linear algebra subject. Other higher mathematics subjects, such as calculus, may be added in the subsequent phases.(5)The software can adjust the difficulty level of the mathematical model according to the historical data of the exam results.
3.2. Approach and Tools
3.2.1. Software Engineering Approach
Since the requirements’ analysis suggested that new mathematical models could be added to the initial functions, and the system should be flexible and easy to scale up. The Rational Unified Process (RUP) is an iterative software development process framework created by the Rational Software Corporation, a division of IBM since 2003 [2, 3]. RUP was selected as the tool for the AHMES project because it is use-case driven and provides a process framework that can be adapted and extended to meet the growing demands.
3.2.2. Development Tools
Python 3.8 was chosen as the programming language for AHMES project because Python is a multiparadigm programming language and has filter, map, and reduce functions, list comprehensions, dictionaries, sets, and generator expressions [4].
In addition, due to Python’s extensive mathematics library and the third-party library NumPy that further extends the native capabilities, it is frequently used as a scientific scripting language to aid in problems such as numerical data processing and manipulation [5, 6].
For these reasons, choosing Python means that, in the AHMES project, we could focus more on design rather than programming. In addition, if a future version requires high performance, Python code can be easily converted to C code.
3.2.3. Text Formatting Tools
We have two solutions to export the exam paper. The first solution is to export as a file in Docx (Microsoft Word) format. The second solution is to export as a PDF file using LaTeX. The benefits of the first solution include that it is easy to implement and files in Docx format can be easily modified. The advantage of the second solution is it can output almost perfect mathematical formulas. In the initial phase, the first option is used temporarily. The second solution may be used in future iterations.
3.3. Workflow Design
RUP is a general process template that contains many development guides, products, and role descriptions involved in the development process. Because it is very large, it needs to be tailored when using RUP for specific development institutions and projects, that is, RUP must be configured. RUP is like a metaprocess. Many different development processes can be obtained by tailoring RUP. These software development processes can be regarded as specific examples of RUP. Reference [7] shows a simplified RUP model.
Similarly, AHMES as a R&D project does not require a complete RUP model. After selection and configuration, the RUP includes these workflows: Business Modeling, Requirements, Analysis and Design, Implementation, Test, and Project Management. The reason why Deployment, Configuration and Change MGMT, and Environment are discarded is that their usage scenarios are usually commercial and large-scale development projects and are not suitable for this project.
The modified RUP model used in this project is shown in Figure 1.
3.4. Mathematical Modeling
A mathematical model is a description of a system using mathematical concepts and language. The process of developing a mathematical model is termed mathematical modeling.
The one characteristic that differs AHMES from traditional examination systems is to generate questions based on mathematical models. The mathematical models are implemented with Python code based on the known topic types.
We use a simple example and a complex example to illustrate the mathematical models of AHMES.
3.4.1. A Simple Example
Figure 2 shows the most basic matrix addition model.
Matrix addition requires that the two matrices have the same shape, that is, have the same number of rows and columns. The number of rows and columns and the matrix values are generated based on the difficulty level input by the teacher. As the difficulty level increases, the amount of calculation will increase.
3.4.2. A Complex Example
The model uses the known matrices A and Y and other fixed conditions to calculate the nth power of the unknown matrix X. The relationship between A, Y, and X is XA = AY. The number of columns, rows, and values of matrices A and Y are determined according to the difficulty level provided by user. When the difficulty level is less than a certain value, n will be a known small integer. When the difficulty level is greater than a certain value, n will be an unknown number.
For students, the questions generated by this model are interesting and challenging. In a later section, we will list an example question generated by this model at the highest difficulty level.
3.5. Difficulty Control
Referring to Qiong Chen’s difficulty setting method in his exam system [8], we subdivided the AHMES difficulty degree into 5 levels instead of 4 levels used by Qiong Chen.
In AHMES, each mathematical model has an attribute of difficulty degree. The difficulty range of a question generating engine is determined by the mathematical models it contains.
Define D as the difficulty reference value of a certain mathematical model:where is the average score out of 100 for all the students on this model.
The five levels of difficulty are shown in Table 1.
Assuming a question-generating engine has 3 mathematical models and the difficulty levels of these mathematical models are 2, 2, and 4, then the difficulty range of this question generating engine is 2 to 4. AHMES will not call this engine if exam papers with difficulty level equal to 1 and 5 are needed.
4. Results
Currently, AHMES is at the beginning of construction phase. We had first implemented the AHMES framework and some mathematical models, successfully generated exam papers and tested them and then designed a solution for the current problems encountered by AHMES.
At each phase, our team will go through all workflows, but the main tasks and goals are different at each phase, so we focused different processes (see Figure 3, for details). Brief introductions were provided to the phases of AHMES in this section, and then, two instances are explained.
4.1. Inception
In beginning of inception phase, we constructed the main business model and sorted out the needs of the entire project. Then, we used Python to code a simple mathematical model (see 3.4.1) and generated a question. In this phase, we focused on the two workflows with Business Modeling and Requirements.
4.1.1. Business Modeling
In AHMES, teachers input parameters such as exam subject, number of exam questions, and difficulty level. The system calls the mathematical model to generate exam papers and the keys based on the parameters entered by the teacher. After importing the exam results, AHMES could adjust the difficulty of the mathematical model based on historical data of students’ scores.
In the RUP, use-case diagrams are often used to describe business modeling. Due to their simplistic nature, use-case diagram can be a good communication tool for stakeholders. The drawings in Figure 3 were made to mimic the real world and provide a view for the stakeholder to understand how the system is going to be designed.
4.1.2. Requirements
The requirements of AHMES could be described from three dimensions (Table 2): roles (who), works (what), and tasks (how).
4.2. Elaboration
The main task of elaboration phase was to initially build AHMES. In this phase, we designed the overall framework and constructed the main functions of AHMES. Then, we designed and constructed some mathematical models. Finally, we tested the complete AHMES and planned the next phase of tasks. In this phase, we focus on the four workflows of Analysis and Design, Implementation, Test, and Project Management.
4.2.1. Analysis and Design
The system structure is designed based on the results from analysis of requirement documents. The main flow controller of AHMES controls the operation of the software and calls various functional modules. The system includes four main functional modules: exam paper design module, question generating module, difficulty controller, and UI. A question generating engine contains one or more mathematical models with a certain commonality. Figure 4 shows the structure of AHMES.
The main functions of each module were listed in Table 3, and their interfaces were summarized in Table 4.
The UI module contains data input and output, exam paper file generating, and export.
Exam paper design module designs exam paper structure and question type of appropriate difficulty level according to the exam paper parameters input by teachers.
Question generating module contains various types of exam question-generating engines, and each exam question-generating engine contains several mathematical models. The exam question-generating module selects the appropriate exam question-generating engine according to the design by the exam paper design module. After the exam question-generating engine selects the appropriate mathematical model, the mathematical model generates exam questions based on the difficulty level and other parameters input by the teacher.
Difficulty controller includes two main functions: storage and adjustment of the difficulty level. The difficulty storage function is responsible for storing the difficulty range of each exam question-generating engine and mathematical model. The difficulty adjustment function can adjust the difficulty degree of the exam question-generating engines and the mathematical models according to historical data.
Figure 5 shows the flowchart.
4.2.2. Implementation
In this study, the framework and functions of AHMES were developed, and some mathematical models of linear algebra were implemented. The mathematical models of linear algebra subjects implemented in AHMES were listed in Table 5.
AHMES has been uploaded to GitHub, but it will not be open accessible until the completion of all work. The structure of code files for the project is illustrated in Figure 6.
4.2.3. Test
In the test, 100 exam papers with different difficulty levels were generated with AHMES and sent to teachers and students taking the exam. Teachers selected a part of these exam papers for students to finish. The feedbacks from the requesting department and students were collected and summarized in Table 6.
After receiving the answers from the students, we imported the data into AHMES and tested the difficulty adjustment function. Table 7 shows the adjustment results. Difficulty 1 is the easiest and difficulty 5 is the most difficult.
Negative points 1 and 2 among the feedbacks on AHMES were because no members of our team were experienced in teaching mathematics. Negative point 3 was because some algorithms of the exam paper design engines are not smart enough. The fourth negative point was because there is not enough historical data to adjust the difficulty levels.
4.2.4. Project Management
The AHMES project is currently at the beginning of the construction phase. The reason is that currently we have only achieved the followings:(1)Verify the feasibility of the mathematical models(2)Implement the framework and interface of the entire system(3)A version for internal testing is completed(4)Plan the improvement direction of the release version
In the current version, we encountered two problems that need to be resolved. These problems and solutions are discussed in Section 4.4.
4.3. Examples
We choose the simplest matrix addition as an example to illustrate. Section 3.4.1 described the mathematical model of matrix addition. Matrix addition adapts the difficulty levels within 2.
Figure 7 shows a question generated by the model of matrix addition with difficulty level of 1.
First, the number of rows and columns of the two matrices needs to be generated. The number of rows and columns are generated according to the degree of difficulty with the help of a random function.(1)row = random.randint (2, 2 + diff_coeff)(2)col = random.randint (2, 2 + diff_coeff)
Then, generate two matrices based on the number of rows and columns:(1)array_a = numpy.random.randint (−10diff_coeff,(2)10diff_coeff, (3)size = [row, col])(4)array_b = numpy.random.randint (−10diff_coeff,(5)10diff_coeff, (6)size = [row, col])
Finally, add the two matrices to get the answer.
Figure 8 shows a more complicated question generated by AHMES. Section 3.4.2 describes its model. It is a problem with a difficulty level of 5 (the difficulty level of 5 is currently the highest difficulty of AHMES) generated by a mathematical model in Unknown matrix 2. The code of this model is more complicated and will be provided after we decide to open the public permissions of AHMES on Github.
4.4. Existing Problems
In the current version, our team encountered two problems that need to be resolved:(1)The exam paper generating module in the current version is a completely random selection of mathematical models under the specified difficulty. The difficulty degrees of the questions generated by this method are fixed, and this cannot satisfy the needs in the actual situation. We hope that, in future versions, teachers using AHMES can control the proportions of questions with high and low difficulty degrees in an exam paper under the premise of controlling the overall difficulty degree of the exam paper (controlling the average score of the exam paper).(2)The difficulty levels of mathematical models in the current version were calculated based on the correct rate of answers to the questions generated by them in the historical data. Although this method is intuitive, teachers need to manually or semimanually record and input the average correct rate of each question in exam papers for offline exams, which is almost impractical.
To solve these two problems, we plan to use some AI algorithms to improve AHMES and release the relevant research results in time.
5. Discussion
After this study, AHMES was proved enable to generate exam questions through mathematical models in linear algebra subjects, but the number and types of existing mathematical models are still not enough. At the same time, other mathematical subjects such as calculus and discrete mathematics need to be added into AHMES. AHMES has high requirements for developers’ mathematical knowledge reserves and teaching experience in mathematical disciplines. To improve the rigor and practicality of AHMES, we suggest inviting math teachers to join our team.
In addition, the traditional and fixed methods currently used for designing exam papers and adjusting the difficulty degree of models are not perfect. In the next step, we plan to implement these functions using A.I. algorithms such as reinforcement learning and optimization algorithms.
6. Conclusion
AHMES is an automatic generating system of mathematics questions developed for university mathematics subjects. As of now, AHMES is the first software system to generate exam questions through mathematical models in linear algebra subjects. Through this study, we confirmed that, in linear algebra subject, the idea of generating exam questions through mathematical models is feasible. The variety, scalability, and practicality of the questions generated in this way are better than the fixed questions randomly picked from a database. AHMES can provide practical help for teachers in their teaching work and solve some objective problems encountered by some students in the exam.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.