Abstract
Measuring and evaluating a learner’s learning ability is always the focus of every person whose aim is to develop strategies and plans for their learners to improve the learning process. For example, classroom assessments, self-assessment using computer systems such as Intelligent Tutoring Systems (ITS), and other approaches are available. Assessment of metacognition is one of these techniques. Having the ability to evaluate and monitor one’s learning is known as metacognition. An individual can then propose adjustments to their learning process based on this assessment. By monitoring, improving, and planning their activities, learners who can manage their cognitive skills are better able to manage their knowledge about a particular subject. It is common knowledge that students’ metacognitive and self-assessment skills and abilities have been extensively studied, but no research has been carried out on the mistakes that novice developers make because they do not use their self-assessment abilities enough. This study aims to assess the metacognitive skills and abilities of novice software developers working in the industry and to describe the consequences of awareness of metacognition on their performance. In the proposed study, we experimented with novice software developers and collected data using Devskiller and a self-assessment log to analyze their use of self-regulation skills. The proposed study showed that when developers are asked to reflect upon their work, they become more informed about their habitual mistakes, and using a self-assessment log helps them highlight their repetitive mistakes and experiences which allows them to improve their performance on future tasks.
1. Introduction
The development of software is a logically complex task. The main four components of software development and its value ability are (i) the process applied, (ii) the tool used in the development, (iii) stakeholders, and (iv) timeline and budget [1]. Producing quality software is the key component to the success of any software product. The implementation of the best development process, quality characteristics, and sophisticated technology have immense effects on the quality of the software [2].
Software quality can be affected by two main components: (a) software process quality and (b) software product quality. Product quality is dependent on process quality [2]. The process quality has received more attention from the research community [3]. Whereas the implementation phase of the development process has received less attention, particularly from a behavioral perspective [4, 5]. The implementation phase consists of coding, testing, installing, and supporting the system [6]. As mature tools are available for coding, testing, installing, and supporting (troubleshooting) the system, along with these tools, the programmer’s ability and skills have additional importance and effect on the software quality.
The programmer’s ability and skills can be commonly viewed as the experience he/she has, different certifications, products developed (portfolio), and communication skills of the individual person [7]. These are the common methods that are generally applied to check the talent and capability of a programmer and have positive measures [4]. However, these factors do not show the individual thinking ability or self-judgment of the programmer during development. To achieve this goal, it is important to measure individual metacognitive skills such as self-exploration, self-assessment, understanding from example (example-based learning) [8], and most importantly learning from their mistakes. These measures can have diverse effects on individual performance and therefore require an in-depth analysis to highlight their importance and influence on young software programmers.
Since metacognitive skills are used subconsciously, many people are either unaware of this phenomenon or unable to manage them properly. Research shows that people who use their self-assessment skills are more progressive than those who do not use their self-assessment skills. Normally, people ignore these types of learning skills in many learning environments. A programming job is a very mentally challenging job, where a programmer’s intellectual abilities, knowledge, and cognitive and metacognitive skills are used aggressively. Different types of studies have been carried out to assess the metacognition skills of different learners in different academic domains. However, minimal work has focused on the industry and no such work has been proposed for young professionals working in the software industry.
Our research objective is to analyze the self-assessment skills of novice developers, which affect their development work and to evaluate their performance on awareness of using self-assessment. The research will focus on(1)Analysis of metacognitive abilities and their usage by novice software developers.(2)Correlation of different types of novice developer groups and their performance on the basis of their improvement/diminishment.
To resolve these issues, the following research question is to be addressed:(i)What role can self-assessment play in the lives of novice software developers in minimizing repetitive mistakes?
To answer this main question, we will ask further (sub) research questions as follows:(i)What are self-assessment skills?(ii)How self-assessments skills help people perform better?(iii)How can self-assessment skills help novice software developers in the field of programming?(iv)What is the effect of self-assessment on novice developers’ work?(v)What is the difference between developers who use self-assessment and those who do not?(vi)What kinds of mistakes a novice developer makes without analyzing their work?
2. Related Work
As like metadata can be defined as data about data, similarly metacognition can be defined as cognition about cognition [2]. Metacognitive learning involves specific knowledge of knowing that what are the different aspects of individual actions and what are the relations between these aspects that affect the learning of that individual [9]. Research on metacognition found that by applying different methods and awareness of metacognition, the learner’s learning ability can improve significantly. Özsoy and Ataman [10] argue that metacognition practices are very important for successful learning strategies for teacher training, students’ learning, and content development [11]. Özsoy and Ataman [10] conducted a study on students with mathematical disabilities and revealed the importance of metacognition in adulthood. He concluded that some students overestimated and some underestimated their mathematics results. The study on metacognition presents two different terms of mistake, which affect the learning process, made by the learner during learning: (1) Underuse: which means not asking for help when the subjects need help and (ii) Overuse: the learner asks for help repeatedly while not using his own mind to think about some problem [12].
2.1. The Social Cognitive Theory (SCT)
In the paper [13], the authors relate the different factors of person metacognition as personal factors, behaviour, and environmental influence. The author [13] relates these factors and concludes that there is a relation between these components which affect the metacognition of students. These factors described in Figure 1 describe the social cognitive theory (SCT). The SCT describe that:(i)The interaction between the person and their behaviour is influenced by their thoughts and actions [14].(ii)The interaction between the person and environment involves beliefs and cognitive competencies developed and modified by social influences.
The interaction between the environment and their behaviours involves the person’s behaviour determining their environment, which in turn affects their behaviour [15].
This SCT has got reputation because it explains individual acts specifically point (i) above. More precisely, the SCT argue that the person’s cognition like self-assessment can guide the behaviour without the actual skills an individual has [4]. For example, the authors have defined a social cognitive conceptual framework using SCT, to simplify theory integration in the field of presenteeism research [16].
Metacognitive knowledge has been described as the knowledge and a deeper understanding of the cognitive process and product [6]. For example, a programmer knows that they must test the function that has a specific task but does not need to test a simply declared variable. Metacognitive knowledge has three important components. Declarative metacognitive knowledge was found to be “what is known in a propositional manner” [17] or the assertion about the world and the knowledge of the influencing factors of human thinking. Procedural metacognitive knowledge can be described as the awareness of processes of thinking [17] or the knowledge of the methods for achieving goals and the knowledge of how skills work and how they are to be applied. Conditional or strategic metacognitive knowledge is “the awareness of the conditions that influence learning such as why strategies are effective, when they should be applied, and when they are appropriate” [17].
2.2. Metacognitive Monitoring
Metacognitive monitoring means the ability to successfully judge one’s own cognitive process [7]. According to the author in [9], “metacognitive monitoring is the conscious monitoring, control, and active regulation of the self-cognitive activities in the whole process of cognitive activity.” The author argues that metacognitive monitoring is self-monitoring for which the individual must have to require some knowledge and experience of metacognition. The real taste of metacognition can be felt in metacognitive monitoring. Metacognitive monitoring affects the regulation of study, which in turn affects overall learning [18].
2.3. Metacognitive Control
Metacognitive control can be defined as the ability to regulate cognitive activity. Metacognitive control is used to control the memory process and to use strategy to improve comprehension. By using a learning strategy, the learner gains knowledge in the area of concentration. The learner will learn to ask themselves whether they are looking for directly stated, implied, or need to connect information from their own backgrounds with the material [17]. Recently, the author in [19] found that due to cognitive control training, the key neural region has been improved in terms of active and proactive activities. To understand the conceptual mapping of different learning strategies, we have represented these different learning strategies in Figure 2.
2.4. Self-Assessment
Self-assessment is defined by the author in [20] as “Self-assessment is a process of formative assessment during which students reflect on and evaluate the quality of their work and their learning, judge the degree to which they reflect explicitly stated goals or criteria, identify strengths and weaknesses in their work, and revise accordingly” (2007, p.160). Investigating the present abilities of oneself is the basic step to learning. Self-assessment builds a way to check out the progress of oneself, after that, one can understand the further improvement in the learning process. It can motivate the learner about his/her responsibility and work. It makes the learner possible to focus on improvement by awarding him/her about their cognition abilities.
Self-assessment abilities commonly can repair what students do not see as superior to anything educators can, in light of the fact that instructors for the most part cannot analyze as absolutely the student comprehension problem [21]. Mostly, learners do not assess themselves until they are guided or made aware to do so [22]. Hence, it is important to assess those individuals so that they can use their self-assessment abilities to solve a problem in any situation.
The authors in [23] argue that “the way in which self-assessment is implemented is critical to its acceptance by students”. Different techniques have been used to assess the self-assessment of different types of learners. For example, the authors in [21] use ACE in ITS to assess the student self-exploration on rum time, Behlau et al. [24] used questionnaires to find the impact of voice problems on individual life, Binali et al. [4] use PSE to assess the student of programming, Boud [23] in their study used State University data to measure self-assessment ads Questionnaire Tutors and to assess the self-assessment of learners, also the Self-Assessment Manikin [3, 25] used ELM-PE to measure the self-assessment of a student in the ITS system [8], professional judgment and expertise oriented framework have been developed by the authors in [26] by postulating on the aspirant/developing coach’s capacity for and development of metacognition within the reflective process, and a lot more.
2.5. Aspects of Self-Assessment
To promote the metacognition aspects of novice developers working in the software industry, we focused on the three well-known aspects of self-assessment skills [10].
2.5.1. Planning
Learners plan better and learn when their attention focuses on learning objectives [27]. A clear discussion of the learning goals starts the metacognitive process by prioritizing the significance of thinking about the learning process over the content. The authors [28] in their study stated that goal setting and strategic planning positively anticipate objective fulfilment in Massive Open Online Course [28]. It is important for every learner to review what is already known about the topic or task on which he/she is working [29]. Also, it is important for the learner to assess the time it will take to complete this task and also to have a clear idea of the resources needed for the successful completion of that task to help him/her think about the process of learning.
While solving a problem, we can divide the type of learner into different categories:(i)Learner with initial planning:(ii)Learner with continuous planning or partial planning.(iii)Learner with no planning
2.5.2. Monitoring
Every learner benefits from monitoring their understanding during learning activities [2]. Grainger et al. [30] also originated that impaired metacognitive monitoring in developmental disorders in children has important educational implications. The importance of the learning process can be reminded by monitoring learning behaviors throughout the learning process. This can be done by chunking, connecting, elaborating, and organizing the materials in such a way that the learner can recognize patterns and associations [31]. In the monitoring process during the work/problem solving, it is easy to figure out the errors and correct them in the thinking process.
2.5.3. Evaluating
By evaluating self-assessment skills, the learner becomes more aware of the self-assessment process and its direct impact on learning [32]. By evaluating students, the authors describe and allocate value or merit to the quality of their knowledge outcomes [33]. Different types of evaluation methods such as a checklist, rating scales, and questionnaires can help the learners evaluate their thinking during problem-solving. The study conducted by Sierra and Frodden [34] concluded that those learners who evaluate their learning process can decide whether they want to continue or need to change the learning process.
Students with self-evaluation judgment ability have high-performance feedback from those who do not evaluate their learning process [35].
2.6. Implicit Thinking
Implicit thinking refers to unconscious effects such as knowledge, perception, or memory that impact a person’s behaviour even while the individual is unaware of those influences [36]. There are times when we act on something and then consider how we might handle it in a different setting or approach. That is implicit cognition at work; the mind will then proceed based on ethical and comparable scenarios when engaging with a certain notion. Implicit cognition and its automatic cognitive process allow a person to make a decision on the occasion. It is frequently characterized as an automatic process in which actions are easily performed without conscious awareness.
3. Analysis of Self-Assessment Skills
This study followed a qualitative approach where we used observation-based research to observe five novice developers at coding tasks. These developers were also required to fill in self-assessment logs daily for five days.
The design of the research is as follows:(i)Define the target participants and select the sampling process.(ii)Task preparation(iii)Tools used(iv)Prepare experiment environment(v)Procedure(vi)Develop and design self-assessment log(vii)Data collection
3.1. Defining the Target Participants and Select the Sampling Process
The populations of our research were novice software developers working in the professional software development industry. The criterion was set to choose those professional developers whose work experience is less than two years. As it was not feasible to analyze the whole population, we selected a convenience sample of five novice developers for experiment and assessment. All these developers were working on Android game development using C# as a development language at a local software firm.
3.2. Task Preparation
We assigned a set of tasks to each developer for five days. The experiment included five tasks, one task for each developer daily, divided into three categories:(i)Simple task (ST)(ii)Intermediate task (IT)(iii)Complex task (CT)
3.3. Tools Used
We selected the Devskiller for observation assessment. Devskiller is a tool that records the time and actions performed by the programmer while performing their coding tasks. Devskiller has been used in other studies for the same purpose [37]. It provides a 14-day trial which was enough for us as our experiment was only five days. On each day, we assigned a specific task in the Devskiller user dashboard and sent invitations to each developer through e-mail. We used Devskiller to investigate different attributes of the developers. These attributes will be discussed in the following section.
3.4. Preparation of Experiment Environment
The five developers were provided seating on separate tables in the same room. As the surrounding environment is also an important factor in self-assessment, the environment was set to be more comfortable and conducive. They were briefed about the purpose of the experiment, the 5-day activities, the plan, and the ultimate goal of the research.
3.5. Procedure
The five tasks that the developers had to complete were divided into three categories. For the first and second days of the experiment, every developer was given a simple task (ST) to complete. The reason to assign simple tasks on days 1 and 2 was to help the programmer revise the basics of coding. On the third and fourth days, some intermediate-level complex task was assigned. On the last day of the experiment, a complex task was assigned to the developers. Table 1 shows the specific task against each day. Aside from the practical task, the developers also filled out the self-assessment log at the end of the daily task.
3.6. Develop and Design Self-Assessment Log
To complement our findings from the data collected through Devskiller, we used a self-assessment log that the developers had to fill out at the end of the daily task. Figure 3 provides an overview of the questionnaire. The log helped us to make our observations reliable. Both, Devskiller log data and data collected through self-assessment log were used to answer our research questions, which are given below:(i)What role can self-assessment play in the lives of novice software developers in minimizing repetitive mistakes?(ii)How can self-assessment skills help novice software developers in the field of programming?(iii)What is the effect of self-assessment on novice developers’ work?(iv)What is the difference between developers who use self-assessment and those who do not?(v)What kinds of mistakes a novice developer makes without analyzing their work?
3.7. Data Collection
To analyze a developer in terms of self-assessment, data could be obtained in two ways: direct source or indirect source. Directly sourced data comes from the developers’ first-hand, which means that the data are collected through the log file of Devskiller and a self-assessment log from developers. Indirect sources are the managerial staff of the software house, where data can be collected through interviews with the team managers and team leads. We used the direct source to collect the data. All the data were collected through the log file of Devskiller and the self-assessment log in Figure 3.
The process of self-assessment includes three main steps, i.e., planning, monitoring, and evaluation. We have considered these three parameters for the self-assessment of developers (see section 3.2.8). In the planning phase, the two other parameters, time taken to make a plan and construct used for planning, were assessed. In the monitoring phase, the number of repetitive mistakes and the type of repetitive mistakes were analyzed and measured, and in the evaluation phase, the technique used by the developer to evaluate his/her solution was analyzed. At the end of the task, the developers were asked to fill in the self-assessment log. The experiment continued for five days.
4. Experiment Evaluation
4.1. Calculating the Attribute of Individual Developers
As discussed, we selected the planning, monitoring, and evaluation phases of self-assessment. A total of seven attributes for these phases were used to assess the self-assessment skills of individual developers. The data about these attributes were collected from the log file of Devskiller and the self-assessment log filled in by the developers on a daily basis. The different attributes that were selected for each phase are shown in Figure 4.
The data about the planning phase were collected from two attributes.(i)The time taken by the developer to plan for the task(ii)The construct used by the developer for planning. This included any pen and paper planning and workflow designs.
The number of repetitive mistakes done by the developer was gathered from the Devskiller log. The log data stores information about all the actions performed by the developer including typos and edits.
The evaluation phase was analyzed by the strategy used by the developer to evaluate his solution. We categorized these strategies into three types.(i)Run code directly; the developer did not spend time in evaluating his code before trying to run it.(ii)Dry run: the developer did a dry run of their code to ensure it would work as required.(iii)Used test cases: the developer generated test cases to test his solution for completeness.
4.2. Assessing Developers Based on Attributes
The self-assessment log was designed to collect information regarding the knowledge and ability of self-assessment of developers in terms of planning, monitoring, and evaluation. Table 2 shows an overview of the data collected from the self-assessment log.
Along with the data collected from the self-assessment log, we also collected data from the Devskiller log file. Table 3 shows an overview of the data collected from the Devskiller log file.
Collectively, the data gathered from the assessment log and Devskiller log file is presented in Table 4.
Table 4 shows the use of self-assessment skills of developer 1 for five days from three perspectives. The first subcolumn of the planning phase tells us that if the developer made any plan to solve the task assigned. If the developer made any plan, the value will be YES, and if they did not make any plan the value will be NO. The next column shows the time taken by the developer for planning. If no plan was made, then the value will be 0. The next column of the planning phase tells us about the strategy used by the developer for making their plan. The specific value was populated using the strategy used by the developer. The next column monitoring has one subcolumn “Number of repetitive mistakes.” This column shows the number of repetitive mistakes made by the developer. The value of this parameter was retrieved from the Devskiller log file. For the evaluation phase, we checked whether the developer evaluated their code before trying to run it. If they did, the next column specifies the method used by the developer to evaluate his solution. The last column task completed or not shows the status of the task given to the developer on the corresponding day. All the collected data from five developers are presented in Table 5.
4.3. Effect of Self-Assessment on Task Completion
From Table 5 and the self-assessment log, we generated Table 6 to show the effect of self-assessment on the task completion of the developers. For self-assessment, it is important that the individual uses his self-assessment abilities uniformly in all the important phases of assessment such as planning, monitoring, and evaluation. The table shows the values of self-assessment and status of task of Developer 1 for five days.
Table 6 shows a positive correlation between the use of self-assessment skills and task completion. The data for the rest of the developers are presented in Table 7.
From the data in Table 7, it is evident that there is always a positive correlation between the use of self-assessment and task completion status. However, there was an exception in the case of developer 3 where on the first day, he did not use any self-assessment ability but still completed the task. On further investigation, it was revealed that the developer had actually used an Internet source and copy-pasted the code that he had found online to complete his task. This does not stand true for all situations in the professional industry since there are many situations for which readily available code is not available and the developer needs to use their experience and skills to solve such problems.
4.4. Effect of Self-Assessment on Repetitive Mistakes
The overall effect of self-assessment on repetitive mistakes is reported in Figure5.
Figure 6 shows the effects of self-assessment on repetitive mistakes. From the results, it can be analyzed that the use of self-assessment skills is negatively correlated to the number of repetitive mistakes. Taking the case of developer 2, on the first day, he did not assess himself and made 10 repetitive mistakes. Since he was more informed on the second day, he was careful to assess himself and be vigilant about what was expected of him. As a result, the number of mistakes he made during the coding exercise decreased to 5. Developer 1’s performance showed the least effect of using self-assessment skills on task performance. His number of mistakes did not see a very big change; however, this number continued to decrease with every passing day, which was a positive sign. This result showed that while still not assessing himself thoroughly, there was still some improvement in his performance due to the awareness of self-assessment. This could also have been affected by the notion of the experiment and therefore would require further investigation which was not the scope of this study.
4.5. Types of Repetitive Mistakes
From the programming perspective, we analyzed only three types of mistakes which were frequent to occur. These were(i)Syntax errors(ii)Semantic errors(iii)Logical errors
The complete data about these mistakes made by developers are shown in Table 8.
For the purpose of analysis, we compared the data on the number of repetitive mistakes (NRMs) for each developer from day 1 and day 5. This was done to investigate whether the experiment had any effect on the use of self-assessment skills by developers and whether this affected the frequency of repetitive mistakes. Table 9 shows the number of mistakes repeated by developers on both day 1 and day 5, respectively.
Figure 5 visualizes the data of Table 9 with categories: NRM (number of repetitive mistakes) without using SA (self-assessment) and NRM with SA.
5. Results and Findings
From the above-performed experiments, we tried to find answers to the following questions.
5.1. What Role Can Self-Assessment Play in the Lives of Novice Software Developers in Minimizing Repetitive Mistakes?
The analysis of the results from the experiment conducted revealed a negative correlation between the use of self-assessment abilities and the number of repetitive mistakes. The results showed a clear diminishment in the number of mistakes made by the developers as the experiment progressed. One of the major factors influencing this trend was the awareness of the intent of the experiment which implicitly led the developers into engaging self-assessment activities. The results provide a clear insight that developers who used self-assessment made fewer repetitive mistakes compared to those who did not assess their work. It is also clear from the results that the awareness of self-assessment has a greater impact on the developer’s aptitude.
5.2. What Kinds of Mistakes a Novice Developer Makes without Analyzing Their Work?
There are three types of mistakes that every developer makes repetitively in everyday work. These mistakes along with their repetition times are presented in Table 9 and visualized in Figure 5. Syntactic mistakes have more frequency compared to other types of mistakes. Syntax mistakes are mostly related to the knowledge of a certain language. Since all the developers involved in this experiment were fresh graduates, they were expected to have an acceptable level of knowledge of the language syntax. Most of the syntax errors that were seen were due to negligence and the fact that the developer did not monitor their own coding practice. The developers showed haste to finish the task and in doing so made plenty of syntax errors. Another important aspect that was discovered here was the dependence on the Integrated Development Environment (IDE) tool. Most of the developers were unsure about their mistakes because of their experience with IDEs. Since all current IDEs provide the IntelliSense feature which highlights syntax errors, most young developers depend upon such tools to keep track of syntax errors. This affects their performance as this allows them to be careless which is why they do not monitor their own coding while performing tasks.
5.3. What Is the Difference between Developers Who Use Self-Assessment and Those Who Did Not?
The experiment was conducted for five days, and during these five days, we saw a clear pattern emerge in terms of the developers who used their self-assessment skills and those who did not. The self-assessment log provided us with first-hand data about the understanding of self-assessment by each developer. It was noted that the developers who were not sure about their metacognitive abilities provided short answers and were reluctant to share more details about the procedures they followed. In contrast, the developers who were more confident about the use of self-assessment were also able to describe their processes more clearly. The performance evaluation also increased with the experiment since the developers started producing fewer errors in their code because they were continuously monitoring and evaluating their work.
6. Conclusion
The purpose of this research was to evaluate novice professional software developers in terms of planning, monitoring, and evaluation to assess their self-assessment skills to improve their work by minimizing repetitive mistakes they make every day. A thorough literature review, a planned experiment and an assessment log were used to identify the factors affecting the performance of developers and reasons for repetitive mistakes and verify the effectiveness of self-assessment on repetitive mistakes through an experiment.
There is a need for self-assessment awareness in the professional software development industry to enhance the self-assessment skills and work of novice software developers. By applying different methods to analyze the self-assessment skills of novice developers like collecting the log data from Devskiller, use of self-assessment log and static observation of developers during a planned experiment. We identified important factors of self-assessment that can influence the work of young developers. From the Devskiller log, self-assessment log data and results can be extracted that self-assessment has a positive correlation with the performance of novice software developers.
This research also addresses the type of mistakes that novice software developer makes repetitively during coding. From the Devskiller log file and static observation of the experiment, it is concluded that most of the developers make syntactic mistakes repetitively because they do not monitor themselves while coding. One of the main reasons for this was found to be their dependence on the latest IDEs which provide them with the facilities of syntax highlighting syntax errors [38–40].
Data Availability
The data used in this research can be obtained from the corresponding author.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
“The authors are grateful to the Taif University Researchers Supporting Project number (TURSP-2020/36), Taif University, Taif, Saudi Arabia.