This paper studies the effect of laboratory activities on the results obtained by the students on 7 core curricular units (CUs) of an Integrated Master in Mechanical Engineering. Each CU is characterized by means of 4 performance indices, over the 6-year period 2014/2015–2019/2020. Firstly, individual indices in a per semester basis are compared. Secondly, the CUs are regarded as objects defined in a 4-dimensional space of features, and the multidimensional scaling (MDS) technique is adopted for clustering and computer visualization. The MDS is powerful for analyzing the multivariate dataset, unveiling patterns not perceived by standard statistical methods.

1. Introduction

Laboratory training is fundamental in engineering education [110], but understanding how the experimental activities contribute to meaningful learning and to achieve the goals established for a specific curricular unit (CU) is challenging [1114]. Researchers often classify the educational engineering laboratories as follows: (i) hands-on with real equipment; (ii) simulated or virtual; and (iii) remote [1519]. Despite this strict classification, many laboratory environments adopt a blended approach, where hands-on, virtual, and remote experiments are combined together, trying to simultaneously potentiate the advantages and mitigate the drawbacks of each one [20, 21]. The laboratory experiments envisage strengthening: (i) the conceptual understanding, namely, the students’ capabilities to understand and to solve problems related to the fundamental concepts presented in the theoretical classes; (ii) the design skills, meaning the students’ skills to solve open-ended problems through the design and creation of new processes; (iii) the social skills, which mean the students’ abilities to learn how to undertake engineering-related work in groups; and (iv) the professional skills, namely, the students’ capabilities to become familiar with the technical expertise that they will need in their professional activities [1, 2]. The best procedure to accomplish these four objectives in the context of undergraduate engineering programs is an open issue, which is hardened by multiple specific factors, as the teaching/learning subjects, the students’ backgrounds, or the number of students enrolled in the CU.

The Integrated Master in Mechanical Engineering (MIEM) at the Faculty of Engineering of the University of Porto (FEUP) prepares students so that, after completing the cycle of studies (CS), they can initiate a professional career in applied and/or research activities in Mechanical Engineering or related fields. To accomplish such objective, MIEM provides the students a solid educational framework with a rich scientific background and emphasis on the interaction between theory and practice within the main domains of Mechanical Engineering.

The importance of laboratory classes has always been recognized by those responsible for the CS, and these classes were implemented on most CUs of MIEM. In fact, the laboratory classes were early considered as an asset that allowed the students to acquire an important training component in technological areas and motivated them greatly to engage in the learning processes. The laboratory classes were also intended to give the students the opportunity to work in teams and an avenue to communicate their ideas in a concise, objective, and pragmatic way, as well as to develop their critical analysis of results. The increase in the laboratory component of the MIEM CUs was successively recommended by the accreditation bodies, namely, for the CUs in the areas of Applied Mechanics and Fluids and Energy, which are core areas of Mechanical Engineering. All CUs of these scientific areas included in the common part of the CS syllabus had no laboratory activities. In some of them, the students had extra difficulties, which translated in getting worse results than those verified in other CUs of the CS program. These CUs had in common the adoption of teaching methods based on theoretical classes for the exposition of subjects and theoretical-practical classes for problem solving. Moreover, they had a high number of enrolled students and high failure rates. To mitigate these issues, while avoiding changing the MIEM syllabus, which would have involved a slow and bureaucratic administrative process, the direction of the CS, with the support of the Mechanical Engineering Department (DEMec), decided to implement mandatory laboratory activities in some CUs of the second up to sixth semesters of MIEM, starting in the academic year 2016/2017. The procedure adopted, still in practice, requires that the students should carry out a set of laboratory works throughout the semester. Such activities are performed in small team groups, following the guidelines provided by the CU staff, with the objective of applying the knowledge acquired at the theoretical and theoretical-practical classes. The laboratorial experiments should follow a script with precise instructions and be completed in a limited time, while the students are supported by monitors, who typically are students of the last semester of MIEM. The students are assessed via small written reports, one per experiment.

In this work the effect of laboratory activities on the results obtained by the students on 7 core CUs of MIEM is assessed. These CUs are {Mechanics I, Solids Mechanics, Mechanics III}, to be denoted by {MI, SM, MIII}, and {Thermodynamics II, Fluid Mechanics I, Fluid Mechanics II, Heat Transfer}, to be denoted by {TII, FMI, FMII, HT}. The two sets belong to the areas of Applied Mechanics and Fluids and Energy, respectively. Each CU is characterized by the 4 performance indices {passed/enrolled students’ ratio, passed/assessed students’ ratio, arithmetic mean of the passed students’ marks, arithmetic mean of the assessed students’ marks}, to be denoted by {Pas/Enr, Pas/Ass, Mean Pas, Mean Ass}, over the 6-year period 2014/2015–2019/2020. Firstly, the performance indices characterizing each CU are compared with those of the CUs in the same semester, and their time evolution is analyzed in an individual basis. Secondly, the CUs in a given semester are considered objects defined in a 4-dimensional space of features, and the multidimensional scaling (MDS) technique is adopted for dimensionality reduction, clustering, and computer visualization of patterns. This is accomplished by choosing a suitable distance metric for object-to-object comparison and Procrustes analysis for obtaining meaningful maps of objects (that is, CUs) which are able to unveil possible hidden relationships between the data.

Having these ideas in mind, the paper organization is as follows. In Section 2, the MIEM program and the CUs are introduced briefly. In Section 3, a statistical study is performed in a per semester basis, to understand the time evolution of the results obtained in each CU. In Section 4, a multivariate analysis is conducted to unveil possible patterns embedded in the data, which are difficult or impossible to be noticed when adopting simple statistics. Finally, in Section 5, the main conclusions are summarized.

2. The Curricular Units in the Context of the MIEM Program

The current MIEM is a CS combining a first and a second cycle degrees into a single 5-year CS, corresponding to the bachelor (3 years) and to the master (2 years), respectively. The MIEM is organized in 10 semesters. At the end of the first cycle, the students have a solid base education, comprising the essential scientific and engineering skills and fundamental, nonspecialized knowledge in the various areas of Mechanical Engineering. After concluding the second cycle, the MIEM graduates have advanced level of training in Mechanical Engineering, focusing in Automation, Design and Manufacturing, Industrial Management, Mechanical Project and Construction, or Thermal Energy.

The 7 CUs of the MIEM program considered herein {MI, TII, SM, FMI, FMII, MIII, HT} occur between the second and sixth semesters. Each semester includes 5 CUs in total, as follows:(i)Second semester: {Mathematical Analysis II, Computer Programming I, Statistics, Thermodynamics I, Mechanics I}, to be referred as S2 = {MAII, CPI, S, TI, MI}(ii)Third semester: {Mathematical Analysis III, Electricity, Numerical Analysis, Mechanics II, Thermodynamics II}, to be referred as S3 = {MAIII, E, NA, MII, TII}(iii)Fourth semester: {Metallic Materials, Mechanical Engineering Drawing, Instrumentation for Measurement, Solid Mechanics, Fluid Mechanics I}, to be referred as S4 = {MM, MED, IM, SM, FMI}(iv)Fifth semester: {Structural Mechanics I, Fluid Power Systems, Computer Aided Design and Manufacturing, Non Metallic Materials, Fluid Mechanics II}, to be referred as S5 = {SMI, FPS, CADM, NMM, FMII}(v)Sixth semester: {Manufacturing Processes I, Industrial Automation and Safety, Information Systems, Mechanics III, Heat Transfer}, to be referred as S6 = {MPI, IAS, IS, MIII, HT}

Table 1 summarizes a few details of the CUs, namely, the students/staff total contact hours and the total amount of time allocated to the CU, regarding autonomous study, classes, and other academic activities. For thorough information, please consult the FEUP’s website at http://www.fe.up.pt.

M = 7 CUs in the set {MI, TII, SM, FMI, FMII, MIII, HT} and they have adopted identical teaching methods based on theoretical classes for subject exposition and theoretical-practical classes for problem solving. Experimental activities have been negligible. The considerable numbers of students enrolled, between 202 and 275 in year 2019–2020, and the high failure rates in the CUs became a concern to the CS and DEMec directors. Therefore, in an attempt to relieve the problem, laboratory activities were adopted in 2016/2017, for {MI, SM, MIII, FMII, HT}, and in 2017/2018, for {TII, FMI}. Such laboratory activities require that the students should carry out predefined laboratory experiments throughout the semester. The work is performed in small groups, according to provided guidelines and in-loco staff support, which envisages consolidating the subjects taught in both the theoretical and the theoretical-practical classes.

3. Statistical Analysis of the Students’ Performance

In this Section, the 4 performance indices {Pas/Enr, Pas/Ass, Mean Pas, Mean Ass} that characterize each CU over the 6 academic years 2014/2015–2019/2020 are compared, in an individual basis, with those that characterize the CUs in the same semester. This means that MI is compared with the CUs in S2, TII with the CUs in S3, and so on, and the comparison involves one index at a time.

Figure 1 depicts the performance indices of the CUs versus time. The red lines represent the values of the performance indices of each indicated CU, and the black lines represent the arithmetic means of the performance indices of the CUs in the same semester, either S2, …, S6, respectively. The bottom and top edges of the blue boxes represent the 25 and 75 percentiles, while the central marks stand for the medians. The whiskers extend between the extreme data points, when excluding the outliers, and the outliers are represented by means of the symbol “o.” We verify that most CUs’ performance indices follow the trend exhibited by the CUs in their semester, over the entire period 2014/2015–2019/2020. This would mean that the effect of the laboratory activities is either negligible or hidden. However, some CUs reveal complex relationships for certain performance indices, as those depicted in the graphs denoted by (xii), (xiv), and (xv), and further analysis is needed.

To get more solid conclusions, we calculate the Pearson correlation as follows:where represents the values of a performance index {Pas/Enr, Pas/Ass, Mean Pas, Mean Ass} for a given CU and stands for the arithmetic mean of the index for the CUs that belong to the same semester, in the 6-year period 2014/2015–2019/2020. The symbol denotes the arithmetic mean of the argument. Table 2 summarizes the results obtained. We verify that is high for most CUs, with the exception of {SM, FMI, MIII}, which exhibit small values of for certain performance indices. This confirms the rough assumptions made previously through simple observation of the graphs since a low translates into discrepancies between the trends exhibited by a specific CU and the CUs in the same semester.

4. Multivariate Analysis of the Students’ Performance

The analysis performed in Section 3 using simple statistics showed that not all performance indices lead to identical results. Therefore, examining the indices separately may reveal inconsistencies and bring the analysis inconclusive. To mitigate this issue, in the follow-up, a multivariate analysis is performed, meaning that all features characterizing the data are considered simultaneously.

Several mathematical techniques are available for multivariate data analysis. The principal component analysis (PCA) [22], additive trees [23], cluster analysis [24], factor analysis [25], and multiple regression analysis [26], among others [27], can be cited. Herein, the MDS [2831] is adopted. MDS has advantages over other methods, such as PCA, which has difficulties to identify clusters surrounded by rings of points. Moreover, MDS is a more general projection method than PCA [28].

4.1. MDS Data Clustering and Visualization

MDS is a computational technique for visualizing information [2831]. MDS detects meaningful fundamental dimensions that allow the researcher to visualize (dis)similarities between the objects of interest. Given M objects in a q-dimensional space, the algorithm receives as input a M × M matrix Δ = [δij] of object-to-object (dis)similarities, calculated based on a distance metric. The MDS assigns coordinates to each object in a p-dimensional space, with p ≤ q, and organizes the set in a symmetric matrix Φ = [ϕij] in order to reproduce Δ as best as possible. This is accomplished by means of a function optimization procedure that tests several configurations. A common index to assess how well a given configuration in Φ reproduces the original distances matrix Δ is the raw stress as follows:where f(δij) stands for a linear or nonlinear function. The smaller the stress, R, the better is the fit between Φ and Δ. For  ≤ 3, the objects can be represented in a graph that can be analyzed in the perspective of the objects location, meaning that similar objects appear close to each other, while dissimilar objects are located distant from each other. The orientation of the final MDS chart has no special meaning, and the map is insensitive to translation, rotation, and magnification.

For each year, 2014/2015–2019/2020, the data characterizing the CUs in a semester are organized in a 5 × 4 array, , where , i = 1, …, 5, k = 1, …, 4, represents the ith CU in a given S2, …, S6 semester and its kth characterizing index in the set {Pas/Enr, Pas/Ass, Mean Pas, Mean Ass}. The array is firstly normalized to prevent numerical saturation. Therefore, the columns of , to be represented by , are expressed as follows:

This leads to a normalized array , where represents the standard deviation of the argument. Then, the rows of , to be represented by , are used to compute the matrix Δ = [δij], i, j = 1, ..., 5, where denotes the Euclidean distance:

Finally, Δ is processed through the MDS algorithm to generate the maps of objects that represent the CUs in per semester and per year basis.

Figures 26 portrait the MDS 3-dimensional maps of the CUs in the semesters S2, …, S6 and, for each, in the 6-year period 2014/2015–2019/2020.

Figure 2 unveils the existence of 2 clusters every year. Clusters are separated by dotted lines. Often, one cluster includes {MI, TI} and the other comprises {CPI, MAII, S}. Exceptions occur in years 2016/2017 and 2018/2019, where CPI leaves the cluster {CPI, MAII, S} and steps into {MI, TI}. Therefore, MI is similar to TI, and the two CUs are quite different from the pair AMII and S. Often, CPI is close to MAII and to S but reveals some variability. In conclusion, over the period 2014/2015–2019/2020, MI did not change its relative positioning with respect to the other CUs in the semester S2, meaning that no effect of the laboratory activities on MI can be noted.

Figure 3 depicts the dynamics of the CUs in semester S3 over the 6-year period 2014/2015–2019/2020. In most years, 3 clusters emerge, namely, {MAIII, NA, TII}, {E}, and {MII}. E and MII are CUs that have better and worst results than the others, respectively. In years 2016/2017 and 2019/2020, the CU E enters the cluster {MAIII, NA, TII}. Focusing on TII, it can be seen that the CU follows the mainstream. Therefore, TII keeps its relative positioning with respect to the other CUs in the semester S3, meaning, as before, that no effect of the laboratory activities on the CU can be noted.

Figure 4 portraits the MDS maps of the CUs in semester S4. In this case, the behavior of the CUs is almost random, which inhibits any attempt to interpret their dynamics in the period 2014/2015–2019/2020.

In Figure 5, we verify the existence of 2 clusters every year. One includes {CADM, NMM}, and the other comprises {FMII, FPS, SMI}. These clusters correspond to CUs with the best and the worst results in the semester S5. The patterns formed by the CUs are stable over the period 2014/2015–2019/2020. Thus, regarding FMII, it means that the effect of the laboratory activities cannot be distinguished.

Figure 6 depicts the dynamics of the CUs in semester S6 over the 6-year period 2014/2015–2019/2020. In most years, 2 clusters emerge, namely, {HT, MIII} and {IAS, IS, MPI}. The exception is years 2017/2018 and 2018/2019, where IAS and IS change a bit. The CUs HT and MIII are always close to each other, but they change their loci in year 2019/2020, meaning that MIII seems to move towards the mainstream, which translates into improving its results.

In synthesis, the multivariate analysis performed unveils some evidences about the behavior of the CUs, but in most cases the results are still somewhat inconclusive.

4.2. MDS with Procrustes Data Clustering and Visualization

To improve the visualization and ease the detection of possible relationships between the data, each set of maps depicted in Figures 26 can be replaced by one chart that collapses all information embedded in the set. Hereafter, this is accomplished by means of Procrustes analysis. Procrustes is a numerical technique that takes a collection of shapes and transforms them for maximum superposition, using the linear transformations O = {translation, reflection, orthogonal rotation, magnification/reduction of size} [29, 3234].

For the CUs in each semester S2, …, S6, Procrustes involves four iterative steps: (i) the user selects a reference MDS map (in the present case, the one corresponding to the year 2014/2015 is adopted but any available instance can be used); (ii) the algorithm superimposes all remaining maps into the current reference using the transformations in O; (iii) it computes the ‘mean form’ of the current set of superimposed maps; and (iv) it calculates the distance between the “mean form” and the reference instances and compares such distance with a given threshold value. If the distance is greater than the threshold, then the algorithm sets the reference to the “mean form” and steps to (ii). Otherwise, it keeps the reference and steps to (ii). At the end, a global representation of all maps that best conforms them is obtained.

Figures 711 depict the MDS 3D maps of the CUs in each semester S2, …, S6 obtained by Procrustes. For good readability, circles denote CUs in a given year of the period of time 2014/2015–2019/2020. Square and star marks indicate the first and last year of the interval. The arrows represent the flow of time.

The results obtained are consistent with those in Subsection 4.1, but further conclusions can be drawn, with the advantage of having just one MDS map to interpret:(i)Figure 7 depicts the map of the CUs in semester S2. The right- and left-hand sides of the chart are populated with the CUs with best {MAII, S} and worst {MI, TI} results, respectively. In the middle, PCI emerges as a CU with intermediate results. It can be seen that, in some years, PCI approaches the left- or the right-hand sides of the map. The CU MI evolves in time, describing a path in the MDS, but it does not move toward the right-hand side, meaning that the results in the UC are not improving;(ii)The CUs in S3 are portrait in Figure 8. In the right- and left-hand sides of the chart are the CUs with best {E} and worst {MII} results, respectively. In the middle of the chart are the CUs with intermediate results {MAIII, NA, TII}. It can be seen that TII describes a path with a certain amplitude but remains in the central region of the MDS, meaning that no significant changes can be noticed on the results over the period 2014/2015–2019/2020. The CU MII evolves within a small volume on the left-hand side, away from the CUs with intermediate and good results. This means that this UC has results clearly worse than the others and should be the focus of special attention;(iii)Figure 9 depicts the map of the CUs in semester S4. The CUs with best results {IM, MED, MM} are placed at the top-left region of the MDS. The CU FMI evolves on the right-hand side, approaching {IM, MED, MM} in the first years and, then, moving away in the most recent years of the period 2014/2015–2019/2020. The CU SM starts on the bottom of the map and approaches the set {IM, MED, MM}, especially from the year 2017/2018 onwards. Therefore, it can be concluded that MFI is not improving, but SM is getting better results;(iv)In Figure 10, which portraits the dynamics of the CUs in semester S5, it can be seen that there are 2 CUs with much better performance than the others. The best {CADM, NMM} are on the right-hand side, while the worst {FMII, FPS, SMI} evolve on the left-hand side of the MDS map. Regarding FMII, it seems to exhibit a timid movement toward the set {CADM, NMM}, which translates into a slight improvement on the results in the UC;(v)Figure 11 shows the MDS of the CUs in semester S6. The right- and left-hand sides of the chart are populated with the CUs with worst {HT, MIII} and best {IS} results, respectively. The middle region is occupied by the CUs that have intermediate results {IAS, MPI}. It can be seen that HT evolves within a relative small volume, with no significant variation on the results obtained over the period 2014/2015–2019/2020. The CU MIII evolves approaching the middle of the map, meaning that the results are getting better.

5. Conclusions

The effect of laboratory activities implemented in 7 core CUs of the MIEM degree at FEUP was investigated in the perspective of the results achieved by the students. The CUs were characterized by 4 performance indices in the 6-year time horizon 2014/2015–2019/2020. In a first phase, the analysis of single indices revealed inconclusive and pointed towards the need of using multivariate analytical models to compare and visualize the data. In a second phase, the CUs were considered as objects defined in the 4-dimensional space of features, and the MDS was adopted for dimensionality reduction, clustering, and data visualization. Finally, Procrustes processing was used to achieve meaningful maps of objects which are able to unveil all hidden relationships between the data. The MDS and Procrustes showed that, for now, the impact of the laboratory activities in the results of most CUs is insipient. However, for SM, FMII, and MIII, the effect seems to be positive since the results in these CUs have been improving in the last few years. Nevertheless, a longer period of analysis is needed to reach more solid conclusions.

Data Availability

The data used to support the findings of this study are available from the author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.