Sensitivity analysis of the SMARTER and MOORA methods in decision making of achieving students

— Evaluation of student learning in Islamic boarding schools is still limited to the results of exams conducted in writing which can lead to the determination of student achievement using simple criteria, resulting in less than optimal results. In addition, the importance of selecting criteria to suit the learning characteristics of the Islamic boarding school students. This study aims to ﬁnd the best method to make decisions. The method used is Rank Order Centroid (ROC) in assigning weight values to the criteria applied to the Simple Multi-Attribute Rating Technique Exploiting Ranks (SMARTER) and Multi-Objective Optimization By Ratio Analysis (MOORA) methods. This study uses 30 alternatives derived from the scores of students at the Al Ma’ruf Islamic Boarding School Kediri. The results showed the same alternative value in the ﬁrst rank. The accuracy is calculated using sensitivity analysis according to the results of preference values in each method. The sensitivity analysis shows that the lowest value is obtained in the ﬁrst sensitivity calculation. The sensitivity value of the SMARTER method on the ﬁrst sensitivity is 0.0714. At the same time, the ﬁrst sensitivity value of the MOORA method is 0.0076. So the best method is owned by the MOORA method because it has the lowest sensitivity value.


I. INTRODUCTION
The development of digitalization is accelerating, especially in education related to learning techniques and evaluation of digital-based student learning. The evaluation of student learning in the Islamic boarding school environment needs to be carried out, namely innovations regarding the evaluation of student learning in following the trend of technological developments. The evaluation of student learning at the Al Ma'ruf Islamic Boarding School Kediri is limited to the results of exams conducted in writing, causing the determination of student achievement to only use simple criteria, giving rise to subjective results. The paper on the characteristics of the Islamic boarding school curriculum program shows weaknesses in the Islamic boarding school curriculum, especially in administration and learning evaluation, which is carried out through simple written and oral tests [1]. As stated in [1], as a learning organization, Islamic boarding schools must be able to adapt and respond to opportunities and challenges in the development of technology and information, namely competing in the field of education services in Indonesia [2].
Previous research has been carried out, including modeling the recommendations of the best Islamic boarding school students using the Multi-Objective Optimization By Ratio Analysis (MOORA) method [3] but has not used a weighting algorithm. So, this study applies the Rank Order Centroid (ROC) weighting method. Furthermore, prediction of the success rate of student performance studies [4] uses data mining techniques that certainly require training data. In addition, it also evaluates student learning in certain fields, namely the best memorization in Munaqosah Tahfizhul Qur'an [5].
Another research on using the Simple Multi Attribute Rating Technique (SMART) method has been applied to the performance assessment of Exemplary Police Members [6]. Furthermore, using the Simple Multi-Attribute Rating Technique Exploiting Rank (SMARTER) method can handle complaints Jurnal Infotel, Vol. 14, No. 3, August 2022 https://doi.org/10.20895/infotel.v14i3.751 Sensitivity analysis of the SMARTER and MOORA methods in decision making of achieving students management on customer services, for an internet connection service company. Which helps provide priority decisions for resolving complaints according to the difficulty level of the disturbance and the area of interference [7]. Even studies show that in 2014 it was known that more than 100 papers had been written over a 20-year period. Which is the Multi-Criteria Decision Support System (CDSS) was used as a decision-making tool, and one of the methods used was SMART [8].
Furthermore, an analysis of the Multi-Criteria Decision Making method shows that the SMART method is a simple method that allows for all types of weighting techniques [9].
The SMART method, first proposed by Edwards in 1970, is a simple decision-making method that was later improved due to excessive subjective decision-making in the SMART method, which is the SMARTER method [10]. Meanwhile, to provide a weighted value, it is known that a multi-criteria decision analysis shows that the weighting that applies ROC has the best performance [11].
The priority level of the criteria becomes a reference in assigning a weight value. This level is what is applied in the weighting of ROC [12]. The formula for weighting W with K criteria is as follows: So the k criteria have the following weights: with W is criteria weighting value, k is number of criteria, and i is an alternative value. Next, (3) is calculation of the utility value. v with w i is the weight that affects the i-th dimension on the overall value of the evaluation. Meanwhile, v i is the evaluation object on the i-th dimension, and n is number of different value dimensions. The formula for calculating the final value is shown in (4).
with w j is the weight of the 1st criterion, u ij is utility value of the -j criteria for the i-th family, and n i is an alternative final score.
A paper shows the best way to get the ranking of criteria into weights is based on the number of criteria. The distribution of weights, which is following the results of a comparison of five methods, namely Rank-Sum Weights (RS), Reciprocal Rank weights (RR), ROC weights, Weights Geometric (GW), and Variable-Slope Linear (VSL). However, about the probability distribution, ROC is the best method in exponential weighting [13].
The use of ROC has been applied to score areas that have the potential to develop renewable energy made from palm oil waste [14], and its measurement uses multi-criteria decision-making sensitivity analysis [15].
The MOORA method is obtained from the multiplication of the attribute ratings. The attribute rating is raised to the first power with the weight of each column. The Preference Value is aimed at the Si alternative. The order of the MOORA method includes [16]:

A. Determination of the Value of the Decision Matrix
The next step is determining the objectives in identifying the evaluation attributes in question.
with x ij is the alternative response j on criterion i, where i = 1, 2, 3, · · · , n is the sequence number of the attribute or criterion, and j = 1, 2, 3, · · · , m is alternative sequence number. Meanwhile, X is the decision matrix.

B. Matrix Normalization
The denominator comes from the square root of the sum of the squares and each alternative for each attribute [17].
with x ij is an alternative matrix j on criterion i, where i = 1, 2, 3, · · · , n is the sequence number of the attribute or criterion, and j = 1, 2, 3, · · · , m is alternative sequence number. Meanwhile, X * ij is the alternative normalization matrix j on criterion i. In multi-objective optimization, profitable attributes, can be added during normalization. On the other hand, unfavorable attributes can be reduced.
Variable g is the number of maximized attributes, while n − g is the number of minimized attributes. Y i is the value that has been normalized from the first alternative to all attributes (as shown in (8)).
where i = 1, 2, 3, · · · , g, is an attribute or criterion with maximized state. Meanwhile, j = g + 1, g + 2, g + 3,· · · , n is an attribute or criterion with minimized state. Finally, w j and y are weight against alternative j, and the normalized value of the alternative j, respectively.

D. Y i Rating Ranking
The Y i value can be positive or negative based on the decision matrix's maximum and minimum total values. In ranking order, then Y i shows the last choice. So it is known that the best alternative is Y i with the highest value, while the worst alternative is the lowest Y i value.
The results of this study are useful for Islamic boarding schools in making decisions for outstanding students. The combination of the MOORA method and ROC weighting is known to be better based on a lower sensitivity value than the SMARTER method. So, this research can be used to make decisions for outstanding students in a particular field or at other educational institutions.

A. Research Steps
The steps of this research consist of conducting a literature review, collecting research data, using ROC weighting, applying ROC weighting in the SMARTER and MOORA methods, and conducting sensitivity analysis between the two methods. The research steps are shown in the flow chart according to Fig. 1.

B. Literature Review
Literature study is an activity to collect references from trusted sources, namely electronic scientific articles from international and national journals, books, and electronic articles, to get the theory needed.

C. Research Data Collection
The research data used came from the Al Ma'ruf Islamic Boarding School Kediri. The criteria data and the priority order, include kitab kuning reading skills (grammatics), morals, written examination, muhafadzoh, thamrin, attendance, and book correction. Data collection for students is carried out at each grade level. So the results of the calculation show that students excel at that class level.

D. Assessment Criteria, and Priorities
The following is an explanation of the criteria, the value of each sub-criteria, and the order of priority. An explanation of the criteria used in this study is shown in Table 1. The value of the C1 criteria is shown in Table  2. Meanwhile, the value of C2 criteria obtained from the sub-criteria by the Islamic boarding school characteristics and provisions is shown in Table 3 and  Table 4.  Students participate in deliberations before teaching and learning activities begin. 4 Students follow Lalaran (communal repetition of memorizing activities) before teaching and learning activities begin. exam, and C5, namely Thamrin, is the midterm exam. The value of the two criteria is taken from the average important value taken by each student. The C3 and C5 criteria values are shown in Table 5. The value of the C4 criteria is obtained from the implementation of the book, not good memorization exam (Muhafadzoh), which was determined at the beginning of the grade promotion. Therefore, the value of the C4 criteria is shown in Table 6. The value of the C6 criteria is shown in Table 7. The value of the C7 criteria comes from the number of lessons a student takes. The requirement for obtaining a perfect assessment (tam) is that none of the lessons (books) are defective (naqish). The assessment of book corrections is shown in Table 8.

E. ROC Weighting
The weighting of the criteria uses the ROC algorithm in order of priority. The value of the weighting criteria using the ROC is shown in Table 9.

A. Implementation of the SMARTER Method
Based on the final value of the calculation using the SMARTER method, namely the total utility value for each alternative, the highest value indicates the first rank owned by the 4th alternative (A4). The utility value for each student is shown in Table 10.

B. Implementation of the MOORA Method
The application of the MOORA method produces a ranking for each alternative. The ranking results are shown in Table 11. The 4th alternative indicates the highest value.
Based on the ranking results from the SMARTER and MOORA methods, it is known that the fourth alternative data has the highest value. The SMARTER method for the highest score of A4 is 2.143. At the same time, the highest alternative MOORA value method owned by A4 with total value 0.229.

IV. DISCUSSION
Sensitivity analysis was performed to measure the ranking results. Measurements are carried out by subtracting the final value of the first alternative and the second alternative, dividing the value of the first alternative by the total number of available alternative values, and dividing by two the sum of the results of the first alternative and the second alternative [15].
Based on the sensitivity comparison between the SMARTER and MOORA methods that apply ROC weighting in Table 12, the first, second, and third sensitivity values have been obtained. The first sensitivity value for the SMARTER method is 0.0714,  while the MOORA method is 0.0076. The same result is obtained for the second sensitivity value for the SMARTER and MOORA, for 0.0334. Furthermore, the third sensitivity value for the SMARTER method is 0.4617, while the MOORA method is 0.1954. Therefore, the results of the sensitivity analysis in this study indicate that the best method is the MOORA method which has the lowest value in the calculation of the first sensitivity value.

V. CONCLUSION
Applying the ROC weighting algorithm in the SMARTER and MOORA methods has resulted in the best alternative ranking that shows students' achievement. The results of the first rank of alternative students are shown on the same alternative that comes from the calculation of the two methods. Based on the calculation of the sensitivity value, it can be concluded that the best method is MOORA because it meets the lowest sensitivity value. This research is still limited to data analysis. In the future, it can be developed by building intelligent applications that can be applied to several platforms such as web-based and mobile. And can be developed more widely by applying other multi-criteria decision-making methods.