Assessing Algorithmic Thinking: A Psychometric Analysis of Different Assessment Approaches in a Pilot Study
DOI:
https://doi.org/10.32870/ap.v18n2.2858Keywords:
Algorithmic thinking, Educational assessment, Psychometric analysis, Assessment approachesAbstract
Algorithmic thinking is a key skill in higher education; however, its assessment has been limited by a predominant focus on final outcomes rather tan underlying cognitive processes. This study aimed to analyze the psychometric performance of an instrument designed to assess algorithmic thinking through different performance assessment approaches in Mexican university students with varying levels of ability. A maximum-performance test comprising four assessment approaches was developed and administered to 141 students from two public institutions, using indicators from classical test theory and a qualitative analysis of participants’ perceptions. The results showed that thefunctioning of the assessment approach depended on students’ ability levels and revealed discrepancias between perceived difficulty and observed performance. As a limitation, the pilot nature of the study and the relatively small group sizes are acknowledged. The study provides evidence supporting the need for differentiated assessment approaches. It is concluded that the assessment of algorithmic thinking should be aligned with students’ level of development
References
Ackerman, P. L. & Kanfer, R. (2009). Test length and cognitive fatigue: An empirical examination of effects on performance and test-taking motivation. Journal of Experimental Psychology: Applied, 15(2), 163–181. https://doi.org/10.1037/a0015719
Adorni, G. & Piatti, A. (2025). Designing the virtual CAT: A digital tool for algorithmic thinking assessment in compulsory education. International Journal of Child-Computer Interaction, 45, 10076. https://doi.org/1.1016/j.ijcci.2025.100760
Allen, M. J. & Yen, W. M. (2002). Introduction to measurement theory. Waveland Press.
Almanasreh, E., Moles, R. & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066
Barr, V. & Stephenson, C. (2011). Bringing computational thinking to K-12: What is Involved and what is the role of the computer science education community? ACM Inroads, 2(1), 48–54. https://doi.org/1.1145/1929887.1929905
Bengtsson, M. (2016). How to plan and perform a qualitative study using content analysis. NursingPlus Open, 2, 8–14. https://doi.org/10.1016/j.npls.2016.01.001
Brennan, K. & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking [Conferencia]. Annual Meeting of the American Educational Research Association, Vancouver, BC, Canada. https://scratched.gse.harvard.edu/ct/files/AERA2012.pdf
Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston.
Crocker, L. & Algina, J. (2008). Introduction to classical and modern test theory. Belmont, CA: Wadsworth.
Cook, D. A. & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7–166.e16.
Denny, P., Luxton-Reilly, A. & Simon, B. (2008). Evaluating a new exam question: Parsons problems [Conferencia]. ICER’o8 International Workshop on Computing Education Research, Sidney, Australia. https://doi.org/10.1145/1404520.1404532
Ebel, R. L. & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.
Elo, S. & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x
Erlingsson, C. & Brysiewicz, P. (2017). A hands-on guide to doing content analysis. African Journal of Emergency Medicine, 7(3), 93–99. https://doi.org/10.1016/j.afjem.2017.08.001
González Villarreal, R. J. (2014). Efectos del aprendizaje basado en juegos en el pensamiento algorítmico de aprendices de programación [Tesis de maestría, Universidad Autónoma de Aguascalientes]. Repositorio Institucional de la UAA. http://bdigital.dgse.uaa.mx:8080/xmlui/bitstream/handle/11317/1090/389736.pdf?sequence=1&isAllowed=y
Gómez Rodríguez, H., González Fernández, M. O. y Aceves Aldrete, C. E. (2025). Impacto del pensamiento computacional desde una mirada interdisciplinaria en la educación superior. RISTI: Revista Ibérica de Sistemas e Tecnologias de Informação, (57), 39–54. https://dialnet.unirioja.es/servlet/articulo?codigo=10386299
Guitert, M. y Romeu, T. (2022). Competencias “Digitales” del siglo XXI. Universitat Oberta de Catalunya. https://www.studocu.com/pe/document/universidad-tecnologica-del-peru/educacion/competencias-digitales/67679719
Haladyna, T. M. & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 37-50. https://doi.org/10.1207/s15324818ame0201_3
Haladyna, T. M. (2016). Developing and validating multiple-choice test items (3rd ed.). Routledge. https://doi.org/1.4324/9781315819798
Méndez Hernández, S. O. y Bermúdez, J. F. (2023). El pensamiento computacional como competencia para el siglo XXI. Ciencia Latina Revista Científica Multidisciplinar, 7(4), Article 4. https://doi.org/1.37811/cl_rcm.v7i4.7044
Lafuente Martínez, M., Lévêque, O., Benítez, I., Hardebolle, C. & Zufferey, J. D. (2022). Assessing computational thinking: Development and validation of the algorithmic thinking test for adults. Journal of Educational Computing Research, 60(6), 1436-1463.
https://doi.org/1.1177/07356331211057819
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., Simon, B. & Thomas, L. (2004). A multi-national study of reading and tracing skills in novice programmers. In Proceedings of the 6th Australasian Conference on Computing Education (pp. 119–150). https://doi.org/10.1145/1041624.1041673
Merino Soto, C. (2009). Intervalos de confianza asimétricos para el índice la validez de contenido: Un programa Visual Basic para la V de Aiken. Anales de Psicología. https://www.um.es/analesps/v25/v25_1/19-25_1.pdf
Montero Sánchez, N. (2025). Pensamiento Algorítmico en la Enseñanza de Matemáticas del Nivel Secundario: Retos, Estrategias y Oportunidades. Sciencevolution, 4(3), 47–60. https://doi.org/10.61325/ser.v4i3.206
Muñiz Fernández, J. (2010). Las teorías de los tests: teoría clásica y teoría de respuesta a los ítems. Papeles del Psicólogo: Revista del Colegio Oficial de Psicólogos, 31(1), 57-66. https://dialnet.unirioja.es/servlet/articulo?codigo=3150824
Navas-López, E. A. (2022). Una Caracterización del Desarrollo del Pensamiento Algorítmico de los Estudiantes de las carreras de Licenciatura en Matemática y Licenciatura en Estadística de la sede central de la Universidad de El Salvador en el período 2018-202. Revista Iberoamericana de Tecnología en Educación y Educación en Tecnología, (33), e15–e15. https://doi.org/1.24215/18509959.33.e15
Navas López, E. A. (2024). Relaciones entre la matemática, el pensamiento algorítmico y el pensamiento computacional. IE Revista de Investigación Educativa de la REDIECH, 15, e1929–e1929. https://doi.org/1.33010/ie_rie_rediech.v15i.1929
Papert, S. (1980). Computers and Computer Cultures. En Mindstorms Children, Computers, and Powerful Ideas (Segunda). Basic Books. https://rafaelleviblog.wordpress.com/wp-content/uploads/2016/11/mindstorms-chap1.pdf
Parsons, D. & Haden, P. (2006). Parsons programming puzzles: A fun and effective learning tool for first programming courses [Conferencia]. ACE’ 06 8th Australasian Conference on Computing Education, Darlinghurst, Australia. https://dl.acm.org/doi/10.5555/1151869.1151890
Pinzón Pérez, D. F. y González Palacio, E. V. (2022). Incidencia de las habilidades de pensamiento algorítmico en las habilidades de resolución de problemas: Una propuesta didáctica en el contexto de la educación básica secundaria. Estudios Pedagógicos (Valdivia), 48(2), 415–433. https://doi.org/1.4067/S0718-07052022000200415
Pinzón Pérez, D. F., Román González, M. y González Palacio, E. V. (2023). El pensamiento algorítmico como estrategia didáctica para el desarrollo de habilidades de resolución de problemas en el contexto de la educación básica secundaria. Revista de Educación a Distancia (RED), 23(73). https://doi.org/1.6018/red.542111
Posso Domínguez, M. E. y Murcia Londoño, E. (2022). Las “actividades desconectadas” y el desarrollo del pensamiento algorítmico. Universidad Católica de Pereira. http://hdl.handle.net/10785/9635
Robins, A., Rountree, J. & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172. https://doi.org/10.1076/csed.13.2.137.14200
Rodríguez-Abitia, G., Ramírez-Montoya, M. S., López-Caudana, E. O. y Romero-Rodríguez, J. M. (2021). Factores para el desarrollo del pensamiento computacional en estudiantes de pregrado. Campus Virtuales, 10(2), 153–164. http://uajournals.com/ojs/index.php/campusvirtuales/article/view/893
Rojas-López, A. y García-Peñalvo, F. J. (2020). Evaluación del pensamiento computacional para el aprendizaje de programación de computadoras en educación superior. Revista de Educación a Distancia (RED), 20(63). https://doi.org/1.6018/red.409991
Sweller, J. (2011). Cognitive load theory. Psychology of Learning and Motivation, 55, 37–76. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Vasconcelos-Santillan, J. (2008). Development of a Web-Based Surveying Instrument to Identify Problem-Solving Abilities Related to Effective Instruction in Computer Programming [Tesis Doctoral, Johns Hopkins University]. ProQuest LLC. https://www.proquest.com/openview/fe9b6f2d64e9202adcce8ce72ee53310/1?pq-origsite=gscholar&cbl=18750
Velasco-Ramírez, M. L. y Otero-Escobar, A. (2023). El impacto del uso de Scratch para favorecer el pensamiento algorítmico en estudiantes de educación superior. Revista Innova Educación, 5(4), 105–12. https://doi.org/1.35622/j.rie.2023.05v.006
Velasco Ramírez, M. L. y Sánchez Hernández, G. L. (2021). Análisis de la competencia de resolución de problemas algorítmicos en estudiantes de educación superior. En Pensamiento Computacional en México (Primera, pp. 53–71). Academia Mexicana de Computación, A. C. https://amexcomp.mx/media/publicaciones/pensamiento-comp-en-mexico-2021.pdf
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.111821
Wing, J. M. (2008). Computational thinking and thinking about computing. ResearchGate. https://doi.org/1.1098/rsta.2008.0118
Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61. https://doi.org/10.1111/emip.12165
Downloads
Published
Issue
Section
License
The copyright belongs to the authors of the text. In case the manuscript is accepted for publication, the Sistema de Universidad Virtual of the Universidad de Guadalajara (UDGVirtual) is authorized to modify the text in the correction of style, translate, fix, disseminate, reproduce, publish and market with this. Likewise, the Universidad de Guadalajara is authorized to carry out the above described jointly or through possible third parties. (See the detailed authorization in the copyright letter.)
The authors guarantee to Apertura the right to be the first means of publication of the work. The reproduction of the texts published in the Apertura is authorized, in whole or in part, as long as it is non-profit, the source is cited and the content or design is not modified. Authors may exclusively distribute the final editorial or postprint version in any medium and at any time.
Apertura, its editors and UDGVirtual dissociate themselves from the information provided by collaborators or advertisers. The articles of the contributors do not necessarily represent the point of view of the publisher or UDGVirtual and the content of the advertisements are the sole responsibility of the contractors.
Apertura is registered in the National Institute of Copyright with the ISSN number: 1665-6180 (printed version) and e-ISSN 2007-1094 (electronic version), and with the Certificate of Reservation of Rights to the Exclusive Use of the Title: 04 -2005-120212472000-102.
It is also certified by the Qualifying Commission for Illustrated Publications and Journals of the Ministry of the Interior with the Licensed Certificates of Title number: 13449; and Content Legality: 11022.
