Spatial Integration of Content in Extended Reality Environments
Abstract
Extended reality is a set of immersive technologies of the Fourth Industrial Revolution, including virtual reality, mixed reality, and augmented reality, used in immersive learning environments. However, to ensure the utility of these technologies, it is essential to understand the impact of multimedia principles that ensure spatial integration of information, especially in 3D virtual environments. These principles aim to facilitate the learner’s proper use of cognitive resources. To identify how these principles are used and their influence on the spatial integration of content in extended reality environments, a systematic review of information was conducted. The results demonstrate the influence of three characteristics associated with user differences, allowing the establishment of a set of techniques that promote spatial integration and proposing a taxonomy. Finally, attributes of 3D virtual environment configuration and their influence on learners’ cognitive load were identified.These findings can guide the application of cognitive load theory principles in the design of extended reality environments for learning and training.
Downloads
References
M. P. Ilić, D. Păun, N. P. Šević, A. Hadžić, A. Jianu, "Needs and performance analysis for changes in higher education and implementation of artificial intelligence, machine learning, and extended reality", Education Sciences, vol. 11, no. 10, 2021.https://doi.org/10.3390/educsci11100568
U. G. Longo, S. De Salvatore, V. Candela, G. Zollo, G. Calabrese, S. Fioravanti, L. Giannone, A. Marchetti, M. G. De Marinis, V. Denaro, "Augmented reality, virtual reality and artificial intelligence in orthopedic surgery: A systematic review", Applied Sciences (Switzerland), vol. 11, no.7, 2021. https://doi.org/10.3390/app11073253
J. Cross, C. Boag-Hodgson, T. Ryley, T. Mavin, L. Potter, "Using Extended Reality in Flight Simulators: A Literature Review", IEEE transactions on visualization and computer graphics, vol. 29, no 9, pp. 3961-3975.https://doi.org/10.1109/TVCG.2022.3173921
A. Voronin, А. Danilova, O. Savelyeva, "Application of virtual reality technologies in the training process", Izvestiya of the Samara Science Centre of the Russian Academy of Sciences. Social, Humanitarian, Medicobiological Sciences, vol. 24, no. 83, pp. 10-14, 2022. https://doi.org/10.37313/2413-9645-2022-24-83-10-14
L. Daling, S. Schlittmeier, "Effects of Augmented Reality-, Virtual Reality-, and Mixed Reality-Based Training on Objective Performance Measures and Subjective Evaluations in Manual Assembly Tasks: A Scoping Review", Human factors, pp. 187208221105135. https://doi.org/10.1177/00187208221105135
H. Gómez-Tone, M. Chávez, L. Samalvides, J. Martín-Gutiérrez, "Introducing Immersive Virtual Reality in the Initial Phases of the Design Process-Case Study: Freshmen Designing Ephemeral Architecture", Buildings, vol. 12, no. 5, pp. 518, 2022.https://doi.org/10.3390/buildings12050518
M. E. LeBel, J. Haverstock, S. Cristancho, L. van Eimeren, G. Buckingham, "Observational Learning During Simulation-Based Training in Arthroscopy: Is It Useful to Novices?", Journal of Surgical Education, vol. 75, no. 1, pp. 222-230, 2018. https://doi.org/10.1016/j.jsurg.2017.06.005
N. Yuviler-Gavish, H. Krisher, "The Effect of Computerized System Feedback Availability During Executive Function Training". Journal of Educational Computing Research, vol. 54, no. 5, pp. 701-716, 2016. https://doi.org/10.1177/0735633115628033
E. N. Spruit, G. P. H. Band, J. F. Hamming, K. R. Ridderinkhof, "Optimal training design for procedural motor skills: a review and application to laparoscopic surgery", Psychological Research, vol. 78 no. 6, pp. 878-891, 2014. https://doi.org/10.1007/s00426-013-0525-5
M. Zhang, G. Hou, Y. Chen, "Effects of interface layout design on mobile learning efficiency: a comparison of interface layouts for mobile learning platform", Library hi tech, 2022. https://www.emerald.com/insight/content/doi/10.1108/LHT-12-2021-0431/full/html https://doi.org/10.1108/LHT-12-2021-0431
C. Zhou, F. Yuan, T. Huang, Y. Zhang, J. Kaner, "The Impact of Interface Design Element Features on Task Performance in Older Adults: Evidence from Eye-Tracking and EEG Signals". International Journal of Environmental Research and Public Health, vol. 19, no. 15, pp.9251, 2022. https://doi.org/10.3390/ijerph19159251
P. Ayres, J. Sweller, "The split-attention principle in multimedia learning" in The Cambridge Handbook of Multimedia Learning, 2 ed, Cambridge: Cambridge University Press, 2014, pp. 206-226. https://doi.org/10.1017/CBO9781139547369.011
J. Sweller, P. Chandler, "Why Some Material Is Difficult to Learn Why Some Material Is Difficult to Learn", Cognition and Instruction, vol. 12, no. 3, pp. 185-233, 1994. https://doi.org/10.1207/s1532690xci1203_1
X. Guo, T. Zhu, C. Wu, Z. Bao, Y. Liu, "Emotional Activity Is Negatively Associated With Cognitive Load in Multimedia Learning: A Case Study With EEG Signals", Frontiers in Psychology, vol. 13, 2022. https://doi.org/10.3389/fpsyg.2022.889427
T. Jin, S. Zhou, X. Lang, J. He, Wang, W, "Combined Effect of Color and Shape on Cognitive Performance", Mathematical Problems in Engineering, 2022.https://doi.org/10.1155/2022/3284313
S. Kernebeck, C. Jux, T. Busse, D. Meyer, L. Dreier, D. Zenz, B. Zernikow, J. Ehlers, "Participatory Design of a Medication Module in an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Approach with Nurses and Physicians", Children, vol. 9, no. 1, pp. 82, 2022. https://doi.org/10.3390/children9010082
E. Redlinger, B. Glas, Y. Rong, "Impact of Visual Game-Like Features on Cognitive Performance in a Virtual Reality Working Memory Task: Within-Subjects Experiment", JMIR Serious Games, vol. 10, no. 2, pp. e35295, 2022.https://doi.org/10.2196/35295
M. Qatrunada, M. Harasta, I. Tosofu, "Utilization of Augmented Reality Technology as an Interactive Learning Media", International Journal of Research and Applied Technology (INJURATECH), vol. 2, no. 1, pp. 188-195, 2022.https://doi.org/10.34010/injuratech.v2i1.6917
F. García-Sanjuan, J. Jaen, V. Nacher, "Toward a General Conceptualization of Multi-Display Environments", Frontiers in ICT, vol. 3, pp. 1-15, 2016. https://doi.org/10.3389/fict.2016.00020
U. Rashid, M. A. Nacenta, A. Quigley, "Factors influencing visual attention switch in multi-display user interfaces", Proc. Int. Symp. Pervasive Displays - PerDis '12, pp. 1-6, 2012. https://doi.org/10.1145/2307798.2307799
D. Kossyva, G. Theriou, V. Aggelidis, L. Sarigiannidis, "Definitions and antecedents of engagement: a systematic literature review", Management Research Review, vol. 46, no. 5, pp. 719-738, 2023. https://doi.org/10.1108/MRR-01-2021-0043
L. Cheng, C. R. Beal, "Effects of student-generated drawing and imagination on science text reading in a computer-based learning environment", Educ. Technol. Res. Dev., vol. 68, no. 1, pp. 225-247, 2020. https://doi.org/10.1007/s11423-019-09684-1.
M. Florax R. Ploetzner, "What contributes to the split-attention effect? The role of text segmentation, picture labelling, and spatial proximity", Learn. Instr., vol. 20, no. 3, pp. 216-224, 2010,https://doi.org/10.1016/j.learninstruc.2009.02.021
F. Anvari, H. M. T. Tran, M. Kavakli, "Using Cognitive Load Measurement and Spatial Ability Test to Identify Talented Students in Three-Dimensional Computer Graphics Programming", Int. J. Inf. Educ. Technol., vol. 3, no. 1, pp. 94-99, 2013,https://doi.org/10.7763/ijiet.2013.v3.241
S. F. Fong, "Effects of segmented animated graphics among students of different spatial ability levels: A cognitive load perspective", Turkish Online J. Educ. Technol., vol. 12, no. 2, pp. 89-96, 2013.
H. Simon, P. Liu, D. J. Palombo, T. C. Handy, C. Krebs. "The role of spatial ability in mixed reality learning with the HoloLens", Anatomical Sciences Education, no. 15, pp.1074-1085, 2021. https://doi.org/10.1002/ase.2146
B. Park, S. Münzer, T. Seufert, R. Brünken, "The role of spatial ability when fostering mental animation in multimedia learning: An ATI-study", Comput. Human Behav., vol. 64, pp. 497-506, 2016, https://doi.org/10.1016/j.chb.2016.07.022
L. Peng, Y. Deng, S. Jin, "The Evaluation of Active Learning Classrooms: Impact of Spatial Factors on Students: Learning Experience and Learning Engagement", Sustainability, vol. 14, no. 8, pp. 4839. 2022. https://doi.org/10.3390/su14084839
J. C. Castro-Alonso, P. Ayres, M. Wong, F. Paas, "Learning symbols from permanent and transient visual presentations: Don't overplay the hand", Comput. Educ., vol. 116, pp. 1-13, 2018, https://doi.org/10.1016/j.compedu.2017.08.011
D. Bodemer, U. Faust, "External and mental referencing of multiple representations", Comput. Human Behav., vol. 22, no. 1, pp. 27-42, 2006.https://doi.org/10.1016/j.chb.2005.01.005
M. A. Ghazanfar, M. Cook, B. Tang, I. Tait, A. Alijani, "The effect of divided attention on novices and experts in laparoscopic task performance", Surg. Endosc., vol. 29, no. 3, pp. 614-619, 2015, https://doi.org/10.1007/s00464-014-3708-2
J. Buchner, K. Buntins, M. Kerres, "A systematic map of research characteristics in studies on augmented reality and cognitive load", Computers and Education Open, vol. 2, pp. 100036, 2021. https://doi.org/10.1016/j.caeo.2021.100036
D. Toker, C. Conati, G. Carenini, M. Haraty, "Towards adaptive information visualization: On the influence of user characteristics", Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 7379 LNCS, pp. 274-285, 2012. https://doi.org/10.1007/978-3-642-31454-4_23
C. Conati, G. Carenini, E. Hoque, B. Steichen, D. Toker, "Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making", Comput. Graph. Forum, vol. 33, no. 3, pp. 371-380, 2014, https://doi.org/10.1111/cgf.12393
M. T. Atta, A. Romli, M. A. Majid, "The Impact of AR/VR on Spatial Memory Performance of Learners: A review", International Conference on Software Engineering & Computer Systems and 4th International Conference on Computational Science and Information Management (ICSECS-ICOCSIM), Pekan, Malaysia, 2021, pp. 75-79, https://doi.org/10.1109/ICSECS52883.2021.00021
N. L. Schroeder, A. T. Cenkci, "Spatial Contiguity and Spatial Split-Attention Effects in Multimedia Learning Environments: a Meta-Analysis", Educ. Psychol. Rev., vol. 30, no. 3, pp. 679-701, 2018, https://doi.org/10.1007/s10648-018-9435-9
G, Xuewang and M. Yamada, "The effects of augmented reality on learning performance and cognitive load using the spatial continuity principle", Proceedings of the 17th International Conference on Cognition and Exploratory Learning in the Digital Age CELDA, 2020.
S. Schneider, et al. "How organization highlighting through signaling, spatial contiguity and segmenting can influence learning with concept maps", Computers and Education Open, vol. 2, pp. 100040, 2021.https://doi.org/10.1016/j.caeo.2021.100040
D. Bodemer, R. Ploetzner, I. Feuerlein, H. Spada, "The active integration of information during learning with dynamic and interactive visualisations", Learn. Instr., vol. 14, no. 3, pp. 325-341, 2004, https://doi.org/10.1016/j.learninstruc.2004.06.006
R. Radkowski, J. Herrema, and J. Oliver, "Augmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty", Int. J. Hum. Comput. Interact., vol. 31, no. 5, pp. 337-349, 2015, https://doi.org/10.1080/10447318.2014.994194
N. F. Polys, D. A. Bowman, C. North, "The role of Depth and Gestalt cues in information-rich virtual environments", Int. J. Hum. Comput. Stud., vol. 69, no. 1-2, pp. 30-51, 2011. https://doi.org/10.1016/j.ijhcs.2010.05.007
J. B. Madsen, M. Tatzqern, C. B. Madsen, D. Schmalstieg, D. Kalkofen, "Temporal Coherence Strategies for Augmented Reality Labeling", IEEE Trans. Vis. Comput. Graph., vol. 22, no. 4, pp. 1415-1423, 2016. https://doi.org/10.1109/TVCG.2016.2518318
S. Kalyuga, P. Chandler, J. Sweller, "Managing split-attention and redundancy in multimedia instruction", Appl. Cogn. Psychol., vol. 25, no. 1, pp. 123-144, 1999. https://doi.org/10.1002/acp.1773
W. Rijnsburger, S. Kratz, "Personalized presentation annotations using optical HMDs", Multimed. Tools Appl., vol. 76, no. 4, pp. 5607-5629, 2016, https://doi.org/10.1007/s11042-016-4064-0
A. Renkl, "Toward an Instructionally Oriented Theory of Example-Based Learning", Cogn. Sci., vol. 38, no. 1, pp. 1-37, 2014.https://doi.org/10.1111/cogs.12086
A. Renkl, K. Scheiter, "Studying Visual Displays: How to Instructionally Support Learning", Educ. Psychol. Rev., vol. 29, no. 3, pp. 599-621, 2017. https://doi.org/10.1007/s10648-015-9340-4
B. Ens, J. D. Hincapié-Ramos, P. Irani, "Ethereal planes: A design framework for 2D information spaces in 3D mixed reality environments", in SUI 2014 - Proceedings of the 2nd ACM Symposium on Spatial User Interaction, Honolulu, Hawaii, USA; 2014, pp. 2-12, https://doi.org/10.1145/2659766.2659769
G. L. Hanley, e-Learning and the Science of Instruction, vol. 18, no. 1. Hoboken, NJ, USA: John Wiley & Sons, Inc., 2016.
N. Gavish, T. Gutiérrez, S. Webel, J. Rodríguez, "Design Guidelines for the Development of Virtual Reality and Augmented Reality Training Systems for Maintenance and Assembly Tasks", Int. Conf. Ski., vol. 29, pp. 1-4, 2011,https://doi.org/10.1051/bioconf/20110100029
R. Radkowski, "Investigation of Visual Features for Augmented Reality Assembly Assistance", in HCI International 2015, vol. 9179, R. Shumaker and S. Lackey, Eds. Cham: Springer International Publishing, pp. 488-498, 2015.https://doi.org/10.1007/978-3-319-21067-4_50
M. T. McCrudden, D. N. Rapp, "How Visual Displays Affect Cognitive Processing", Educ. Psychol. Rev., vol. 29, no. 3, pp. 623-639, 2017.https://doi.org/10.1007/s10648-015-9342-2
D. Gupta, An Empirical Study of the Effects of Context-Switch, Object Distance, and Focus Depth on Human Performance in Augmented Reality, Virginia Polytechnic Institute and State University, 2004.
M. S. Arefin, N Phillips, A Plopski, JL Gabbard, JE Swan,"The Effect of Context Switching, Focal Switching Distance, Binocular and Monocular Viewing, and Transient Focal Blur on Human Performance in Optical See-Through Augmented Reality." IEEE Transactions on Visualization and Computer Graphics, vol. 28, 2014-2025 pp. 2018. https://doi.org/10.1109/TVCG.2022.3150503
A. Eiberger, P. O. Kristensson, S. Mayr, M. Kranz, J. Grubert, "Effects of Depth Layer Switching between an Optical See-Through Head-Mounted Display and a Body-Proximate Display", in Symposium on Spatial User Interaction, Oct. 2019, pp. 1-9, https://doi.org/10.1145/3357251.3357588
U. Rashid, J. Kauko, J. Häkkilä, A. Quigley, "Proximal and distal selection of widgets: Designing distributed UI for mobile interaction with large display", Mob. HCI 2011 - 13th Int. Conf. Human-Computer Interact. with Mob. Devices Serv., pp. 495-498, 2011, https://doi.org/10.1145/2037373.2037446
T. Pladere, et al. "When virtual and real worlds coexist: Visualization and visual system affect spatial performance in augmented reality", Journal of Vision, vol.21, no. 17, 2021. https://doi.org/10.1167/jov.21.8.17
M. García Ogueta, "Atención y sus cambios en el espacio visual", Cognitiva, vol. 3, no. 2, pp. 205-236, 1991.
R. E. Su, B. P. Bailey, "Put them where? Towards guidelines for positioning large displays in interactive workspaces", Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 3585 LNCS, pp. 337-349, 2005.https://doi.org/10.1007/11555261_29
B. Ens, R. Finnegan, P. Irani, "The personal cockpit: A spatial interface for effective task switching on head-worn displays", in Conference on Human Factors in Computing Systems - Proceedings, pp. 3171-3180, 2014. https://doi.org/10.1145/2556288.2557058
T. M. R. Dauenhauer, A Taxonomy for Information Linking in Augmented Reality, Lecce, Italy: University of Salento, 2016. https://doi.org/10.1007/978-3-319-40621-3
X. Lei, Y.-L. Tsai, P.-L. P. Rau, "Effect of Layout on User Performance and Subjective Evaluation in an Augmented-Reality Environment", in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Orlando, Florida, USA; vol. 11576 LNCS, 2019, pp. 376-385.https://doi.org/10.1007/978-3-030-22577-3_27
N. Kishishita, K. Kiyokawa, J. Orlosky, T. Mashita, H. Takemura, E. Kruijff, "Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks", ISMAR 2014 - IEEE Int. Symp. Mix. Augment. Real. - Sci. Technol, pp. 177-186, 2014, https://doi.org/10.1109/ISMAR.2014.6948425
E. Kruijff, J. Orlosky, N. Kishishita, C. Trepkowski, K. Kiyokawa, "The influence of label design on search performance and noticeability in wide field of view augmented reality displays", IEEE Trans. Vis. Comput. Graph., vol. 25, no. 9, pp. 2821-2837, 2019.https://doi.org/10.1109/TVCG.2018.2854737
P. Renner, T. Pfeiffer, "Attention guiding techniques using peripheral vision and eye tracking for feedback in augmented-reality-based assistance systems", IEEE Symp. 3D User Interfaces, 3DUI 2017 - Proc., pp. 186-194, 2017. https://doi.org/10.1109/3DUI.2017.7893338
K. Yin, Z. He, K. Li, S. Wu, "Polarization-dependent combiner for wide field-of-view glasses-like AR displays", Proceedings, vol. 11931, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR), no. III, pp. 119310J, 2022. https://doi.org/10.1117/12.2614966
R. Zayas, B. Cox, "Enseñando Precauciones de Seguridad en un EVA: el efecto de la ubicación de la Información e Interactividad", Comput. y Sist., vol. 13, no. 1, pp. 96-110, 2009. http://sro.sussex.ac.uk/24933/.
P. S. Jian, P. Chen, D. A. Bowman, "Testbed evaluation of navigation and text display techniques in an information-rich virtual environment", IEEE Virtual Reality 2004, vol. 52, no. 3, pp. 181-289, 2004. https://doi.org/10.1109/VR.2004.1310072
Y. Fujimoto, G. Yamamoto, J. Miyazaki, H. Kato, "Relation between location of information displayed by Augmented Reality and user's memorization", ACM Int. Conf. Proceeding Ser., pp. 0-7, 2012. https://doi.org/10.1145/2160125.2160132
R. Lawrence, L. Schneider, J. Pratt, "EXPRESS: Can arrows change the subjective perception of space? Exploring symbolic attention repulsion", Quarterly Journal of Experimental Psychology, vol. 75, no. 11, pp. 1997-2011, 2022. https://doi.org/10.1177/17470218221076135
D. Tan, M. Czerwinski, "Effects of Visual Separation and Physical Discontinuities when Distributing Information across Multiple Displays", Proc. Interact'03, pp. 252-259, 2003, http://research.microsoft.com/en-us/um/people/desney/publications/interact2003short-visualseparation.pdf https://doi.org/10.1145/765891.765967
R. D. Vatavu, M. Mancaş, "Visual attention measures for multi-screen TV", TVX 2014 - Proc. 2014 ACM Int. Conf. Interact. Exp. TV Online Video, pp. 111-118, 2014. https://doi.org/10.1145/2602299.2602305
L. Shupp, C. Andrews, M. Dickey-Kurdziolek, B. Yost, C. North, "Shaping the display of the future: The effects of display size and curvature on user performance and insights", Human-Computer Interact., vol. 24, no. 1-2, pp. 230-272, 2009.https://doi.org/10.1080/07370020902739429
U. Rashid, M. A. Nacenta, A. Quigley, "The cost of display switching: A comparison of mobile, large display and hybrid UI configurations", Proc. Int. Work. Conf. Adv. Vis. Interfaces, pp. 99-106, 2012, https://doi.org/10.1145/2254556.2254577.
K. Kia, J. Hwang, I. Kim, J. Kim. "The effects of target size and error rate on the cognitive demand and stress during augmented reality interactions." Applied ergonomics, vol. 97, pp. 103502, 2021.https://doi.org/10.1016/j.apergo.2021.103502
S. Bang, H. Lee, W. Woo, "Effects of Augmented Content's Placement and Size on User's Search Experience in Extended Displays", in IEEE International Symposium on Mixed and Augmented Reality Workshops (ISMARW), pp. 184-188, 2020, https://doi.org/10.1109/ismar-adjunct51615.2020.00056
X. Bi, S. H. Bae, R. Balakrishnan, "Effects of interior bezels of tiled-monitor large displays on visual search, tunnel steering, and target selection", Conf. Hum. Factors Comput. Syst. - Proc., vol. 1, pp. 65-74, 2010, https://doi.org/10.1145/1753326.1753337.
J. R. Wallace, D. Vogel, and E. Lank, "Effect of bezel presence and width on visual search", PerDis 2014 - Proc. 3rd ACM Int. Symp. Pervasive Displays, pp. 118-123, 2014. https://doi.org/10.1145/2611009.2611019
S. Kim, B. Rehder, "How prior knowledge affects selective attention during category learning: An eyetracking study", Mem. Cognit., vol. 39, no. 4, pp. 649-665, May 2011. https://doi.org/10.3758/s13421-010-0050-3
T. Müller, "Challenges in representing information with augmented reality to support manual procedural tasks", AIMS Electron. Electr. Eng., vol. 3, no. 1, pp. 71-97, 2019. https://doi.org/10.3934/electreng.2019.1.71
D. Markov-Vetter, M. Luboschik, A. T. Islam, P. Gauger, O. Staadt, "The Effect of Spatial Reference on Visual Attention and Workload during Viewpoint Guidance in Augmented Reality", Proc. - SUI 2020 ACM Symp. Spat. User Interact., 2020. https://doi.org/10.1145/3385959.3418449
M. Czerwinski, D. S. Tan, G. G. Robertson, "Women take a wider view", Conference on Human Factors in Computing Systems - Proceedings, vol. 4, no. 1, pp. 195-202, 2002. https://doi.org/10.1145/503376.503412
Copyright (c) 2023 Ciencia e Ingenieria Neogranadina
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
