CodeBloc

A mixed reality (MR) application that harnesses the power of immersive technology to teach foundational programming concepts to K-12 students.

Chiao Lin + Tara Dorje | University of Michigan School of Information

THE PROBLEM

Both traditional text-based & block-based visual programming languages limit students to stationary environments & two-dimensional interactions unsuitable for all learning styles.

OUR RESEARCH

Diverse Modes of Absorbing Information

According to the VARK model of learning, there are four distinct learning styles: Visual, Aural, Read/Write, & Kinesthetic (Fleming & Mills, 1992). Educational resources for learning how to code should address this diversity of learning modalities.

OUR RESEARCH

Benefits of Embodied Learning

Research has highlighted the numerous benefits of embodied learning, illustrating how integrating physical movement into instruction can significantly enhance learning outcomes.

OPPORTUNITY

Mixed Reality (MR) Technologies

Modern MR devices feature sophisticated scene understanding, passthrough, semantic tags, and advanced hand tracking capabilities, which pave the way for immersive learning experiences. These MR technologies support interaction with both real and virtual content, allowing for a more intuitive and engaging learning process.

Passthrough

MR headsets enable users to see the world around them through built-in device cameras. This bridges the gap between a fully immersive, virtual reality environment and the user’s need to interact with or be aware of their physical surroundings without removing their headset.

Hand Tracking

Refers to methods used to capture and interpret the movements of a user’s hands. This technology enables users to interact with content in a natural and intuitive manner, as they can use their hands and fingers to manipulate virtual objects, navigate menus, and communicate through signs/gestures.

Scene Understanding

Involves recognizing the objects, surfaces, and spatial relationships within a user’s surroundings. By understanding a scene, MR devices seamlessly overlay virtual content onto the physical world in ways that are contextually relevant and interact properly with real-world elements.

OUR RESEARCH

Learning in Mixed Reality

Recent studies make a strong argument for the adoption of mixed reality into educational settings. MR technology can serve various learning needs and preferences, with the potential to bring about more effective and inclusive education practices.

Henriksen et al. (2023)

Explores the use of virtual reality (VR) and embodied learning approaches to improve letter-sound knowledge and attentional control in preschool children.

  • Use of virtual reality in instruction can diminish cognitive load for children with low working memory
  • Possibility of improved attentional control and greater motivation to learn
  • Increased sense of presence and interaction opportunities can advance learner engagement and attentional control within learning environment

Lindgren and Johnson-Glenberg (2013)

Outlines the benefits of learning in mixed reality environments, emphasizing the role of embodied learning in improving educational outcomes.

  • MR technologies have potential to transform existing computer-assisted instruction by providing immersive, personalized educational experiences
  • Physical engagement and embodiment in MR environments can lead to improved memory consolidation and retention of learned material
  • Educational benefits can be accessed by a wide range of learners due to the diverse multimodal learning experiences afforded by MR
  • MR environments can support and enhance collaborative learning by providing chances to engage in group activities that promote social skills

OPPORTUNITY

The Market Gap

Most mixed reality experiences available on the market today are skewed towards entertainment, particularly gaming.

Some types of MR apps currently available on Meta Quest and Apple Stores:

  • AR/VR Games (e.g., sports, shooting)
  • Fitness and Wellness
  • Online Streaming
  • Educational/Learning Apps in Other Domains (e.g., piano instruction)

OUR SOLUTION

CodeBloc

An innovative educational platform that uses mixed reality to demystify the fundamentals of coding and programming logic for young learners. 

Product Overview

CodeBloc invites users to assemble interactive 3D code blocks and generate output in an immersive MR programming environment, effectively converting a child’s room into a dynamic coding playground.

Playground

A blank canvas where users can start coding by manipulating code blocks without any predefined objectives, allowing for a free-form exploration of coding concepts. Users can interact with these blocks in a collaborative environment by leveraging the Meta multiplayer feature for an enhanced social learning experience.

Lesson Module

Offers structured learning paths with prompts and visual aids designed to teach various coding concepts, helping to progressively build users’ skills and knowledge.

Mission Module

This feature challenges users with tasks that reward them with experience points upon completion. These XP points unlock more advanced concepts, code blocks, and syntax, to enrich the user’s coding toolkit and encourage continued engagement.

Community Hub

A social space where users can share their projects, exchange ideas, and showcase their achievements, celebrating and promoting creativity while fostering a sense of community and shared learning.

Interaction Walkthrough

Object Blocks

When a user enters the CodeBloc environment, real-world objects (e.g., chair, bed, table, wall) are semantically tagged to become interactive object block elements. Object blocks serve as variables for use in combination with function blocks, blending the virtual and the physical.

Function Blocks

Users can access basic programming functions (e.g., move, rotate, change color) through a gesture-activated menu. After selection, a corresponding function block materializes in the user’s hand, ready for integration into the project. Each function definition is provided through speech feedback from the application upon block selection, in order to ensure understanding.

Initiation of Interaction

Assembling object and function blocks initiates commands which carry out specified action sequences on the designated target object, transforming virtual code blocks into real-world outcomes. This hands-on, interactive approach makes abstract ideas visual and accessible.

CodeBloc Dictionary

User Validation

  • The original CodeBloc prototype (“AIRBLOCKS”) was tested with 60 users during the MIT Reality Hack 2024. Most of them were experienced developers, designers, & researchers actively working in the field of XR (extended reality).
  • The enthusiastic response we received from these experienced XR leaders and professionals indicates that our project fills a genuine gap in the market and holds real potential for consumer adoption.
  • Our project went on to receive an Honorable Mention in the Enhanced Learning category, following evaluation by a panel of 30 expert judges.

Acknowledgement

The Minimum Viable Product (MVP) for this project was developed during the MIT Reality Hack 2024 by the AIRBLOCKS team, which includes:

  • Yu-Chiao (Chiao) Lin: Product Designer, focused on user experience and interface design, developing product features framework, and 3D asset creation.
  • Gustavo Madrigal: Originated the project idea and co-developed the Bezi prototype, enhancing hand gesture experience design.
  • Max Eder: Project Manager, coordinated with engineers on coding logic and managed the production of the introduction video and script.
  • Pin-Jen (Sophia) Wang: Engineer,  responsible for integrating and testing the new Meta All-in-One SDK and setting up scene in the Unity environment.
  • TJ Freeman: Engineer, worked on making the function blocks operational using C# in the Unity environment.

Further development and refinement for the James A. Kelly Learning Levers Prize was undertaken by Chiao Lin and Tara Dorje, and focused on product design, background research, personas and scenarios, and the development of new prototypes to address full product features.