TL;DR

Leading a team of mechanical engineers, computer programmers, and educational learning students to design and experiment with an innovative multi-screen handheld device for interactive language learning. Proposed decentralized software architecture for most efficient prototyping, recruited experts on required fields, and designed the software stack for screen controllers, central file & motion processor, and communication protocols.

As of Aug 2025, we have completed the construction of the dice! It works beautifully, and we are excited to start testing it in real classrooms. We presented this work in a panel session at the Serious Play 2025 Conference!

Project Background

While I was working at the Shapiro Design Lab, two members of the U-M language resource center requested collaboration to build an interactive device with multiple screens that supports multimodal language learning in the form of a physical game. The inspiration of the prototype is from the rolling of a dice, and the clients hope that a multi-screen device will enable unique real-time interactions between students on various language learning goals, and gamify the challenging tasks in language teaching, so the learning process is more efficient and enjoyable for the students.

In the field of creative learning research, it is widely recognized that learning as a game is multiple times more effective (and more efficient) as learning with mechanical memorization and repetition. The goal of this project is to create a versatile hand-held device that is sharable among students in a language learning classroom, supporting activities such as formulating sentences from random images, or collaborative/adversarial games, where only texts are given to one student, and a student on the opposing side receives images, and they must complete a series of tasks with only partial information to complete the game.

The device, originally envisioned as a six-screened dice-like object (bigger, of course), enables just those scenes. Ahead of our team, there were two major challenges: 1. how to design the mechanical, electrical, and software prototype in the most efficient way; and 2. what gamified strategies of learning are we able to evaluate with this device in a way that previous tools were not able to support.

We thought of these questions carefully over the following months. Starting with the engineering question of architectural design, we parallelized into two groups, where the engineering group develops the product, and the field study group designs the experiments, with active communication between the two groups.

Development process

When I took over this project at SDL, a group of engineers and data science students at U-M already gave the project a go, and yet the clients at Language Resource Center is have a hard time approving of their prototype. That prototype contains multiple rectangular screens, each requiring an Arduino and an SD card reader, meaning 6 hidden SD card readers for the package. The user experience for instructors to load and unload six hidden micro-SD cards will be miserable. Secondly, that prototype failed to take into consideration the usecase of this device, thus missing important features, such as a battery and power management board. Additionally, there is a lack of input devices and interconnects to interpret the inputs. As an output-only device, the use cases is extremely limited.

Recognizing the failure of the previous prototyle, I introduced myself into the project and immediately reset the creative design process back to the drawing board. I took the challenge to organize a series of brainstorming sessions with my engineer colleagues. Despite not trained as an electrical or microprocessor engineer, I observed that the limiting factor in this design is the extraordinary bandwidth requirement in a hand-held prototype, unseen in any other arduino-based scren projects before.

Finally, we devised a flexible prototyping architecture to achieve functionality that no prototypes exist for. With what I learned here and there from the Michigan Hackers embedded systems experience, I proposed a decentralized architecture that allows one driver board to drive one individual screen, and the screen controllers would be connected to a central processor that handles the communication fabric. This architecture enables each screen to update its content in a timely, smooth manner, while offloading the graphics tasks to the edge processors, freeing the processor time for the central processor to focus on input detection and media file processing. Additionally, since not all six screens will be updated simultaneously, we are able to deliver a smooth media playback experience that would seemingly require higher bandwidth.

Checkpoint Status

Currently, the hardware of the dice is mostly designed and manufactured, with one screen available for demonstration and development purposes. More screens will be fitted to the prototype once software development catches up.

The software for the screen controllers, each of which is an ESP32, is under validation phase. This controller software supports the parsing of text, image, and gif packets from a custom protocol over SPI, queue media to the display, and control the display update frequency in a consistent and timely manner.

The field experience study team is still in early stages of brainstorming ideas for test cases, in collaboration with the Language Resource Center. A more detailed plan for sets of experiments and arrangements is expected by the end of this year.