Augmented reality gaming (AR gaming) had its moment of greatness in 2016, when Nintendo and The Pokémon Company released Pokémon GO. The game turned out to be a huge success. Hundreds of millions of people have played it every year since its premiere, and its popularity has never waned. One could draw the conclusion that the potential of AR games is large and within easy reach. Is it really the case?

It turns out that in fact the AR gaming market is very difficult to conquer. Despite strenuous attempts by many developers, it has not been possible to even get close to the success of Pokémon GO. Only some productions with an idea similar to the original, differing mainly in the setting (type of world, graphics, sound effects and music) in which very similar game mechanics are embedded, were able to get a small piece of this pie.

The next Pokémon GO

The main problem seems to be finding an equally attractive but new formula for the game. It is by no means an easy task. Rapid development of AR technology and devices supporting it seem to be the developers’ ally. The use of modern features of AR technology is a direction that may have a chance of success. Games like Pokémon GO make little use of the potential that AR offers in multiplayer games. Modern mobile devices supported by software (ARCore and ARKit) allow developers to build applications that are able to provide much more advanced interactions with the environment and with other participants.

Figure 1: Steps of building a collision mask based on the surrounding environment

Mixing real with virtual environments

What may cause a significant comeback of AR games is the game’s “understanding” of the environment in which the game takes place. A convincing, seamless mix of real-world environments with the virtual ones is the ultimate bait for players’ interest. Direct interaction of the virtual game objects with the environment – like the ability to hide behind real-world obstacles, drawing a virtual graffiti on a real building, or bouncing a virtual ball among physical players – shows directions which may lead AR games to success. However, in order to achieve this goal, non-trivial technical difficulties must be overcome.

One of the use cases in the EC-funded Horizon 2020 project CHARITY aims to overcome these technical issues and provide a solid foundation for advanced AR games. The EC funded project CHARITY aspires to leverage the benefits of intelligent, autonomous orchestration of cloud, edge, and network resources, to create a symbiotic relationship between low and high latency infrastructures that will serve the needs of emerging applications.

Main technical challenges

User location discovery and synchronisation. Modern mobile devices are equipped with both GPS receivers and very precise accelerometers – this makes the device localisation much easier. Both ARCore and ARKit provide an API that allows for motion tracking. Our goal is to verify whether the provided solutions will work in the context of a multiplayer game and will be sufficiently precise throughout the whole game session.

Environment scanning and digitisation. Before virtual objects will be able to interact with the real-world environment, their shape and features must be digitised. While working on this aspect of the game, we explored LiDAR, a method for determining ranges (variable distance) by targeting an object with a laser and measuring the time for the reflected light to return to the receiver. We found that the use of LiDAR technology gives incomparably more precise results, and also much quicker.

Therefore, we have decided that further work in the project will be continued on iOS devices where ARKit and LiDAR are available. An extremely important functionality that we must test is the dynamic building and synchronisation of the mesh-collider based on the data received from several simultaneously operating devices connected to the game session. Figure 1 shows the steps of building a collision mask based on the surrounding environment.

Data merging and synchronisation. Environment data can be collected by all players connected to a game session. It is necessary to build an additional online service for combining many meshes into one. This service must have the functionality for understanding the mutual position of individual meshes by recognizing common parts or other characteristic elements of the mesh. Mesh synchronisation also requires efficient transfer of large amounts of data between connected clients and the server. In addition, the optimal location of the game server needs to be identified to minimise the delays arising at the network layer.

Figure 2: High-level schematic view of the AR multiplayer mobile game deployed in CHARITY

Conclusion

I am convinced that AR games have a bright future ahead. We only need to harness the potential offered by modern technology, make it work efficiently as a uniform system and, most importantly, create games that will bring joy and satisfaction to players.

The hope is that the technology available today will turn out to be complete and mature enough to support modern, feature-reach AR multiplayer mobile games. With the AR gaming use case in the CHARITY project, we are confident to make an important step in this direction.

Further information
Horizon 2020 project CHARITY – www.charity-project.eu