Proliferation of VR technology continues at a rapid pace. Given the affordances provided by VR, it’s no surprise that VR training applications have found early success and stand among the top VR application use cases as indicated in the impact section .OVR is an SME active in VR/AR with the mission to transform education and training through innovative VR simulations with low-cost, massively produced VR h/w coupled with deep, proprietary, psychomotor, gamified and multi-user VR s/w platform.
The existing collaborative platform of OVR integrates a custom interpolation engine as part of their SDK offering network optimizations for reducing data transfer and compressing broadcasted values from ConCurrent Users (CCU’s) following a client-server topology over the network. The architecture exploits Unity networking to generate online rooms and manage network traffic through a relay server hosted by Unity in the cloud.
The OVR platform is currently able to support up to eight 8 participants, in contrast to other available collaborative/social VR platforms (e.g. Facebook spaces: 3, Bigscreen: 4). The networking layer of OVR’s platform is depicted in the Figure.
Use case analysis
As part of gamified training scenarios OVR is focused at, solutions are sought to enable fully immersive, untethered/mobile collaborative VR experiences, overcoming the necessity to attach a separate processing unit for unrestricted mobility and in parallel support sensory rich multi-modal interactions and high-fidelity experience of a large number of remote users in a collaborative, shared VR environment with low-latency. Such multi-modal collaborative environments require natural delivery of multiple inputs, the orchestration of multiple modalities and high perceptual fidelity towards enhancing the user experience, enriching the embodiment factor and authenticity of virtual experience.
Although the last years we have witnessed rapid improvements in the GPU processing capabilities of mobile devices and HMDs, still they can not reach the sophisticated graphical detail of high-end desktop GPUs with the primary reason for this performance gap being power consumption. Therefore applications that aim to support a large group of participants suffer of network traffic issues or restrictions on type of interactions based on predefined, predictable user actions with the key bottleneck to realizing untethered VR to be the significant high demand on rendering computation, in tandem with judder-free, blur-free, full visual acuity(*).
Expected Benefits
As part of the project the platform by OVR will be further extended to dynamically manage data services (i.e. prerendering, caching, compression) across the compute continuum exploiting available network resources starting from the end device, i.e. the HMD, towards edge miniclouds and cloud resources. The ACCORDION framework will enable to address challenges related to both latency and local energy consumption constraints considering QoE and QoS. In terms of compression, we will consider the computation to take place on the end-device exploiting the Geometric Algebra Interpolation Engine of OVR, to support fast and efficient compression, simplifying and compacting the broadcasted information on the network. State-of-the-art mechanisms for advanced rendering adopting motion-state estimation, context priority, physics-based constraints will be exploited to support the dynamic optimization of the interpolation engine, considering also the network characteristics and to set the rendering core workflow in the edge minicloud to deliver accurate results at low latency optimizing the average transmission rate.
(*)E. Cuervo, K. Chintalapudi, and M. Kotaru. Creating the perfect illusion: What will it take to create life-like virtual reality headsets? In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications, HotMobile ’18, pages 7–12, New York, NY, USA, 2018. ACM. ISBN 978-1-4503-5630-5. doi: 10.1145/3177102.3177115. URL http://doi.acm.org/10.1145/3177102.3177115.
The mobile games market is nowadays flooded with all types of single player and turn-based multiplayer games.
Real-time multiplayer games market with its huge potential is dominated by the biggest developers, leaving small and medium studios far behind due to the very high technological cost and entry threshold. High latency, low bandwidth network connections together with relatively high usage costs divert the real-time multiplayer games market to personal computers and gaming consoles. Therefore, more mobile game developers usually chose to create turnbased over real-time multiplayer games. The pace of the game is usually dictated by players reactions, preceded by a decision-making phase thus even high network latency remains unnoticed for users.
Use case analysis
Based on previous experience network latencies in turn-based multiplayer games can reach values up to 500 milliseconds and user experience will remain acceptable, whereas in real-time multiplayer games 100 milliseconds (with client side prediction) and 10-30 milliseconds (without client side prediction) is the upper limit for network latency.
Therefore, any technology that will allow to achieve these lower latency limits will be a milestone to allow more developers enter the real-time multiplayer games market. We predict that within 3-4 years mobile gaming platforms due to high-end technology will dominate the market and 5G access will have a key role.
Expected Benefits
Adopting the ACCORDION platform in mobile gaming could potentially bring about significant benefits. First and foremost, it aims to tackle the high technological cost and entry threshold currently experienced by small and medium studios in the real-time multiplayer games market. This shift would make the market more inclusive and competitive.
By lowering network latency to acceptable levels for real-time gaming, ACCORDION allows for a smoother and more responsive gaming experience, which in turn increases user satisfaction. It would thus help developers shift from turn-based to real-time multiplayer games on mobile platforms.
The improved user experience, combined with the cost-effective nature of the platform, would likely lead to an increase in user base and retention for developers. It would also enable developers to diversify their game portfolios and increase their market presence.
The ACCORDION platform has the potential to improve the real-time multiplayer mobile gaming market by addressing its key challenges and facilitating more developers to tap into its immense potential.
The more the application architectures are “drifting” towards the cloud and the more businesses are driven by high end-user QoE, the more challenging the delivery of the application content becomes. Optimizations in terms of quality, response times and bandwidth utilization are needed across the cloud-client continuum in order to materialize novel applications such as cloud gaming, video streaming, mobile advertising and even localization services. Plexus is well positioned in this type of business with off-the-shelf big data analysis tools and user-context models and a strong commercial interest towards the optimization of end-user QoE , particularly in the content delivery application domains.
Use Case Analysis
Plexus objective is the optimization of the content delivery path from cloud to low-end user devices, especially mobile devices. A typical scenario involves the generation of content at a centralized infrastructure and its streaming to the individual clients based on their own feedback/input. This is a scenario equally applied to cloud gaming and multimedia streaming with an extremely wide impact range in NextGen applications. Plexus seeks to investigate the optimizations that can be achieved at the level of QoE in such applications, by employing cloud gaming engines on a hybrid cloud/edge infrastructure and coupling the deployment of the components with its sophisticated end-user models.
Expected Benefits
Deployment of sandboxed components in the edge/cloud continuum will trigger the need for intelligent deployment decisions that will be based on the respective ACCORDION management framework. These decisions will be informed by the complete user context, that include not only the end-users behavior and preferences in general, but also the capacity of the supporting infrastructures. ML/DL techniques would be explored to validate their ability to learn from the high-volume data and help in compressing video chunks to super low-resolution that are later sent at a most appropriate network and upscaled at the device’s end for an ultra HD gaming experience. The challenge will be with on-the-edge processing while avoiding delays that dedicated applications or gamers are highly sensitive to. To this end, TensorFlow Lite, Apache MXNET and ONNX will be employed so as to test the models efficiency at various levels in the continuum. All of the orchestration will be made within the ACCORDION ecosystem, allowing us to decrease latency between first level nodes and the different sensors, and increasing communication speed between the first level nodes and the end nodes. The platform will receive the information from each terminal through their communication, it will clean the data by rejecting erroneous measures, it will do extensive calculations thanks to AI techniques, and thanks to the use of edge computing capacities in order to offer different outputs almost in real time, whilst it will enable decision making and also will provide a platform with location information and event driven actions based on user location that will enable developers of games and applications gather position of multiples devices and have them available to develop gamification experiences with very low client device requirements and high speed information with very low power consumption.