In Neal Stephenson’s 1992 science fiction novel Snow Crash, there is a digital world where people’s avatars can live, which is perhaps the origin of the “meta-universe” concept that has become so popular in recent years. The movie “Top Gun” released in 2018 brought a more concrete image of the future meta-universe to the world. With the rapid development of VR, AR, AI, 5G and other key metaverse technologies in recent years, the metaverse that used to exist only in the imagination is approaching people’s lives.
Among them, as the window of interaction between the metaverse and people, rendering has become the top priority among many key metaverse technologies, which is likened to the crown of the metaverse technology stack. How to render realistic images on portable terminals quickly and efficiently has become a pressing problem to be solved in the implementation of the metaverse.
ICARUS is a custom chip based on the NeRF rendering algorithm. After loading the chip with trained NeRF network parameters, ICARUS can quickly render the image under that viewpoint and location by simply inputting the observation location and view angle.
ICARUS is a customized hardware acceleration chip based on NeRF (Neural Radiance Fields) algorithm technology, which makes it possible to render extremely delicate and realistic images in real-time on cell phones, VR glasses and other terminals. becomes possible.
“Real-time photorealistic image rendering on terminal devices using traditional rendering techniques is almost impossible to achieve, but ICARUS makes the previously impossible task a reality based on the latest imaging rendering technology, coupled with its customized hardware optimization,” said Lou Xin, assistant professor and researcher at the School of Information Science and Technology of Shanghai University. The ICARUS-related paper has now been accepted by ACM SIGGRAPH ASIA 2022, the top conference on computer graphics.
ICARUS integrates Positional Encoding, MLP Engine, and Volume Rendering Unit to support a variety of NeRF and SLF-like networks. The chip uses configurable fixed-point operations internally and uses a multiplicative approximation algorithm in the MLP module. With sufficient computational accuracy, ICARUS can be used to render different scenes and greatly reduce the area and power consumption of the chip.
NeRF technology: simplifying the process of traditional rendering technology
In the past, hyper-realistic rendering techniques were mostly used in the production of top-notch science fiction movies and some large 3A games, which mostly used traditional rendering techniques of ray tracing. The ray tracing algorithm usually requires the creation of a large number of virtual models and the debugging and calibration of materials, textures and other related optical parameters. Based on the high-precision model, the simulation is iterated according to the laws of physics, and the final image can be faked so that the audience does not notice the difference between the virtual object and the real object.
However, the use of traditional ray-tracing rendering technology alone does not allow us to embrace the metaverse. Rao Chaolin, a PhD student at the School of Information Science and Technology of the University of Shanghai and the first author of the paper, said that excluding the complex rendering process that cannot be avoided in traditional rendering techniques, the huge amount of arithmetic power and power consumption limit the application scenarios of traditional image rendering techniques, making it impossible to be widely used in end devices that can truly access the metaverse. “Currently, high-quality images can only be obtained through the most advanced graphics card rendering, which means huge device size and extremely high computing power consumption, making cell phones, VR glasses and other portable devices do not have access to these realistic images. And based on traditional image rendering techniques, rendering an image usually takes anywhere from a few seconds to several hours depending on the fineness of the model, and the goal of real-time rendering remains out of reach.”
Neural Radiation Field (NeRF) is a new image rendering technology that has emerged in recent years. Its essence is to implicitly encode the entire object space using neural networks and obtain a specific model, which can be used to reconstruct a 3D scene and render images from various new perspectives.
NeRF technology can model real scenes and objects after scanning to achieve the purpose of fast mapping and fast rendering, which greatly simplifies the process of manual modeling compared to traditional rendering technology. It has become the new direction of metaverse development and may become the core technology of next-generation image rendering.
ICARUS can render images with high fidelity using very low power consumption. The figure shows the comparison of ICARUS rendering results with the original image and GPU rendering image results. It can be found that ICARUS has a PSNR (a coefficient used to measure image similarity) similar to GPU rendering in most scenarios and achieves rendering results comparable to it in terms of visual perception.
ICARUS: a hardware acceleration chip customized for the NeRF algorithm
The advancement of NeRF at the algorithm level alone is far from enough in the face of the high demands of the metaverse on image rendering. There is a large amount of fully connected layer computation in the NeRF algorithm, which requires extremely high hardware computing power, and past image rendering hardware is not directly adaptable to new algorithms like NeRF. “Throughout the history of computer development, software and hardware advances have always gone hand in hand, and now that a new generation of image rendering technology has emerged, it inevitably requires a new generation of hardware devices to adapt to it. And ICARUS is also proposed at such a point in time.” Lou Xin said.
In ICARUS chip, the position encoding module (Positional Encoding Unit), multi-layer perceptron module (MLP Engine) and volume rendering unit (Volume Rendering Unit) are integrated to realize the whole process of NeRF rendering on-chip. In order to achieve a high energy efficiency ratio in a small chip area, many techniques to improve the energy efficiency ratio are applied in the chip design process.
Haochuan Wan, a graduate student in the School of Information Science and Technology of Shanghai University and co-author of the paper, introduced the use of quantized fixed-point models in ICARUS and the hardware design of approximation algorithms such as shift-accumulation in the multilayer perceptron module. These hardware-optimized designs for NeRF can greatly reduce chip area and computational energy consumption with little change in the final rendered image quality and can be applied to various lightweight terminals in people’s daily life in the future. When using the same NeRF network to render an image with a resolution of 800×800, an NVIDIA V100 graphics card running at 1.245GHz takes 27.74 seconds and has a chip area of about 815mm^2 and consumes about 300W, while ICARUS running at 400MHz takes 45.75s according to theoretical calculations, but its chip area is only 16.5mm^2 and the power consumption is only 282.8mW.
The next step, ICARUS will follow the rapid development of NeRF-like algorithms, and more coding and network structures will be integrated into the ICARUS chip in the future to support more kinds of application scenarios. ICARUS will be flowed (pilot production) and lit next year when it will become the first NeRF algorithm for custom hardware acceleration chip.
