Deepview: Deep-Learning-Based Users Field of View Selection in 360 degrees Videos for Industrial Environments

Abstract

The industrial demands of immersive videos for virtual reality/augmented reality applications are crescendo, where the video stream provides a choice to the user viewing object of interest with the illusion of "being there." However, in industry 4.0, streaming of such huge-sized video over the network consumes a tremendous amount of bandwidth, where the users are only interested in specific regions of the immersive videos. Furthermore, for delivering full excitement videos and minimizing the bandwidth consumption, the automatic selection of the user's Region of Interest in a 360 degrees video is very challenging because of subjectivity and difference in contentment. To tackle these challenges, we employ two efficient convolutional neural networks for salient object detection and memorability computation in a unified framework to find the most prominent portion of a 360 degrees video. The proposed system is four-fold: 1) preprocessing; 2) intelligent visual interest predictor; 3) final viewport selection; and 4) virtual camera steerer. First, an input 360 degrees video frame is split into three Field of Views (FoVs), each with a viewing angle of 120 degrees. Next, each FoV is passed to the object detection and memorability prediction model for visual interestingness computation. Furthermore, the FoV is supplied as a viewport, containing the most salient and memorable objects. Finally, a virtual camera steerer is designed using enriched salient features from YOLO and LSTM that are forwarded to the dense optical flow to follow the salient object inside the immersive video. Performance evaluation of the proposed system over our own collected data from various Websites as well as on public data sets indicates the effectiveness for diverse categories of 360 degrees videos and helps in the minimization of the bandwidth usage, making it suitable for industry 4.0 applications.