Mobile 3DTV content delivery
optimization over DVB-H system

Public summaries - Technical reports - Scientific publications

Public summaries

Technical reports

Complete end-to-end 3DTV system over DVB-H
This report describes the Mobile3DTV end-to-end system and the final selection of settings for the encoder, transmitter and receiver parts. The choice of settings has been based on the available hardware and software components as well as on the extensive objective and subjective tests carried out during the project. The end-to-end system supports H.264/MVC in base profile (IPPP, 25fps), and provide error protection at application layer by slicing and at link layer by MPE-FEC. Two burst per stereo channel are assumed. Three TV channels are programmed and the corresponding transport streams are pre-recorded and stored at the play-out for continuous streaming, ODFM modulation and broadcasting of baseband signal. The latter is received by the prototype handheld device, which demodulates, decapsulates and decodes streams to play the decoded video on the integrated auto-stereoscopic display. Decoding and playing of stereo videos of nHD resolution at 30 fps is supported by the decoder. Subjective tests have verified the usability of the device and the TV service of the system. The tests have involved 41 participants in a mobile 3DTV use scenario, including free watching of the available three TV channels at spare time in a cafeteria and participating in semi-structured interviews to give answers about the overall quality, system components, use scenario and future use of 3D handheld devices, services and applications. The usability tests have revealed the interest toward mobile 3DTV technology and persisting problems related with the current display technology.
Atanas Gotchev, Stanislav Stankovic, Dominik Strohmeier, Done Bugdayci, Gozde Bozdagi Akar, Heribert Brust, Nikolay Vladimirov, Technical report D6.8, March 2011

Final Version of Terminal Device
This report describes the design and implementation of the final version of the terminal device. This version integrates the new generation NEC auto-stereoscopic LCD and a new receiver antenna. 3DTV application is developed to demonstrate the features of the device. The software includes additional drivers for NEC LCD and support for streams with corresponding resolution 640x360x2. The device was demonstrated at the ICT Event 2010 in Brussels 27-29 September 2010. It is the final form factor of the mobile device, and this version includes all components of the final system
Nikolai Daskalov, Ivan Pushkarov, Venceslav Petrov, Georgi Roussev, Technical report D6.7, March 2011

Tests of broadcasting encoded stereo-video streams with embedded error resilience features
In this study we compare the results real reception experiments with simulations averaged over 100 trials. We simulate the physical layer of the DVB-H channel using a Simulink model. The simulated channel is Outdoor Residential with max Doppler freq=1 Hz. The coding and transmission parameters are tuned according to a joint source-channel optimization technique developed within the project. In the real experiment, the data is transmitted by a PC-based DVB-H transmission system featuring a DECTEC modulator and received by a PC-connected DVB-H receiver. The Outdoor Residential channel characteristics are modeled by slowly moving the receiver at a pedestrian speed. The results show that the optimized parameters for a given transmission bitrate and channel condition give similar SNRs with the real tests.
Done Bughdayci, Gozde Bozdagy Akar, Technical report D6.6, March 2011

Optimized visualization on portable autostereoscopic displays
There are various optical parameters that influence the quality of a 3D display – for example resolution, number of views, angular visibility of each view, crosstalk, etc. However, not all optical parameters have direct influence on the perceived quality. In this report, we develop a group of algorithms for optimized visualization on 3D displays. We discuss the operation principles of mobile 3D displays and identify the optical parameters which have the biggest influence over the visual quality degradation. We develop measurement methodology, which allows deriving of the important optical parameters of a 3D display. Additionally, we propose a method for estimating the subjective perceptibility of crosstalk in a typical observation condition. We give measurement results and perform comparative analysis for 10 different 3D displays. We discuss a set of artefact mitigation algorithms - two algorithms for point-of- view optimization, and two for visually optimal antialiasing.
Atanas Boev, Jarkko Pekkarinen, Atanas Gotchev, Technical report D5.7, March 2011

3D Video Processing Algorithms - Part II
This report describes novel algorithms developed to enhance the quality of 3D video. It concentrates on deblocking and sharpening of stereo video, filling in disoccluded holes in depth image –based rendering, error concealment at the receiver side for both stereo and view plus depth. In addition, optimized implementations of transform-domain filtering and bilateral filtering are presented. Deblocking and sharpening of stereo video is achieved by non-local transform-domain collaborative filtering, while dissoclusions are handled by non-local mean type of inpainting, proved to be superior to other methods by subjective tests. Two schemes for error concealment of video plus depth, using either complementary frame or complementary motion vectors are presented. For the case of MVC, concealment using complementary motion vectors is studied. Implementations of transform-domain filtering and bilateral filtering at the DSP core of OMAP3 and a GPU by OpenCL are reported.
Lucio Azzari, Olli Suominen, Suomeet Sen, Atanas Gotchev, Done Bugdaici, Gozde Bozdagi Akar, Technical report D5.6, March 2011

Novel Stereo-Video Quality Metric
We present the design of a new objective metric for evaluating the quality of stereo video. The design process has undergone into two stages. First, specific subjective tests were designed and conducted to facilitate the design of the metric. Then, the metric was designed and validated using the results of the tests. Correspondingly, the report is given into two parts. The first part describes the subjective tests and the second part describes the metric design. The results show that the amount of artefacts in the visual content dominances the experienced quality. The depth influences the experienced quality and 3D was valuated favourably to 2D. However, the amount of artefacts is critical for the experienced quality and the acceptance of the visual output. The results also show that videos with two best QP values (25 and 30) are practically the only acceptable options. The new metric was designed with the aim to provide modules for characterizing the 2D-type and 3D-type of distortions. Specifically, a recent 2D metric abbreviated as PSNR- VHS has been selected as the main block to evaluate the 2D type of artefacts, as it proved superior for large classes of distortions. The metric takes into account the contrast sensitivity masking mechanism of the human vision through block-DCT transform-domain masking. The metric was modified to work for smaller blocks (i.e. 4x4) in order to make it more suitable for H.264 encoding applications.
Lina Jin, Atanas Boev, Satu Jumisko-Pyykkö, Tomi Haustola, Atanas Gotchev, Technical report D5.5, March 2011

Results of user-centered quality evaluation experiments and usability tests of prototype
In this report we present our work towards finalization of the user-centered quality of experience evaluation framework. During the standardization activities for our framework, we identified the need for comparisons of OPQ with related methods and evaluation approaches to increase its validity. The first study compares OPQ evaluation done in laboratory and the context of use and shows comparable results in the statistical analysis. The second study introduces the Extended-OPQ approach which allows deriving components of Quality of Experience from a series of OPQ studies. The developed QoE terminology for mobile 3D video has been applied in a second study and the results are compared to OPQ evaluation. The comparison shows again comparable results for OPQ and the descriptive evaluation with fixed vocabulary, but also reveals a need for further work towards an optimized terminology. Altogether, the results of the studies contribute to the validity of OPQ and finalize its methodological development. The second part of the report includes the preparations of a final prototype evaluation for MOBILE3DTV. Although we were not able to conduct the study due to unavailability of the prototype at the planned time, we report in detail the test procedure and the selection of the independent variables to provide a valid research plan.
Dominik Strohmeier, Kristina Kunze, Satu Jumisko-Pyykkö, Technical report D4.5, March 2011

Optimized Methods and Algorithms
In this report, we present the results based on the optimization of the parameters in the coding and transmission chain. For this purpose, first we have conducted more tests to narrow down the number of parameters to be used in the optimization process and point out the most important ones. We have also extended the simulations to see whether a layered structure is beneficial or not, since the use of data partitioning could provide us better robustness under lossy channel conditions. While evaluating the quality of the transmitted video we have used both PSNR and a novel perceptually oriented 3D metric. We also propose a heuristic methodology for modeling the end-to-end rate-distortion (RD) characteristic of a DVH transmission system to optimally select the parameters of the video encoder and the error correction scheme to minimize the overall distortion.
Döne Bugdayci, Gözde Bozdagi Akar, Technical report D3.5, March 2011

Final report on coding algorithms for mobile 3DTV
A low complexity view synthesis algorithm suitable for mobile devices has been developed and implemented. The implemented renderer provides two different modes. The first mode enables very fast processing by rounding disparities to integer values to avoid interpolation. The second mode supports floating point disparities by interpolation at sub pixel positions. For both modes pre-processing filters for the depth and post-processing filters for the rendered view have been implemented.
A method for removing irrelevant information from depth maps in Video plus Depth coding is presented. Irrelevant edges and features in the depth map can be damped while the quality of the rendered view is retained. The processed depth maps can be coded at a reduced rate compared to unaltered data. Coding experiments show gains up to 0.5dB for the rendered view at the same bit rate.
The integration of the PSNR-HVS, to the JMVC Software for Multiview coding is described. A QP dependent correction factor for the Lagrange multiplier has been determined. The modified rate-distortion optimization process leads to gains up to 1.6dB PSNR-HVS using the new video quality metric.
A final summary of the stereo video formats and coding methods evaluated in the Mobile3DTV project is given; results and advancements are pointed out.
Gerhard Tech, Karsten Müller, Philipp Merkle, Heribert Brust, Lina Yin, Technical Report D2.6, February 2011

3rd report of the state of the art of technology and standards
In this report, we overview the RTD activities in the area of 3D video and its delivery to mobile devices for the period of June 2009 – September 2010. The report is structured in three parts: standards, devices and systems, and research. The activities of various standardization bodies relevant to the building blocks of the Mobile 3DTV value chain, are reviewed. We briefly describe the activities of: MPEG – in the area of multi-view and 3D video coding; DVB – in the area of digital video transmission to mobile devices and 3D; we overview also broadcasting standardization activities in Asia and USA. We overview the development of new embedded platforms and auto-stereoscopic displays for their applicability to handheld devices and we describe known experimental 3DTV and mobile 3DTV systems and broadcasting trials. We also briefly describe relevant ongoing research projects in Europe and Asia and review new research papers. In the conclusions, we try to identify the major trends in the overviewed technology and standards.
Atanas Gotchev, Karsten Müller, Done Bugdayci, Dominik Strohmeier, Maija Mikkola, Nikolay Daskalov, Technical report D7.7, October 2010

Fourth Version of Terminal Device
This report describes the design and implementation of the fourth version of the terminal device. In this version, the final form factor of the mobile device has been designed to include all components of the final system. Namely, the processing platform has been designed and coupled to newer version of NEC auto-stereoscopic LCD. New software support for the key project-specific components such as for processing and playing stereo video content, have been developed as well. This includes a new DSP version of the MVC decoder and new drivers for the Android operating system.
In this version, the final form factor of the device has been improved. It contains now the final auto-stereoscopic display and the DVB-H receiver as a chip, both integrated into the platform.
Nikolai Daskalov, Ivan Pushkarov, Venceslav Petrov, Taner Halibryam, Technical report D6.5, July 2010

Report on generation of stereo video data base
Availability of suitable test data is a crucial condition for research and development in video processing and coding. In order to lay the fundaments for further research and development in the project, a data base of stereo video was created. This deliverable gives an overview of the available stereo video material. In total 32 stereo and multiview sequences are available to the project, which is an extremely rich data set. To our knowledge it is the biggest stereo video data set worldwide. For instance, the test data set that is used in MPEG for development of the new 3D video standard is only a subset of our data set. Half of the material is available with full copyright from the server of the project. The available data are listed and illustrated. Main characteristics of the content in terms of motion, colour, texture, light, indoor/outdoor, depth, etc. are provided. Computer animated stereo sequences have been created from 3D models available under creative commons licence using the professional rendering software Blender. In collaboration with WP4 a subset of gathered and new created content has been chosen and declared as the projects test set for subjective evaluations. This document reflects the current status of the stereo video data base. The process of collecting stereo video test data will be ongoing and new content will be added as soon as available.
Aljoscha Smolic, Gerhard Tech and Heribert Brust, Technical report D2.1, July 2010

Modelling of the stereoscopic HVS
The report presents an overview of fields of study, related with modelling of the human visual system and perception of visual information – HVS anatomy and physiology, HVS modelling, and visual quality metrics. State-of-the-art in HVS modelling, including modelling of light adaptation, contrast sensitivity, and spatiotemporal masking is presented. Special attention to the visual mechanisms responsible for stereoscopic perception is paid. The algorithms behind the most popular visual quality metrics are described along with a comparative study of 15 metrics over a database of stereoscopic video streams. Arguments are provided about why 2D metrics fail when applied to 3D quality evaluation. The overview grounds the need of running subjective tests about the relative importance of different 3D visual cues on portable autostereoscopic display and about the influence of varying depth on the overall visual quality. The results of first stage of these studies are presented. Finally, a model of the stereoscopic HVS, suitable for feature based 3D video quality evaluation and the corresponding image processing channel are discussed.
Atanas Boev, Maija Poikela, Atanas Gotchev, and Anil Aksay, Technical report D5.3, July 2010

State of the art of technology and standards
In this report, we overview the RTD activities in the area of mobile 3D video and related fields over the period of May 2008 - June 2009. The review is structured in three parts, as standards, systems and research. We review the activities of various standardization bodies being relevant to the building blocks of the Mobile 3DTV value chain. We briefly describe the activities of: MPEG - in the area of multi-view and 3D video coding; DVB - in the area of digital video transmission to mobile devices; SMPTE - in the area of 3D delivery to the home; ITU-T - in the area of user experience of multimedia content and VQEG - in the area of objective video quality assessment. We overview the development of new embedded platforms and auto-stereoscopic displays for their applicability to handheld devices and we describe known experimental 3DTV and mobile 3DTV systems. We also briefly describe relevant research projects, mainly within FP7, but also in Asia and review new research papers. In the conclusions, we try to identify the major trends in the overviewed technology and standards.
Atanas Gotchev, Karsten Mueller, Gozde Bozdagi Akar, Dominik Strohmeier and Ivan Pushkarov, Technical report D7.5, June 2010

3D Video Processing Algorithms - Part I
This report describes algorithms developed to enhance the quality of 3D video. At the pre-processing side, we have addressed the following scenarios: stereo-video with higher resolution to be downscaled to meet the resolution of mobile 3D display; stereo-video captured at noisy conditions (e.g. user-created content) to be denoised; depth map in the format "view+depth" to be further refined. At the post-processing side we address the problem of dealing with depth map impaired by blocky artifacts resulted from block transform based encoders, such as H.264. For all these cases, we investigate advanced algorithms and present experimental results illustrating their performance.
Sergey Smirnov, Atanas Gotchev, Sumeet Sen, Gerhard Tech, Heribert Brust, Technical report D5.4, February 2010

Results of quality attributes of coding, transmission and their combinations
The report presents the results of two large-scale user studies in which we evaluated the perceived quality of mobile 3D television and video along its production chain. Open Profiling of Quality (OPQ) was applied as research method. OPQ is a combined research method of psychoperceptual evaluation and sensory profiling. It allows connecting user preferences to experienced quality factors. In our studies, we evaluated different coding approaches and transmission parameters in two large-scale studies. The results showed that mobile3DTV can reach acceptable quality when parameters are chosen correctly. Multiview Coding and the Video+Depth approach are the preferred coding methods. Transmission parameters strongly depend on the channel characteristics. The critical quality factor in the perception of mobile3DTV is the level of perceivable artifacts. The application of OPQ showed that added value by depth perception is only given if artifact-free content is perceived. The results of these studies give recommendations for the development of the critical system components.
Dominik Strohmeier, Satu Jumisko-Pyykkö, Kristina Kunze, Gerhard Tech, Döne Bugdayci, Mehmet Oguz Bici, Technical report D4.3, February 2010

Third Version of Terminal Device
This report describes the design and implementation of the third version of the terminal device. It is the final form factor of the mobile device, and this version includes all components of the final system. Namely, the chosen processing platform has been designed and produced in the final form factor and coupled to current version of the auto-stereoscopic LCD. Software support for the key project-specific components such as for processing and playing stereo video content, have been developed as well. From platform perspective the goal of this version is to help the team to validate the integration of the proposed device, the performance of targeted components and the interoperability between them from both HW and SW. The second goal is to verify the system interfaces (again both HW and SW), power management and to enable validation of the performance metrics like quantify the performance of the integrated display, DVB- H stack and developed player.
Nikolai Daskalov, Ivan Pushkarov, Venceslav Petrov, Technical report D6.4, January 2010

Stereo DVB-H broadcasting system with error resilient tools
In this report, we present a complete framework of an end-to-end transport of 3D video over DVB-H. The framework includes both Video + Depth and Stereo representations of 3D Video and corresponding coding methods (either multi-view or simulcast video coding), with or without slice; different error control schemes such as Unequal Error Protection (UEP) where the channel coding ratio among the streams is determined according to the priority level of the streams and Equal Error Protection (EEP), error concealment schemes such as slice and frame concealment. Extensive summations have been performed to find an effective compromise between compression efficiency/MPE-FEC code rates and robustness with respect to typical channel conditions.
Done Bugdayci, M. Oguz Bici, Anil Aksay, Murat Demirtas, Gozde B Akar, Antti Tikanmäki, Atanas Gotchev, Technical report D3.4, December 2009

Development and optimization of coding algorithms for mobile 3DTV
Error resilience tools for H.264/AVC are presented. Slice encoding has been implemented in the H.264/MVC reference software JMVC 5.0.5. An evaluation of the new encoder shows that the additional bit rate needed for the error resilience can be neglected for error free channels. In case of error-prone channel the new slice mode provides sufficient error resilience. The Mixed Resolution Stereo Coding (MRSC) approach has been evaluated. The optimal bit rate distribution between left and down sample right view has been examined. The Advanced Mixed Resolution Stereo Coding (AMRSC) approach has been developed. The three main features of AMRSC are optimized down sampling, interview prediction and view enhancement using unsharp masking. The suitability of sub sampling and low pass filtering together with interview prediction has objectively been evaluated. Further improvements of coding efficiency have been achieved by optimizing the bit rate distribution between the full view and the predicted down-sampled view. AMRSC is compliant with the overall MVC coding strategy in Mobile3DTV. For subjective evaluation of coding methods 96 test stimuli have been generated from the six sequences of the coding test set using Simulcast, Multi View, Mixed Resolution and Video + Depth coding. Two codec profile and a low and a high quality level have been used. 24 test stimuli have been generated for subjective evaluation of transmission approaches using Simulcast, Multi View and Video + Depth coding. Half of the 24 sequences have been coded using the new encoder supporting error resilience methods.
Gerhard Tech, Heribert Brust, Karsten Müller, Anil Aksay, Done Bugdayci, Technical report D2.5, December 2009

Results of the user-centred quality evaluation experiments
This study had two main goals. Firstly, we explored the impact of different evaluation contexts on users' experienced quality. Secondly, we studied the impact of audio and video coding parameters on experienced overall 3D quality. We conducted two extensive quality evaluation experiments in four different contexts of use with several relatively low audio-video coding parameters (bitrate, framerate, presentation mode) and content types. The experiments were conducted on a portable device using parallax barrier display technology and stereovideo coding with each view independently encoded. Our results showed contextual differences in quality requirements between the laboratory and field circumstances. They were influenced by the behaviour of human attention, physical and social distractions, and users actively seeking for good viewing conditions. We recommend using contextual quality evaluation method with hybrid data-collection scheme parallel to controlled laboratory studies to confirm that the optimization of system factors are also meaningful for the field settings. The results of parameter comparisons showed that the current mobile television and near future bitrates (320kbps and 768kbps) provided equally pleasant viewing experience independently on used framerates in 3D mode. The significant increase in the bitrate-framerate resources did not either improve the perceived quality or reach the most pleasantly rated level of 2D quality. 2D viewing was experienced as pleasurable, easy to view and error free compared to 3D viewing and required fewer adjustments in the viewing conditions. These results show strong dependency on the used display technology. To go beyond this limitation, further studies comparing critical system factors should be conducted with at least two display technologies.
Satu Jumisko-Pyykkö, Timo Utriainen, Technical report D4.4, November 2009

State of the art of technology and standards
In this report, we overview the RTD activities in the area of mobile 3D video and related fields over the period of May 2008 - June 2009. The review is structured in three parts, as standards, systems and research. We review the activities of various standardization bodies being relevant to the building blocks of the Mobile 3DTV value chain. We briefly describe the activities of: MPEG - in the area of multi-view and 3D video coding; DVB - in the area of digital video transmission to mobile devices; SMPTE - in the area of 3D delivery to the home; ITU-T - in the area of user experience of multimedia content and VQEG - in the area of objective video quality assessment. We overview the development of new embedded platforms and auto-stereoscopic displays for their applicability to handheld devices and we describe known experimental 3DTV and mobile 3DTV systems. We also briefly describe relevant research projects, mainly within FP7, but also in Asia and review new research papers. In the conclusions, we try to identify the major trends in the overviewed technology and standards.
Atanas Gotchev, Karsten Mueller, Gozde Bozdagi Akar, Dominik Strohmeier and Ivan Pushkarov, Technical report D7.5, June 2010

Report on generation of mixed spatial resolution stereo data base
Availability of suitable test data is a crucial condition for research and development in video processing and coding. In order to provide this fundamental data for further research and development in the project a database of Mixed Resolution data was created. For this, one view was sub-sampled by a factor of two in both dimensions. The resulting sequence was up-sampled again to serve as uncoded reference view. Tests show that the subjective quality of an uncoded Mixed Resolution stereo sequence is impaired compared to the Full Resolution sequence (both views have full resolution), but bitrate can be saved or spent for a higher coding quality. Further tests show that the difference between Mixed Resolution and Full Resolution sequences is reduced when displayed on small screens or large distance. Complexity evaluations demonstrate that it is less complex to decode and up-sample a Mixed Resolution sequence, than decoding a Full Resolution sequence.
Heribert Brust, Gerhard Tech and Karsten Müller, Technical report D2.4, June 2009

Evaluation of the strategies for error resilient transmission of 3D data through a transmitter-channel-receiver pipeline
In this report, we study error-resilient approaches aimed at better error protection of stereo video content to be transmitted over DVB-H. We focus on channel models described in the recent DVB-H implementation guide. For this models, we run extensive simulations to quantify the advantages and disadvantages of different protection schemes for different video bit rates. Our error-protection schemes are based on the standard for DVB-H. DVB-H uses FEC for error protection at physical layer and comes with an optional FEC tool (MPE-FEC) at the link-layer. In our approach, we propose to use the a priori knowledge of the transmitted media and apply MPE-FEC intelligently to provide better robustness. That is, we investigate Unequal Error Protection (UEP) favouring the more important content. Extensive simulation results are given to show the effect of MPE-FEC and UEP on 3D coded video under different channel models, conditions and bitrates.
Gozde B Akar, M. Oguz Bici, Anil Aksay, Done Bugdayci, Antti Tikanmäki, Atanas Gotchev, Technical report D3.3, July 2009

Second version of the terminal device
This report describes the hardware and software design and implementation of the second version of the terminal device. At that stage the goal has been twofold: first, to start developing a novel general framework for integration of all software components needed for the normal functioning of the final device, including the architecture and the resource control, second, to integrate the DVB-H front-end to the development platform. This integration consists of hardware interfacing of the selected DVB-T receiver module and design and implementation of all the required software components. Namely, appropriate OS, Linux SPI driver, driver support for the DVB-T module, DVB-T signal processing software, appropriate player, flexible framework in which all the software components will be implemented. The goal of this device version is to help the team to verify the selected interfaces both SW and HW. The other goal is to evaluate all the SW components, and to do some measurements of the performance for future optimization.
Nikolay Daskalov, Ivan Pushkarov, Nikolay Vladimirov, Technical report D6.3, June 2009

Results of the user-centred quality evaluation experiments
This study had two main goals. Firstly, we explored the impact of different evaluation contexts on users' experienced quality. Secondly, we studied the impact of audio and video coding parameters on experienced overall 3D quality. We conducted two extensive quality evaluation experiments in four different contexts of use with several relatively low audio-video coding parameters (bitrate, framerate, presentation mode) and content types. The experiments were conducted on a portable device using barallax barrier display technology and stereovideo coding with each view independently encoded. Our results based on quantitative evaluations showed some differences in the contextual quality requirements. Our initial analysis indicate that the current mobile television and near future bitrates (320kbps and 768kbps) provided equally pleasant viewing experience independently on used framerates in 3D mode. The significant increase in the bitrate-framerate resources did not either improve the perceived quality or reach the most pleasantly rated level of 2D quality. These results might strongly display dependant. The further analysis needs to complement these results with the analysis of content dependant differences, qualitative quality impressions and gaze patterns to understand the quality and contextual differences.
Satu Jumisko-Pyykkö, Timo Utriainen, Technical report D4.4, June 2009

State of the art of technology and standards
This report reviews the state-of-the-art in the field of mobile 3DTV technologies. We distinguish the following main drivers of that field: standards, systems and research. Standardization activities for representation, coding, transmission, user studies and objective video quality evaluation are reviewed. Commercial DVB-H systems and services are listed, followed by experimental systems for stereo video broadcast over DVB-H. The state of multimedia HW and SW platforms and auto-stereoscopic displays is presented. Relevant research in coding, error resilience, user studies, and video quality enhancement is described. The report makes conclusions about the maturity of different mobile 3DTV technology components and positions the MOBILE3DTV consortium members along the field.
Atanas Gotchev, Karsten Mueller, Gozde Bozdagi Akar, Dominik Strohmeier, Atanas Boev, Antti Tikanmäki, Mikko Oksanen, Technical report D7.4, June 2008

Report on research methodologies for the experiments
Subjective quality evaluation is a widely used method in the optimization of system performance as a part of end-products. Conventionally used psychoperceptual evaluation methods do not relate the evaluation to factors of the potential use of the system or service. It is often desirable to know whether the quality of the components of system meets the user requirements and that the performance of system is not over or under optimized. The major aim of this report is to present a framework for holistic user-centered (audio) visual quality evaluation method. Minor aims of this report are to describe the overall plan for the content of the experiments and continue the work of user-center design for mobile 3DTV by describing the relevant use cases.
Satu Jumisko-Pyykkö, Dominik Strohmeier, Technical report D4.2, November 2008

Mobile stereo video broadcast
In this report, the DVB-H channel is studied for its capabilities to transmit stereo video in different formats and to provide adequate error protection for such content. The following stereo video representations are considered: two channel stereo, encoded either independently with H.264 encoder (simulcast) or by multi-view coding (MVC) and video plus depth both encoded by H.264. These encoding approaches are implemented and combined with an IP streaming application. DVB-H specific tools, such as MPE-FEC encapsulator and decapsulator and physical layer simulator complement the software environment built to run end-to-end video transmission simulations. In the experimental tests, various physical channel models are used to represent different reception conditions. These are put against different stereo video encoding modes and MPE-FEC rates. The preliminary results show the feasibility of the DVB-H channel to handle encoded stereo video streams. They also illustrate the problems of transmission in bad channel conditions, which should be addressed by optimizing the MPE-FEC rates and developing novel application layer error protection schemes.
Gozde B Akar, M. Oguz Bici, Anil Aksay, Antti Tikanmäki, Atanas Gotchev, Technical report D3.2, December 2008

Software for simulation of artefacts and database of impaired videos
In this report we present a framework for simulating stereoscopic artefacts. Within the framework, an arbitrary combination of artefacts can be introduced to a video, with controlled amount of impairment for each artefact. We utilize our taxonomy of artefact built earlier, which follows the typical flow of a mobile 3D video over a DVB-H channel and summarises the artefacts into the following groups: capture artefacts, coding artefacts, conversion artefacts, transmission artefacts and visualization artefacts. We introduce an artefact- simulation channel, in which the artefacts are introduced in an order, natural for the mobile 3D video dataflow - first capture artefacts are added, then coding ones, and so forth. For each artefact included in the framework, we explain the algorithm we use for its simulation. Our framework is modular and could be extended to simulate additional types of artefacts. Any particular modelling block can be also modified or replaced. The framework has been implemented under MATLAB. We provide a quick user guide of the implementation, covering the GUI, installation, basic usage and batch operations over a library of stereoscopic videos.
Atanas Boev, Danilo Hollosi, Atanas Gotchev, Technical report D5.2, December 2008

First version of terminal device
This report describes the design and implementation of the first version of the terminal device. While not having the final form factor of a mobile device, this version includes key components of the final system. Namely, the chosen processing platform has been coupled with two types of auto-stereoscopic LCD. Key SW components for the processing and playing stereo video have been targeted as well. From platform perspective the goal of this version is to help the team to assess the system architecture of the proposed device, the feasibility of implementation, the performance of targeted components and the interoperability between them from both HW and SW point of view. The second goal is to verify the system interfaces (again both HW and SW) and to establish some performance metrics so to quantify the performance of the integrated displays and developed player.
Nikolai Daskalov, Ivan Pushkarov, Angel Gavrailov, Atanas Boev, Technical report D6.2, November 2008

Adaptation and optimization of coding algorithms for mobile 3DTV
As a starting point and reference for the further research, this deliverable specifies and evaluates available coding standards for 3D video for the specific conditions in the Mobile3DTV project. The following stereo video formats and codecs are specified and evaluated: H.264/AVC simulcast, H.264 Stereo SEI message, H.264/MVC, MPEG-C Part 3 using H.264 for both video and depth, H.264 auxiliary picture syntax for video plus depth. Significant coding gains can be achieved with hierarchical B pictures for temporal prediction, but the gain differs largely for individual sequences. On the other hand hierarchical B pictures also mean increased complexity and memory requirements. Inter-view prediction leads to a significant reduction of bitrate for some sequences. However, in other cases the gain is negligible. We achieved up to 35% bitrate savings from inter-view prediction compared to stereo simulcast. Inter-view prediction whether as H.264 SEI or MVC does not add substantial complexity. A representation as video plus depth is an interesting alternative for 3D video. It allows to adjust the stereo rendering at the decoder and to optimally adapt the 3D impression for any given display. However, this extended functionality comes at the cost of an increased complexity. MPEG-C Part 3 is suitable for encoding of video plus depth data. Good depth quality is essential for good overall quality.
Philipp Merkle, Heribert Brust, Kristina Dix, Yongzhe Wang, Aljoscha Smolic, Technical report D2.2, November 2008

Report on generation of video plus depth data base
Availability of suitable test data is a crucial condition for research and development in video processing and coding. In order to lay the fundaments for further research and development in the project, a database of video plus depth data was created. Depth was estimated using Hybrid Recursive Matching (HRM) previously developed at FHG-HHI for the left view of each stereo pair. Then, the corresponding right view was rendered from the left view and the estimated depth. For quality assessment the visual impression was evaluated inspecting synthesized right views and the stereo impression on small displays for mobile applications. Artefacts are visible in the synthesized right views. But due to masking effects with the original left view, they are almost not noticeable when watching stereo on a small display.
Aljoscha Smolic, Yongzhe Wang, Marcus Müller, Peter Kauff, Technical report D2.3, November 2008

Report on generation of stereo video data base
Availability of suitable test data is a crucial condition for research and development in video processing and coding. In order to lay the fundaments for further research and development in the project, a data base of stereo video was created. This deliverable gives an overview of the available stereo video material. In total 28 stereo and multiview sequences are available to the project, which is an extremely rich data set. To our knowledge it is the biggest stereo video data set worldwide. For instance, the test data set that is used in MPEG for development of the new 3D video standard is only a subset of our data set. Half of the material is available with full copyright from the server of the project. The available data are listed and illustrated. Main characteristics of the content in terms of motion, colour, texture, light, indoor/outdoor, depth, etc. are provided. This document reflects the current status of the stereo video data base. The process of collecting stereo video test data will be ongoing and new content will be added as soon as available.
Aljoscha Smolic Technical report D2.1, June 2008

Characteristics of DVB-H typical errors
The goal for this deliverable is to study real and simulated communication channels and different error traces for different channels to be used in simulation of transmitted video content.
Gozde B Akar, M. Oguz Bici, Anil Aksay, Antti Tikanmäki, Mikko Oksanen, Technical report D3.1, June 2008

Requirements and technical specifications of the chosen development platform and DVB-H front-end
This report describes the requirements and technical specifications for selected silicon solution sub-platforms for video encoding/decoding, DVB-H front-end and auto-stereoscopic LCD. The selection process includes the definition of specific requirements, shortlisting the available solutions and final selection of the best fit. The specifications of the selected platforms are delivered and checked against the requirements.
Nikolai Daskalov, Atanas Boev, Technical report D6.1, June 2008

Classification of stereoscopic artefacts
Stereoscopic artefacts which occur during delivery of mobile stereoscopic video are described and categorised. In order to predict how an artefact would be interpreted by the human visual system, the stages of 3D video delivery dataflow (content creation, conversion to the desired format, coding/decoding, transmission, and visualisation) were matched against the different visual subsystems of human (structural, colour, motion and binocular vision). The artefacts are summarized in a table and for each artefact the causes for occurrence, how it affects the 3D vision, and studies referring to it are presented.
A. Boev, D. Hollosi, A. Gotchev, Technical report D5.1, July 2008

Report on user needs and expectations for mobile stereo-video
The report describes user requirements, needs and expectations on mobile 3DTV based on a literature review about user requirements of Mobile TV and 3DTV, and a methodological triangulation of online survey, focus groups and probe study. The results are concluded in first guidelines to design user experience for mobile 3D television and video with respect to three categories of user, system and services, and context.
D. Strohmeier, S. Jumisko-Pyykkö, M. Weitzel, S. Schneider, Technical report D4.1, July 2008

Scientific publications

Source and channel coding recipes for mobile 3D television
A. Gotchev, D. Strohmeier, K. Mueller, G. Bozdagi Akar, V. Petrov "Source and channel coding recipes for mobile 3D television", Proceedings of the 17th International Conference on Digital Signal Processing 2011.

Methodology for design of antialiasing filters for autostereoscopic displays
A. Boev, R. Bregovic, A. Gotchev "Methodology for design of antialiasing filters for autostereoscopic displays", IET Signal Processing.

3D-DCT Based Perceptual Quality Assessment of Stereo Video
L. Jin, A. Boev, A. Gotchev, K. Egiazarian "3D-DCT Based Perceptual Quality Assessment of Stereo Video", IEEE Int. Conf. Image Processing 2011, Sept. 2011, Brussels.

Three-Dimensional Media for Mobile Devices
A. Gotchev, G. B. Akar, T. Gapin, D. Strohmeier, A. Boev "Three-Dimensional Media for Mobile Devices", Proceedings of IEEE, Vol. 99, No. 4, pp. 708-741, April 2011.

Juxtaposition Between Compression and Depth for Stereoscopic Image Quality on Portable Auto-stereoscopic display
S. Jumisko-Pyykkö, T. Haustola, A. Boev, A. Gotchev "Juxtaposition Between Compression and Depth for Stereoscopic Image Quality on Portable Auto-stereoscopic display", Proceedings of 3DTV-CON 2011, Antalya, Turkey.

Evaluation of View Synthesis Algorithms for Mobile3DTV
G. Tech, K. Müller, T. Wiegand "Evaluation of View Synthesis Algorithms for Mobile3DTV", Proceedings of 3DTV-CON 2011, Antalya, Turkey.

The Extended-OPQ method for User-centered Quality of Experience evaluation: A study for mobile 3D video broadcasting over DVB-H
D. Strohmeier "The Extended-OPQ method for User-centered Quality of Experience evaluation: A study for mobile 3D video broadcasting over DVB-H", EURASIP Journal on Image and Video Processing, Special issue "Quality of Multimedia Experience".

3D Video Delivery To Mobile Devices
G. B. Akar "3D Video Delivery To Mobile Devices", 2010 VSF October Meeting Series, IEEE International Symposium, Austin, Texas, USA.

A Modified Non-local Mean Inpainting Technique for Occlusion Filling in Depth-Image Based Rendering
L. Azzari, F. Battisti, A. Gotchev, M. Carli, K. Egiazarian "A Modified Non-local Mean Inpainting Technique for Occlusion Filling in Depth-Image Based Rendering", Proceedings of SPIE, Stereoscopic Displays and Applications, part of Electronic Imaging Symposium, 27-30 January 2011, San Francisco.

Subjective Evaluation of Mobile 3D Content: Depth Range versus Compression Artefacts
S. Jumisko-Pyykkö, T. Haustola, A. Boev, A. Gotchev "Subjective Evaluation of Mobile 3D Content: Depth Range versus Compression Artefacts", Proceedings of SPIE, Multimedia for Mobile Devices, part of Electronic Imaging Symposium, 27-30 January 2011, San Francisco.

Comparative Study of Autostereoscopic Displays for Mobile Devices
A. Boev, A. Gotchev "Comparative Study of Autostereoscopic Displays for Mobile Devices", Proceedings of SPIE, Multimedia for Mobile Devices, part of Electronic Imaging Symposium, 27-30 January 2011, San Francisco.

Mobile 3D Video Broadcast
M. Oguz Bici, D. Bugdayci, A. Gotchev, G. B. Akar "Mobile 3D Video Broadcast", IEEE Int. Conf. Signal Processing (ICIP) 2010, Hong Kong.

MOBILE3DTV - Mobile 3DTV Content Delivery Optimization over DVB-H System
G. B. Akar, A. Gotchev "MOBILE3DTV - Mobile 3DTV Content Delivery Optimization over DVB-H System", SMPTE International Conference on Stereoscopic 3D for Media and Entertainment Program, USA.

Diffusion Filtering Of Depth Maps In Stereo Video Coding
G. Tech, K. Müller, T. Wiegand "Diffusion Filtering Of Depth Maps In Stereo Video Coding", Picture Coding Symposium 2010, Nagoya, Japan.

Profiling Experienced Quality Factors of Audiovisual 3D Perception
D. Strohmeier, S. Jumisko-Pyykkö, U. Reiter "Profiling Experienced Quality Factors of Audiovisual 3D Perception", Proceedings of QoMEX 2010, Trondheim, Norway.

Comparative Analysis of Occlusion-Filling Techniques in Depth Image-Based Rendering for 3D Videos
L. Azzari, F.a Battisti, A. Gotchev "Comparative Analysis of Occlusion-Filling Techniques in Depth Image-Based Rendering for 3D Videos", Proceedings of ACM Multimedia.

Relative Importance of Depth Cues on Portable Autostereoscopic Display
M. Mikkola, A. Boev, A. Gotchev "Relative Importance of Depth Cues on Portable Autostereoscopic Display", Proceedings of ACM Multimedia.

Measuring and modeling per-element angular visibility in multiview Displays
A. Boev, R. Bregovic, A. Gotchev "Measuring and modeling per-element angular visibility in multiview Displays", Journal of the Society of Information Display, Vol 18.

Simulator Sickness – Five Experiments Using Autostereoscopic Mid-Sized Or Small Mobile Screens
S. Jumisko-Pyykkö, T. Utriainen, D. Strohmeier, A. Boev and K. Kunze "Simulator Sickness – Five Experiments Using Autostereoscopic Mid-Sized Or Small Mobile", Proceedings of 3DTV-CON 2010, Tampere, Finland, June 2010.

Experienced Audiovisual Quality For Mobile 3d Television
T. Utriainen and S. Jumisko-Pyykkö "Experienced Audiovisual Quality For Mobile 3d Television", Proceedings of 3DTV-CON 2010, Tampere, Finland, June 2010.

User Directed View Synthesis On Omap Processors
M. Yildiz and G. Akar "User Directed View Synthesis On Omap Processors", Proceedings of 3DTV-CON 2010, Tampere, Finland, June 2010.

Video + Depth based 3D Video Broadcast over DVB-H
D. Bugdayci "Video + Depth based 3D Video Broadcast over DVB-H", Proceedings of IEEE 18th Signal Processing and Communications Applications Conference, Turkey.

Open Profiling of Quality as a mixed method approach to study multimodal experienced quality
D. Strohmeier "Open Profiling of Quality as a mixed method approach to study multimodal experienced quality", 8th European Conference on Interactive TV and Video EuroITV, workshop “Methods for User Studies of Interactive (TV) Technologies” (USIT), 9th-11th June 2010, Tampere, Finland.

On comparing different codec profiles of coding methods for mobile 3D television and video
D. Strohmeier et al. "On comparing different codec profiles of coding methods for mobile 3D television and video", International conference on 3D Systems and Applications (3DSA), 19th – 21st May 2010, Tokyo, Japan.

Open Profiling of Quality – A mixed method approach to understanding multimodal quality perception
D. Strohmeier et al. "Open Profiling of Quality – A mixed method approach to understanding multimodal quality perception", Advances in Multimedia, Volume 2010, Article ID 658980, doi:10.1155/2010/658980

Mixed Resolution Coding with interview prediction for mobile 3DTV
H. Brust, G. Tech, K. Müller and T. Wiegand "Mixed Resolution Coding with interview prediction for mobile 3DTV", 3DTV Conference 2010, Tampere, Finland, June 2010.

User-centered quality of experience of mobile 3DTV: how to evaluate quality in the context of use?
S. Pyykk? et al. "User-centered quality of experience of mobile 3DTV: how to evaluate quality in the context of use?", Proceedings of SPIE vol. 7542, Image Processing Algorithms and Systems VIII at Electronic Imaging 2010, San Jose, California, USA, January 2010.

User-Centered Quality of Experience: Is Mobile 3D Video Good Enough In The Actual Context of Use?
S. Pyykk? et al. "User-Centered Quality of Experience: Is Mobile 3D Video Good Enough In The Actual Context of Use?", VPQM 2010, January 2010.

Comparative analysis of local binocular and trinocular depth estimation approaches
S. Smirnov et al. "Comparative analysis of local binocular and trinocular depth estimation approaches", Proceedings of SPIE vol. 7724, part of Electronic Imaging Symposium 2010, Image Processing Algorithms and Systems VIII at Electronic Imaging 2010, San Jose, California, USA, January 2010.

A Memory-efficient and Time-consistent Filtering of Depth Map Sequences
S. Smirnov et al. "A Memory-efficient and Time-consistent Filtering of Depth Map Sequences", Proceedings of SPIE vol. 7532, Image Processing Algorithms and Systems VIII, part of Electronic Imaging Symposium 2010, Image Processing Algorithms and Systems VIII at Electronic Imaging 2010, San Jose, California, USA, January 2010.

"Sharp, Bright, Three-Dimensional" - Open Profiling Of Quality For Mobile 3DTV Coding Methods
D. Strohmeier, and G. Tech, ""Sharp, Bright, Three-Dimensional" - Open Profiling Of Quality For Mobile 3DTV Coding Methods", Proceedings of "Multimedia on Mobile Devices" as part of the SPIE Electronic Imaging Conference 2010, Multimedia on Mobile Devices at Electronic Imaging 2010, San Jose, California, USA, January 2010.

New, Lively, And Exciting Or Just Artificial, Straining, And Distracting: A Sensory Profiling Approach To Understand Mobile 3D Audiovisual Quality
D. Strohmeier et al. "New, Lively, And Exciting Or Just Artificial, Straining, And Distracting: A Sensory Profiling Approach To Understand Mobile 3D Audiovisual Quality", Proceedings of VPQM 2010, The Fifth International Workshop on Video Processing and Quality Metrics (VPQM 2010), Scottsdale, Arizona, USA, January 2010.

Optimized Visualization Of Stereo Images On An OMAP Platform With Integrated Parallax Barrier Auto-Stereoscopic Display
A. Boev et al. "Optimized Visualization Of Stereo Images On An OMAP Platform With Integrated Parallax Barrier Auto-Stereoscopic Display", 17th European Signal Processing Conference (EUSIPCO-2009), Glasgow, UK, August 2009.

Mobile 3DTV Technology Demonstrator Based On OMAP3430
A. Gotchev et al. "Mobile 3DTV Technology Demonstrator Based On OMAP3430", 16th International Conference on Digital Signal Processing (DSP 2009), Santorini, Greece, July 2009.

DVB-H Üzerinden Stereo Video Yayýný - Stereo Video Broadcast Over DVB-H
M. O. Bici et al. "Stereo Video Broadcast Over DVB-H", SIU 2009, Antalya, Turkey, April 2009. In Turkish.

Framing the Context of Use for Mobile HCI
S. Jumisko-Pyykkö, and T. Vainio, "Framing the Context of Use for Mobile HCI", Review paper about mobile contexts of use between 2000-2007, International Journal of Mobile-Human- Computer-Interaction (IJMHCI), in press.

A Study On The Effect Of MPE-FEC For 3D Video Broadcasting Over DVB-H
A. Aksay et al. "A Study On The Effect Of MPE-FEC For 3D Video Broadcasting Over DVB-H", MobiMedia 2009, London, UK, September 2009.

Intra-Frame Depth Image Compression Based On Anisotropic Partition Scheme And Plane Approximation
N. Ponomarenko et al. "Intra-Frame Depth Image Compression Based On Anisotropic Partition Scheme And Plane Approximation", Immerscom 2009 at University of California, Berkeley, USA, May 2009.

Optimization And Comparison Of Coding Algorithms For Mobile
G. Tech et al. "Optimization And Comparison Of Coding Algorithms For Mobile", 3DTV-Conference, Potsdam, Germany, May 2009.

Mixed Resolution Coding Of Stereoscopic Video For Mobile Devices
H. Brust et al. "Mixed Resolution Coding Of Stereoscopic Video For Mobile Devices", 3DTV-Conference, Potsdam, Germany, May 2009.

Video Plus Depth Compression For Mobile 3D Services
P. Merkle et. al. "Video Plus Depth Compression For Mobile 3D Services", 3DTV-Conference, Potsdam, Germany, May 2009.

Stereo Video Compression For Mobile 3D Services
P. Merkle et al. "Stereo Video Compression For Mobile 3D Services", 3DTV-Conference, Potsdam, Germany, May 2009.

Evaluation of Stereo Video Coding Schemes for Mobile Devices
A. Aksay and G. B. Akar, "Evaluation of Stereo Video Coding Schemes for Mobile Devices", 3DTV-Conference, Potsdam, Germany, May 2009.

Methods for restoration of compressed depth maps: A comparative study
S. Smirnov et al. "Methods for restoration of compressed depth maps: A comparative study", The International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM-09, Scottsdale, Arizona, USA, January 2009.

Stereoscopic artifacts on portable auto-stereoscopic displays: What matters?
A. Boev et al. "Stereoscopic artifacts on portable auto-stereoscopic displays: What matters?", The International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM-09, Scottsdale, Arizona, USA, January 2009.

Use scenarios - mobile 3D television and video
D. Strohmeier et al. "Use scenarios - mobile 3D television and video", special session 'Delivery of 3D Video to Mobile Devices' at the conference 'Multimedia on Mobile Devices', a part of the Electronic Imaging Symposium 2009 in San Jose, California, USA, January 2009.

Mobile 3D television: Development of core technological elements and user-centered evaluation methods toward an optimized system
Atanas Gotchev et al. "Mobile 3D television: Development of core technological elements and user-centered evaluation methods toward an optimized system", special session 'Delivery of 3D Video to Mobile Devices' at the conference 'Multimedia on Mobile Devices', a part of the Electronic Imaging Symposium 2009 in San Jose, California, USA, January 2009.

Classification and simulation of stereoscopic artifacts in mobile 3DTV content
A. Boev et al. "Classification and simulation of stereoscopic artifacts in mobile 3DTV content", Conference "Stereoscopic Displays and Applications", a part of Electronic Imaging Symposium 2009, San Jose, CA, USA, January 2009.

Distributed 3D Dynamic Mesh Coding
M. Oguz Bici and G. Bozdagi Akar, "Distributed 3D Dynamic Mesh Coding", The International Conference on Image Processing (ICIP 2008), San Diego, USA, October 2008.

Designing for User Experience: What to Expect from Mobile 3D TV and Video?
S. Jumisko-Pyykkö et al. "Designing for User Experience: What to Expect from Mobile 3D TV and Video?", uxTV Con 2008, Silicon Valley, USA, October 2008.

Stereo Video Broadcasting Simulation for DVB-H
M. Oguz Bici et al. "Stereo Video Broadcasting Simulation for DVB-H", 2008 NEM Summit, Saint Malo, France, October 2008.

Delivery of 3D Video over DVB-H: Building the Channel
M. Oksanen, A. Tikanmäki, A. Gotchev, and I. Defee, "Delivery of 3D Video over DVB-H: Building the Channel", 2008 NEM Summit, Saint Malo, France, October 2008.

Mobile 3DTV System: Quality and User Perspective
A. Gotchev, S. Jumisko-Pyykkö, A. Boev, D. Strohmeier, "Mobile 3DTV System: Quality and User Perspective", European Workshop of Media Delivery (EUMOB’08), Oulu, Finland, July 2008.

Quality Assessment of 3D Video in Rate Allocation Experiments
A. Tikanmäki, A. Gotchev, A. Smolic, K. Müller, "Quality Assessment of 3D Video in Rate Allocation Experiments", Proc. IEEE International Symposium on Consumer Electronics (ISCE'08), Algarve, Portugal, April 2008.