Chapter 43.

Video Streaming Quality Index VSQI

43.

Video Streaming Quality Index

VSQI

The information element VSQI (Video Streaming Quality Index) estimates the
viewer-perceived video and audio quality achieved during video streaming
sessions. This chapter takes a look the VSQI algorithm.
See also the document Video Streaming Quality Measurement with VSQI
which is included in the TEMS Investigation documentation package.
Compare chapter 44 on MTQI.

43.1.

General Properties of VSQI

Like SQI (chapter 41), VSQI is a no-reference method which judges the
quality of the received signal on its own merits, without knowledge of the
original.
The kind of subjective test which VSQI strives to imitate is one where viewers
are instructed to assess both video and audio and combine their perception
of each into an overall multimedia quality score.
The output from the VSQI algorithm is expressed as a value between 1
and 5, conforming to the MOS (Mean Opinion Score) scale which is
frequently used in subjective quality tests. The unit for VSQI is called MOSVSQI.

43.2.

What VSQI Is Based On

The VSQI score is based on the following non-perceptual input:

1 The quality of the encoded (compressed) signal prior to transmission.
This quality is straightforwardly a function of the video and audio codecs
used, and their bit rates. The information actually used by the VSQI
algorithm is the video codec type and the total (video + audio) bit rate.
The clean quality has been computed in advance for the codecs listed
in section 43.4.1.

NT13-22100 ver 1.0

UM-457

TEMS Investigation 15.2 Users Manual

2 The amount of initial delay and the subsequent interruptions during

playback of the video sequence: that is, the time required for initial
buffering and the incidence of rebuffering.
3 The amount of packet loss at the application level (i.e. in the video
streaming client).

43.3.

What VSQI Does Not Consider

VSQI does not directly consider the signal presented to the human viewer;
that is, no analysis of perceptual input is performed to detect specific visible
artifacts. The transferred video is not analyzed frame by frame in any way.
Thanks to the monitoring of packet loss (item no. 2 in section 43.2 above),
however, even slight problems with blockiness, jitter, and so on will still be
noticed by the algorithm and affect the VSQI score.

43.4.

Static and Dynamic VSQI

Two versions of the VSQI algorithm have been devised: one static and one
dynamic version.
Static VSQI is presented in the event Streaming Quality VSQI. It does not
appear as an information element. Dynamic VSQI, on the other hand, is
contained in the information element Streaming VSQI.

43.4.1.

Static VSQI

The static version of VSQI takes an entire streamed video clip as input and
assigns a single quality score to it.
Input parameters to the static version of VSQI are as follows:

Video codec used (H.263, H.264, or MPEG4)

Total bit rate (video + audio)

Duration of initial buffering

Number of rebuffering periods

Duration of rebuffering periods

Amount of packet loss

With some degree of simplification, we may describe the calculation of static

VSQI with the following formula:

UM-458

Chapter 43. Video Streaming Quality Index VSQI

Here, VSQIclean is the clean value obtained for the clip prior to transmission.
This score is determined by the quality of the encoding, which is in turn
dependent on the choice of codecs and bit rate.
The size of the buffering penalty depends on the time taken for initial
buffering, the time spent rebuffering, and the number of rebuffering events.
The size of the packet loss penalty is determined as follows. A running packet
loss average over the last 4 s is computed approximately every second, and
the values thus obtained are weighted and summed to yield an appropriate
overall measure of the packet loss. The latter is then translated into a
deduction from the VSQI score.
The static VSQI algorithm has been fine-tuned for clips of around 30 s and
should therefore in practical use be applied to clips of similar duration. The
video sequences must not be too short because of how the buffering works:
each instance of rebuffering takes several seconds to complete, and
moreover if the clip is short enough it will have been buffered in its entirety
before the replay starts, so that no rebuffering will ever occur. For clips
considerably longer than 30 s, on the other hand, disturbances towards the
end will be more harshly penalized by viewers than those occurring early on,
simply because the late ones are remembered more vividly. Therefore, since
the current VSQI algorithm does not take into account such memory effects, it
would probably perform slightly worse for long clips. (The dynamic version of
VSQI naturally is not affected by this limitation.)

43.4.2.

Dynamic (Realtime) VSQI

The dynamic or realtime version of VSQI estimates the quality of a streaming

video clip as perceived by viewers at a moment in time. It is updated regularly
at intervals of the order of 1 s while the video clip is playing. Each VSQI
output value is dependent on the recent history of the streaming session (i.e.
recent packet loss levels and possible recent buffering events).
The design of dynamic VSQI is based on the following:

Previous research suggesting approximate times taken for the perceived

quality to drop to MOS-VSQI 1 (during buffering) and to rise to the highest
attainable VSQI (during normal replay)

Modeling of the impact of packet loss on perceived quality

NT13-22100 ver 1.0

UM-459

TEMS Investigation 15.2 Users Manual

Tailoring of mathematical functions for expressing viewer annoyance/

satisfaction as a function of time (in each of the states that are possible
during replay)

Codec and bit rate parameters as in the static version

The graph below shows in rough outline the different ways in which dynamic
VSQI can evolve during the replay of a streaming video clip. The best
achievable quality, i.e. the ceiling in the graph, is dependent on the codec/bit
rate combination but is also affected by the amount of packet loss. In this
example the packet loss is assumed to be constant so that the influence of
buffering can be clearly discerned.

1 The user tolerates (and might even expect) a certain amount of initial
delay; but the longer the buffering drags on, the more the user loses
patience.
2 Once the replay gets going, the perceived quality picks up again and
soon approaches the highest achievable level.
3 If rebuffering occurs, VSQI deteriorates rapidly. Rebuffering events are
much less tolerated by viewers than initial buffering, especially if
repeated; VSQI captures the latter by making the slope of the curve
steeper for each new rebuffering event.
4 After the replay has recommenced, VSQI recovers reasonably quickly,
but not infrequently from a rock bottom level.

UM-460

Chapter 44. Mobile TV Quality Index MTQI

44.

Mobile TV Quality Index

MTQI

MTQI (Mobile TV Quality Index) is a refinement of the video part of the VSQI
quality measure (see chapter 43). Unlike VSQI, MTQI does not include an
assessment of audio quality.
The MTQI algorithm can be concisely characterized as follows.

Algorithm components:

Modeling of clean quality

Modeling of packet loss

Modeling of corruption duration (total duration of corrupted frames)

Buffering with and buffering without skipping are distinguished. Buffering

with skipping means that frames are skipped in connection with buffering;
no skipping means that every frame is replayed.

Supported video codecs: H.263, H.264, REAL, MPEG4

Supported video formats: QCIF, QVGA

NT13-22100 ver 1.0

UM-461

TEMS Investigation 15.2 Users Manual

45.

VQmon Video and Audio

Quality Metrics

This chapter deals with certain aspects of VQmon video and audio perceptual
quality metrics, which are computed in the course of HTTP streaming. The
metrics are listed in Information Elements and Events, section 3.8: Streaming
IEs; for the service testing setup, see section 12.16.6.4 of the present
document.
The VQmon algorithms have been developed by Telchemy, and the
information that follows is taken from Telchemy documentation.

45.1.

VQmon Mean Opinion Scores (MOS)

VQmon provides a set of Mean Opinion Scores (MOS) estimating the quality
of each video and audio stream as perceived by end-users. Each MOS value
ranges from 1 to 5, where 1 represents the verdict Unacceptable and 5
means Excellent:

MOS-V: Video MOS, considering the effects of the video codec, frame
rate, packet loss distribution, and group-of-pictures structure on video
quality.

MOS-A: Audio MOS, considering the effects of the audio codec, bit rate,
sample rate, and packet loss on viewing quality.

MOS-AV: AudioVideo MOS, considering the effects of both picture and

audio quality as well as audiovideo synchronization on the overall user
experience.

45.1.1.

Absolute and Relative MOS-V

When comparing MOS values, it is important to consider that some types of

video inherently produce a higher level of quality than others. Relying solely
on absolute MOS values can be misleading when comparing dissimilar types
of video service, as viewers tend to form expectations of quality based in part
on the perceived capabilities of the medium.

UM-462

Chapter 45. VQmon Video and Audio Quality Metrics

For example, a video viewed on a handset with a small screen might receive
an absolute MOS of 3.1 when little or no quality degradation is evident, while
for a handset with a larger screen, the same MOS value might suggest that
some noticeable impairments were present.
To facilitate quality comparisons between different video service types,
VQmon provides both Absolute and Relative MOS-V:

Absolute MOS-V considers the impact of frame resolution, frame rate,

codec, compression level, transmission impairments, and frame loss
concealment on video quality.

Relative MOS-V considers the impact of all of the factors used to

determine Absolute MOS-V except frame resolution, producing a MOS
relative to the ideal for the current video format.

All VQmon MOS scores are reported as separate instantaneous, minimum,

maximum, and average values, the last three spanning the current streaming
session.

45.2.

Video Service Transmission Quality (VSTQ)

The VQmon metric Video Service Transmission Quality (VSTQ) is a codecindependent score that measures the ability of the network to reliably
transport video. VSTQ is expressed in the range 050.

NT13-22100 ver 1.0

UM-463