I-Frame Delay Explained

I-Frame Delay (IFD) is a scheduling technique for adaptive streaming of MPEG video. The idea behind it is that streaming scheduler drops video frames when the transmission buffer is full because of insufficient bandwidth, to reduce the transmitted bit-rate. The characteristics of the algorithm.:^[1]

1. number of frames currently in the buffer (not the number of bytes) is indicating buffer fullness,
2. less important frames (B-frame) from the buffer are dropped before the more important frames (I-frame and P-frame),
3. the transmission of I-frames is delayed when conditions are bad, even if they are out-of-date w.r.t. the display time (they can still be used to decode subsequent interpredicted frames).

I-Frame Delay algorithm

The IFD mechanism is divided on two parts:^[1]

1. as the stream is parsed and packetized into network packets, it is also analyzed and the packets are tagged with a priority number reflecting the frame type (I-frame, P-frame or B-frame). Non-video packets are given a highest priority number, which causes that audio will never be dropped.
2. during transmission, packets are dropped by the IFD scheduler when the bandwidth is insufficient.

The size of the IFD buffer should be big enough to hold a number of frames but minimum required is two frames, one to hold the frame currently being sent (indicated below as ScheduledFrame), and one currently waiting to be sent (indicated as WaitingFrame). Increasing the IFD buffer size could potentially permit a more elaborate prioritization, however it can cause increased latency and memory usage.^[1] The figure below depicts an example of the buffer filling. The numbers represent the priority a packet.

Here the video frames priority numbers are 10 and higher. The packets with priority number 12 belong to the frame scheduled for sending, and the packets with number 11 belong to the waiting frame. On the figure a packet which belongs to the next frame is about to enter the buffer.

As can be seen, it is possible to interleave video packets with non-video packets (audio or system with priority numbers 2 and 0 respectively). When a packet belonging to next frame is about to be written to the IFD buffer and the buffer is full, the IFD scheduler will drop a frame based on the priority assigned earlier. When the network bandwidth is so low that also P-frames need to be dropped, then the GOP (Group of Pictures) is set to be "disturbed" and the rest of the GOP (which depends on the P-frame) is also dropped.

If only B-frames are dropped there should be no distortions in the frame image because there are no subsequent frames depending on them.^[1] The dropping of frames by IFD causes the effect of the video playback being temporarily frozen, the duration of which depends on the number of frames dropped after which the playback resumes from the next frame which got through.^[1] For an IFD implementation with a buffer of the size of two frames the algorithm is shown in figure below.

procedure Enqueue(NextFrame) if DisturbedGOP

True then if NextFrame is type I then # New GOP is encountered DisturbedGOP = False # Reset disturbed GOP flag end end if DisturbedGOP

True then Drop NextFrame # Discard rest of disturbed GOP return end if WaitingFrame is empty then WaitingFrame = NextFrame else if NextFrame is type I then WaitingFrame = NextFrame else if NextFrame is type B then Drop NextFrame else if WaitingFrame is type I or P then Drop NextFrame if NextFrame is type P then # Discarded frame is P-frame DisturbedGOP = True # Set disturbed GOP flag end else WaitingFrame = NextFrame end end end end end

Notes and References

1. Marek Burza, Jeffrey Kang, Peter van der Stok; Adaptive Streaming of MPEG-based Audio/Video Content over Wireless Networks; Journal of Multimedia vol. 2, no. 2, April 2007;