Pensieve model - WEBRTC, Cheat Sheet of Computer Science

Download Pensieve model - WEBRTC and more Cheat Sheet Computer Science in PDF only on Docsity!

Neural Adaptive Video Streaming

with Pensieve

Hongzi Mao

Ravi Netravali Mohammad Alizadeh

https://gigaom.com/2012/11/09/online-viewers-start-leaving-if-video-doesnt-play-in-2-seconds-says-study/ Video: La Luna (Pixar 2011)

Users start leaving if video doesn’t play in 2 seconds

2

Why is ABR Challenging?

Throughput

Video bitrate

Network throughput is variable & uncertain Conflicting QoE goals

• (^) Bitrate
• (^) Rebuffering time
• (^) Smoothness Cascading effects of decisions Throughput Bitrate^ (Mbps) Buffer size (sec)

buffer

ABR agent bitrates 240P 480P 720P 1080P network and video measurements

bandwidth

bit rate

720P

1. First network control system using modern “deep” reinforcement learning
2. Delivers 12-25% better QoE, with 10-30% less rebuffering than previous ABR algorithms
3. Tailors ABR decisions for different network conditions in a data-driven way Our Contribution: Pensieve Pensieve learns ABR algorithm automatically through experience

Example: Model Predictive Control

Throughput

Video bitrate

t + T

maximize QoE (t, t + T) subject to system dynamics

t

Problem: Needs accurate throughput model

Conservative Throughput

Prediction

Throughput Bitrate^ (Mbps) Buffer size (sec)

Solution: learn from video streaming sessions

in actual network conditions

Reinforcement Learning Goal: maximize the cumulative reward Agent Environment Observe state Take action Reward

10 How to Train the ABR Agent ABR agent state

Neural Network

240P 480P 720P 1080P policy πθ( s, a ) Take action a next bitrate Observe state s parameter θ estimate from empirical data Training : Collect experience data : trajectory of [state, action, reward]

What Pensieve is good at

• (^) Learn the dynamics directly from experience
• (^) Optimize the high level QoE objective end-to-end
• (^) Extract control rules from raw high-dimensional signals

Pensieve MPC Demo Rebuffering chances of outage

Pensieve

buffer (sec)

MPC

buffer (sec)

Throughput

(mbps)

0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal 0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal 0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal 0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal 0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal 0

1. 5 1

• 1 2 5 8 11 14

C

D

F

Average QoE Buffer-based Rate-based BOLA MPC robustMPC Pensieve offline optimal Trace-driven Evaluation

• (^) Dataset: Two datasets, each dataset consists of 1000 traces, each trace 320 seconds.
• (^) Video: 193 seconds. encoded at bitrates: {300, 750, 1200, 1850, 2850, 4300} kbps.
• (^) Video player: Google Chrome browser Video server: Apache server

Norway 3G cellular dataset FCC broadband dataset

bett er bett er Pensieve improves the best previous scheme by 12-25% and is within 9-14% of the offline optimal 14

Does Pensieve Generalize? 3G network trace

• (^) Trace generated from a Hidden Markov model
• (^) Covers a wide range of average throughput and network variation Synthetic trace

Does Pensieve Generalize?

• 1 2 5 8 11 14 C D F Average QoE robustMPC Pensieve (synthetic) Pensieve

• 1 2 5 8 11 14 C D F Average QoE robustMPC Pensieve (synthetic) Pensieve Train on synthetic traces then test on real 3G network trace Only 5% degradation compared with Pensieve trained on real network trace bett er

1. Build a fast experimentation/simulation platform
2. Data diversity is more important than “accuracy”
3. Think carefully about controller state space (observation signals)
   • (^) Too large a state space ⟶slow & difficult learning
   • (^) Too small a state space ⟶loss of information
   • ⟶ When in doubt, include rather than cut the signal Lessons We Learned Pensieve agent Coarse-grain chunk simulator

• (^) Pensieve uses Reinforcement Learning to generate ABR algorithms
• (^) Pensieve optimizes different network conditions through experience
• (^) Pensieve outperforms existing approaches across a wide range of network environments and QoE preferences
• (^) Policies generated by Pensieve have strong ability to generalize Summary http://web.mit.edu/pensieve/