Platform

Apache

Python

Linux $($SuSe$)$

MySQL

psyco, a dynamic python$->$C compiler

lighttpd for video instead of Apache

What's Inside?

The Stats

Supports the delivery of over $100$ million videos per day.

Founded $2/2005$

$3/200630$ million video views$/$day

$7/2006100$ million video views$/$day

$2$ sysadmins, $2$ scalability software architects

$2$ feature developers, $2$ network engineers, $1$ DBA

Recipe for handling rapid growth

while $($true$)$

$\{$

identify$\_$and$\_$fix$\_$bottlenecks$()$;

drink$()$;

sleep$()$;

notice$\_$new$\_$bottleneck$()$;

$\}$

This loop runs many times a day.

Web Servers

NetScalar is used for load balancing and caching static content.

Run Apache with mod$\_$fast$\_$cgi.

Requests are routed for handling by a Python application server.

Application server talks to various databases and other informations sources to get all the data and formats the html page.

Can usually scale web tier by adding more machines.

The Python web code is usually NOT the bottleneck, it spends most of its time blocked on RPCs$.$

Python allows rapid flexible development and deployment. This is critical given the competition they face.

Usually less than $100$ ms page service times.

Use psyco, a dynamic python$->$C compiler that uses a JIT compiler approach to optimize inner loops.

For high CPU intensive activities like encryption, they use C extensions.

Some pre$-$generated cached HTML for expensive to render blocks.

Row level caching in the database.

Fully formed Python objects are cached.

Some data are calculated and sent to each application so the values are cached in local memory. This is an underused strategy. The fastest cache is in your application server and it doesn't take much time to send precalculated data to all your servers. Just have an agent that watches for changes, precalculates, and sends.

Video Serving

Costs include bandwidth, hardware, and power consumption.

Each video hosted by a mini$-$cluster. Each video is served by more than one machine.

Using a a cluster means:

$-$ More disks serving content which means more speed.

$-$ Headroom. If a machine goes down others can take over.

$-$ There are online backups.

Servers use the lighttpd web server for video:

$-$ Apache had too much overhead.

$-$ Uses epoll to wait on multiple fds$.$

$-$ Switched from single process to multiple process configuration to handle more connections.

Most popular content is moved to a CDN $($content delivery network$)$:

$-$ CDNs replicate content in multiple places. There's a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.

$-$ CDN machines mostly serve out of memory because the content is so popular there's little thrashing of content into and out of memory.

Less popular content $(1-20$ views per day$)$ uses YouTube servers in various colo sites.

$-$ There's a long tail effect. A video may have a few plays, but lots of videos are being played. Random disks blocks are being accessed.

$-$ Caching doesn't do a lot of good in this scenario, so spending money on more cache may not make sense. This is a very interesting point. If you have a long tail product caching won't always be your performance savior.

$-$ Tune RAID controller and pay attention to other lower level issues to help.

$-$ Tune memory on each machine so there's not too much and not too little.

Serving Video Key Points

Keep it simple and cheap.

Keep a simple network path. Not too many devices between content and users. Routers, switches, and other appliances may not be able to keep up with so much load.

Use commodity hardware. More expensive hardware gets the more expensive everything else gets too $($support contracts$).$ You are also less likely find help on the net.

Use simple common tools. They use most tools build into Linux and layer on top of those.

Handle random seeks well $($SATA$,$ tweaks$).$

Serving Thumbnails

Surprisingly difficult to do efficiently.

There are a like $4$ thumbnails for each video so there are a lot more thumbnails than videos.

Thumbnails are hosted on just a few machines.

Saw problems associated with serving a lot of small objects:

$-$ Lots of disk seeks and problems with inode caches and page caches at OS level.

$-$ Ran into per directory file limit$.$ Ext$3$ in particular. Moved to a more hierarchical structure. Recent improvements in the $2\mathrm{.}6$ kernel may improve Ext$3$ large directory handling up to $100$ times, yet storing lots of files in a file system is still not a good idea.

$-$ A high number of requests$/$sec as web pages can display $60$ thumbnails on page.

$-$ Under such high loads Apache performed badly.

$-$ Used squid $($reverse proxy$)$ in front of Apache. This worked for a while, but as load increased performance eventually decreased. Went from $300$ requests$/$second to $20.$

$-$ Tried using lighttpd but with a single threaded it stalled. Run into problems with multiprocesses mode because they would each keep a separate cache.

$-$ With so many images setting up a new machine took over $24$ hours.

$-$ Rebooting machine took $6$ please write summury