YouTUBE Caching using Squid as a Transparent Proxy server

Umair Junaid DATED : 09 September 2012

In this tutorial we will be creating a youtube caching server using squid and apache server
We will be using squid proxy server and apache php web server to redirect and store youtubevideos.

We will need

Server Machine with two NICs

eth0 (192.168.1.111, 255.255.255.0, gw 192.168.1.1) (eth0 WAN interface ) eth1 (192.168.0.254, 255.255.255.0) ( eth1 LAN interface for local clients) Centos base OS installation ver 5

LAB Setup
STEP 1 (SQUID SERVER) eth0 192.168.1.111
Squid version used squid-3.1.8-1.el5.i386.rpm (at the time of writing it is the latest rpm it can be downloaded from squid website I have not tested on older versions and cannot say they will work or not).

wget http://people.redhat.com/jskala/squid/squid-3.1.8-1.el5/x86_64/squid3.1.8-1.el5.x86_64.rpm rpm -ivh squid-3.1.8-1.el5.x86_64.rpm Step 2 (Apache Server with PHP support) eth0 192.168.1.111

NOTE: PREFRABBLY APACHE AND SQUID CAN BE ON DIFFERENT MACHINES THEN YOU HAVE TO CHANGE THE REDIRECTION IP IN PER.PHP TO THE IP OF APACHE SERVER MACHINE.
rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm yum install mysql mysql-bench mysql-server mysql-devel mysqlclient10 php-mysql httpd gcc pcre-devel php-gd gd mod_ssl glib2-devel gcc-c++ libpcap-devel php php-pear Note: we will not use mysql directly but some of the libraries will be required for the program so that why we are installing these.

chkconfig httpd on service httpd start

STEP 3 SQUID.CONF
Then replace squid.conf with the attached squid.conf add place phpredir.php in /etc/squid/ with permissions 770 owner root and group squid = phpredir.php

STEP 4 APACHE ADDITIONS

Then add per.php in /var/www/html/ add a videos folder in /var/www/html with permissions drwxrwxrwx 2 root root videos at the time youtube caching is being done only i am still looking to make metacafe and blip.tv to work and some other after that if possible

STEP 5 WIN XP CLIENT SETTINGS

User Client Settings Ip address 192.168.0.___ Netmask 255.255.255.0 Gateway 192.168.0.254 For Now as Youtube caching is done with the help of apache server Squid access logs will show TCP_MISS when accessing cached videos but dont worry just ignore this warning You will be able to verify the youtube caching is working properly by replaying already played video in the clients browser. And no functionality exists for deleting videos that are not accessed often. If someone can guide in this regard it will be very helpful indeed.

per.php
<?php $file_path="/var/www/html/videos"; $logfile="$file_path/cache.log"; $url=urldecode($_GET['url']); $urlptr=fopen($_GET['url'],"r"); $blocksize=32*1024; //attempt to get. a 404 shouldn't happen, but... if($urlptr===FALSE){

header("Status: 404 Not Found"); die(); } //echo $_GET['url']; //find content type and length foreach($http_response_header as $line){ //echo $line; if(substr_compare($line,'Content-Type',0,12,true)==0) $content_type=$line; else if(substr_compare($line,'ContentLength',0,14,true)==0){ $content_length=$line; } } /**Youtube will detect if requests are coming form the wrong ip (ie, if only video requests are redirected, so, we must redirect all requests to youtube. As such, we must capture all requests t youtube. Most are unimportant, so we can pass them straight through **/ if(!preg_match("@.*youtube.*videoplayback.*@",$url)){ fpassthru($urlptr); fclose($urlptr); exit(0); } //send content type and length header($content_type); header($content_length); //find youtube id; $url_exploded=explode('&',$url); $id=""; foreach($url_exploded as $line){ if(substr($line,0,3)==='id=') $id=substr($line,3); } //Get the supposed file size $length=intval(substr($content_length,16)); file_put_contents($logfile,"\nFound id=$id, content-type: $content_type content-length=$content_length\n",FILE_APPEND); //Do we have it? delivar if we do $fname="$file_path/$id-$length"; //Check if we have the file, and it is the correct size. incorrect size implies corruption //print ($fname); //echo $fname; //echo $length;

//echo filesize($fname); if(file_exists($fname) &&filesize($fname)==$length){ readfile($fname); // echo $fname; logdata("HIT",$url,$fname); exit(0); } //file not in cache? Get it, send it & save it logdata("What the hell",$url,$fname); $fileptr=fopen($fname,"w"); //no validity check, simply don't write the file if we can't open it. prevents noticeable failure/ while(!feof($urlptr)){ $line=fread($urlptr,$blocksize); echo $line; if($fileptr) fwrite($fileptr,$line); } fclose($urlptr); if($fileptr) fclose($fileptr); function logdata($type,$what, $fname){ $file_path="/var/www/html/videos"; $logfile="$file_path/cache.log"; $line="@ ".time()."Cache $type url: $what file: $fname client:".$_SERVER['REMOTE_ADDR']."\n"; file_put_contents($logfile,$line,FILE_APPEND); } ?>

phpredir.php
#!/usr/bin/php <?php while ( $input = fgets(STDIN) ) { // Split the output (space delimited) from squid into an array. $input=explode(" ",$input); if(preg_match("@youtube@",$input[0])){ $input[0]=urlencode($input[0]); $input= implode(" ",$input); echo "http://192.168.1.111/per.php?url=$input"; //URL of my web server // change this ip as per apache server }else echo ""; // empty line means no re-write by Squid. }

squid.conf
# # Recommended minimum configuration: # acl manager proto cache_object acl localhost src 127.0.0.1/32 ::1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1 # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/24 # my internal networks acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT visible_hostname squidpxy.umair.com # # Recommended minimum Access Permission configuration: # # Only allow cachemgr access from localhost http_access allow manager localhost #http_access deny manager # Deny requests to certain unsafe ports http_access deny !Safe_ports # Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports # We strongly recommend the following be uncommented to protect innocent # web applications running on the proxy server who think the only # one who can access services on "localhost" is a local user

#http_access deny to_localhost # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS # # Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost # And finally deny all other access to this proxy http_access deny all # Squid normally listens to port 3128 http_port 3128 transparent acl youtube dstdomain .youtube.com .googlevideo.com cache deny youtube acl youtubeip dst 74.125.15.0/24 cache deny youtubeip acl youtubeip2 dst 74.125.236.0/24 cache deny youtubeip2 maximum_object_size 13107200 KB maximum_object_size_in_memory 102400 KB cache_mem 100 MB # pass requests url_rewrite_program /etc/squid/phpredir.php url_rewrite_access allow youtube # leave caching up to the local web server cache deny youtube cache deny youtubeip cache deny youtubeip2 # We recommend you to use at least the following line. hierarchy_stoplist cgi-bin ? # Uncomment and adjust the following to add a disk cache directory. #cache_dir ufs /var/spool/squid 100 16 256 # Leave coredumps in the first cache dir coredump_dir /var/spool/squid # Add any of your own refresh_pattern refresh_pattern ^ftp: 1440 20% refresh_pattern ^gopher: 1440 0% refresh_pattern -i (/cgi-bin/|\?) 0 refresh_pattern . 0 20% entries above these. 10080 1440 0% 0 4320

STEP 6 IPTABLES RULES

Configure the iptables as shown below. And it will make your server act as a router for clients. Add the following rules in vi /etc/rc.local to make squid act as transparent proxy server on startup. iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE iptables -A FORWARD -i eth1 -j ACCEPT echo 1 > /proc/sys/net/ipv4/ip_forward iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128

Note : if you have trouble having to view youtube then try http://www.youtube.com instead of https://www.youtube.com

You can always contact me on umairjunaid137@gmail.com This tutorial was inspired and taken from http://wiki.squidcache.org/ConfigExamples/DynamicContent/YouTube