Shoaib Ali It's bloggin time

This is what i've spent my sunday morning hacking around.

I seem to watch content over and over again on youtube and other websites. This was costing me a lot of bandwidth so, I decided to use my Squid proxy server to cache it.

The requirements are very simple. Had to do a bit of tinkering around to get a basic setup working.

1. Squid >= 2.6
2. Python >= 2.4
3. Python-urlgrabber (urlgrabber module of python)
4. Python-iniparse (iniparse module of python)
5. Apache (httpd) Web Server

You can find those details on http://cachevideos.com/installation

I did mine with lighttpd. Infact you can you use any webserver as long as it can serve files over HTTP.

On Debian

You can install Python-urlgrabber using apt-get. The Python-iniparser you can download of here http://code.google.com/p/iniparse/

Extract it and run the following


cd python-iniparser
python setup.py install


Basically all that fancy setup.py script does it copy files over to /var/lib/python2.4/ directory. So something is really bothering you then you can take things in your own hands.

After that extract the videocache archive. Get it from here http://cachevideos.com/sites/default/files/pub/videocache

tar -zxvf /videocache-1.6.tar.gz
cd /videocache-1.6
python setup.py install


Once again, all that does is this :-

1) Copy the template folders in to /var/spool/videocache. This directory contains other directories for youtube, google, meta-cafe, redtube etc etc..
2) Copy the configuration file over to /etc/videocache.conf
3) Copy the real deal in /usr/share/videocache/. This is the directory with the python script that does all the magic! So feel free to hack it like I did ;)
4) Creates the log file under /var/videocache/videocache.log
5) Lastly it ensures the appropriate permissions are applied to the directories it created i.e whatever user Squid runs under sometimes it runs as squid, proxy so find out using ps -axu | grep squid

If you feel annoyed by the setup.py go ahead hack it away and do it manually!

Note: You will need to open up setup.py and change a few things like

# The user which runs the squid proxy server or daemon. Change this according to your system.
squid_user = 'proxy'
# The group which runs the squid proxy server or daemon. Change this according to your system.
squid_group = 'proxy'
# The location of videocache installation directory. You don't need to change this.
install_dir = '/usr/share/'
# The directory to store application specific configuration files for Apache Web Server. Change this according to your system.
# For Red Hat and derivatives, you don't need to change this.
# For Debian and derivatives its normally /etc/apache2/conf.d/
#apache_conf_dir = '/etc/httpd/conf.d/'
#apache_conf_dir = '/etc/lighttpd/'
apache_conf_dir = '/etc/apache2/conf.d/'

Now that is install all you need to do is tell the Squid proxy to use the helper program videocache.pl with some other extra options.


# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 10
acl youtube_query url_regex -i \.youtube\.com\/get_video
acl youtube_query url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/videoplayback
acl youtube_query url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/get_video
acl youtube_deny url_regex -i http:\/\/[a-z][a-z]\.youtube\.com
acl metacafe_query dstdomain v.mccont.com
acl dailymotion_query url_regex -i proxy\-[0-9][0-9]\.dailymotion\.com\/
acl google_query dstdomain vp.video.google.com
acl redtube_query dstdomain dl.redtube.com
acl xtube_query url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl vimeo_query url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl wrzuta_query url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
url_rewrite_access deny youtube_deny
url_rewrite_access allow youtube_query
url_rewrite_access allow metacafe_query
url_rewrite_access allow dailymotion_query
url_rewrite_access allow google_query
url_rewrite_access allow redtube_query
url_rewrite_access allow xtube_query
url_rewrite_access allow vimeo_query
url_rewrite_access allow wrzuta_query
redirector_bypass on
# --END-- videocache config for squid

Once that's added to the end of /etc/squid/squid.conf you can restart squid.

By default squid runs on 3128 port. So, on your firefox change the proxy settings to point to your squid server with that port. Also make sure you take away the bit that says "Don't use proxy for the following address localhost, 127.0.0.1" remove that! This is where your cached content will be served from.

Now go to youtube.com and in the background run the following command


tail -f /var/log/videocache/videocache.log


You should see results similar to this

DOWNLOAD YOUTUBE 10167389 Video was downloaded and cached.
CACHE_SERVE YOUTUBE Video was served from cache.

Next time you reload the same video it will send it from cache and won't cost you any bandwidth! In my next blog post I'll post my findings about ipcop transparent proxy and running the videocache script on ipcop.