Previous Blog Next Blog
Prev/Next Blog
by date

S3 will revolutionize content distribution

S3 will revolutionize content distribution
Name:Dreamer 
Date Posted:Mar 15, 2006
Rating:4.0 out of 5
Public:YES
Comments:YES
RSS Feed:GarageGames Blog feedor Subscribe with .
Profile Page:View profile page for Dreamer

Blog post
Hello everyone,
I've been evaluating methods for easily and cost effectively pushing updates to clients applications.
My options thus far have been SVN, FTP, Bittorrent, and a whole host of others.

FTP was my original thought, but my own experience with FTP in terms of controlling access, and the amount of load it places on the server has shown that this would be at best a suboptimal solution.

Next in line is Bittorrent, this is a fantastic protocol, basically everyone who wants your content has to share that content with others to recieve more content, while it does save you from placing excess load on the server, again access control is limited, also if you have to push multiple updates/bug fixes you may end up with stale patches and/or incomplete downloads. Furthermore setting up a proper bittorrent tracker or seed has shown to be a little problematic in my experience, and some hosts or even ISP, will even shape your traffic or possibly cut you off if you try to host one.

Thus far SVN has looked to be the natural winner, since each update is really only a diff from the previous revision, thereby reducing bandwidth on the server. And yes SVN can be used for any collection of files.
However controlling access to SVN could be time consuming and/or buggy and if a large number of people all need updates at once, your server will still be swamped.

Recently I read an article on a new service from Amazon called S3.
If you haven't heard about it before, this awesome new service allows you to store unlimited amounts of data for $0.15 per gig per month, and then $0.20 per gig of bandwidth for access to those files.
While at first this may not sound like a great deal, or even of any relevance or importance to you check this out.
You can make the content readable by anyone, or only a select list of people, through an easy to use ACL (access control list), which you can update on the fly.
Furthermore bittorrent is an integral option and using it will drastically reduce the amount of money you spend on bandwidth, since you only spend for actual data transfered and not network overhead.
This essentially allows you to upload any file or set of files, and have automatic bit-torrent capability. Without any of the drawbacks.
But what about the versioning capabilities of SVN?
Well here's the interesting thing, the S3 API descriptions mention that you can associate metadata of up to 4k with the file, and part of this metadata is accessible directly through the filename via a key value pair.
What does all of this mean?

It means that you could upload your data to your S3 storage service and use it for an update and distribution server.
For instance lets say you modified the root animation file player_root.dsq
After it is uploaded you associate some information with it for instance a time stamp.
You would wind up with a file you could easily distribute to any of your customers via any bittorent client, and you would have the added security of an access control list. You would also have the advantage of SVN in that if you later decided you made a mistake in the animation or whatever, you could roll back to the earlier version of the file simply by deleting it.

For me anyways this is great news, and as soon as I launch the MMORPG Enhancement Kit and the newest version of http://www.mydreamrpg.com/ I intend to begin creating tools to facilitate this.

I hope you find this exciting.
Regards,
Dreamer

p.s. For more information on S3 click here

Recent Blog Posts
List:01/18/07 - Shameless Plug
11/29/06 - Infuzor Installer and Autopatcher now shipping!
11/07/06 - We Got A New Look!
11/01/06 - MMORPG Enhancement Kit 1.5 Halloween Release
05/13/06 - MMORPG Kit Full Release Day is Here!
05/12/06 - MMORPG Kit Release is Imminent!
05/01/06 - MMORPG Enhancement Kit Screenies (56k Warning)
03/15/06 - S3 will revolutionize content distribution

Submit ResourceSubmit your own resources!

Phil Carlisle   (Mar 15, 2006 at 13:08 GMT)
Hmm, this doesnt sound as useful as SVN actually. Think about what you said. You can associate metadata with the file sure. But there is no automatic versioning, there is no rollback mechanism, there is no "what is the current version" kind of thing.

I was going to write a blog entry about using SVN for updating customers, as I see this as the quickest route to keeping people synchronised initially.

There *must* be a better option though.

Phil.

Brian Marshall   (Mar 15, 2006 at 14:20 GMT)
My company is currently working on a low-cost SVN-like system that would do a lot of what everyone is talking about here (minus the bittorrent stuff). For obvious reasons I can't get into the details of what were doing (www.infernallabs.com), but I am interested in what everyone is looking for. Honestly we designed the system around the "content-creation" industry, not as much distribution. It would be nice to know how close we are to another service.

So fire away, what would people be looking for in a distribution-handler?

-Brian

BigPapa   (Mar 15, 2006 at 14:28 GMT)
@Phil - I'm really interested in SVN so I will eagerly wait for your blog.

Prairie Games   (Mar 15, 2006 at 14:33 GMT)
@Phil: We used SVN (client-side) initially, then leveraged SVN as the backend and wrote a frontend to it...

Dreamer   (Mar 15, 2006 at 15:00 GMT)   Resource Rating: 5
@Phil, automatic versioning is nice and could easily be done, however you would want to do it in the tool you create to interface your initial patch commiter (uploader) whatever to S3, which actually looking at the code samples could pretty easily be an SVN or CVS server.

@Everyone, This is just a much, much cheaper way IMHO to distribute infrequently updated content like game patches etc.
Think about it, if you have 200+ users who all need a 50MB patch thats several gigs of data your server has to handle, you are either going to need a server farm to handle it or offload that stress to someone else.
Also your customers may take a break from playing for a week or two and come back, they may need 2-3 patches to get brought up to date, this allows you to get those to them for very little cost.

By my estimation if you used this it would save the need to purchase extra servers just to handle updates, which may be something that is only pushed once every week or so but would otherwise be sitting idle.

Phil Carlisle   (Mar 15, 2006 at 18:40 GMT)
I just think that SVN already serves that purpose. I'm advocating a system pretty much as Josh noted above.. use RapidSVN initially to get beta/alpha testers up and running, then when you go live, build in a mechanism using SVNCPP (the C++ core code in RapidSVN).

Of course your idea of the S3 thing would work equally well, but it doesnt currently have that functionality, whereas SVN does.

It depends I guess on wether you already have servers and extra bandwidth on them. I've got spare capacity on current provision, so SVN is ideal for me. Your case might be different (and yeah, I take your point about the cost of the data transfer being pretty small for S3).

I just dont want to have to develop anything to have this system work, SVN does it as a solution with no effort at all on my part (at least initially).

Phil.

You must be a member and be logged in to either append comments or rate this resource.