Hedge funds and grid computing April 6, 2008
Posted by James Webster in : tech, finance , trackbackContinuing the topic of hedge funds, Matt wonders if any might consider using Amazon EC2 to quickly and cheaply scale up a grid computing environment, much as I asked a while back. As a way to scale up processing of complex overnight risk reports over exotic derivative books it may well be ideal. But I think there are significant latency issues that would prevent hedge funds or investment banks from using Amazon’s service at this stage for grid applications involving intra-day risk, that probably needs to be much closer to home. Definitely a business opportunity here somewhere. Disappointingly EC2 still does not support multicast.
Meanwhile TechCrunch reports that Google may be about to launch an opponent to Amazon Web Services;
A source says Google has plans to announce next week that it will make BigTable available to outside developers as a service. Amazon provides a similar service through SimpleDB, a cloud database solution announced in December.
Its an interesting move, given that Yahoo has thrown their weight behind the open-source Hadoop project and the effort to develop an open-source BigTable in HBase. Is Google trying to squash Hadoop and HBase innovation by simply giving application developers access to BigTable as a service, thereby denying Yahoo the ability to leverage an open-source effort to build a software stack that ultimately make them competitive against Google again? Perhaps that is too long a bow…
Back to financial technology, KX Systems have made their kdb+ database free for personal developers (via magmasystems). Q for Mortals might be worth checking out.
Comments»
I’ve been using both EC2 and Slicehost lately. Both offer virtualization, but approach it differently. I find it doubtful that very many hedgefunds would use EC2, mostly because hedgefunds are usually completely paranoid, and secondly, they often require large datasets. I think it would be possible to load the source data into SimpleDB or S3, but it would require some maintenance.
What I’d like to see is for larger banks to copy these sorts of technologies internally. 1) I like the model of Slicehost, being able to get a VM, IP address and such in under a minute, complete with backups if you want. Having a system like this could save months off ordering hardware, waiting for it to arrive, and get build, etc.. 2) EC2 charges a fixed price per VM per hour. I could see a different auction based model, that would allow for various business within a bank to share the compute grid for calculations. Businesses need different amounts of CPU power, and at different times. In most somewhat mature computing grids, you end up with different priorities of calculations. If a user presses ‘reprice’, you may want the result calculated as soon as possible. However, for a security which you don’t have a position in, having the system calculate a price once a day is acceptable. Other businesses need lots of CPU during trading hours, and none outside. The peak demand periods would basically pay for the cost of new hardware, and the relatively idle periods would pay to keep the machines on and powered.
Amazon recently allowed you to select a data center location. Right now they only have US east and west coasts, but hopefully they add other location soon as well.
I’d be real interested in Google’s BigTable being made available. The Google Code site has about 6-8 developers on it, and it manages 90,000 projects right now. They haven’t needed a whole lot of staff, because they’ve been able to use other Google technologies, like BigTable.
Brian, the paranoia angle is definitely accurate but I wonder if the attitude will shift over time? I think Salesforce.com had a similar problem across the business world as well.
The notion of an auction-driven model for pricing CPU power within a bank is interesting. The pendulum has definitely swung back towards centralisation of compute resources. I wonder if we would end up seeing an internal options market for CPU time in an investment bank that developed this type of grid…
[…] a few thoughts about that. I previously wondered about using Amazon EC2 to cheaply scale up on demand. It looks like DataSynapse, one of the leading grid vendors on the […]
[…] on the scalability of GridGain on Amazon EC2. Having previously thought about whether hedge funds might find EC2 a quick way to scale up batch processing of derivatives risk reports, I found the following EC2 limitation discovered by Max and his team interesting: Our second […]
DataSynapse Announces Support for Amazon Web Services and Beta Program to Advance Enterprise Adoption of Cloud Computing
http://www.pcworld.com/businesscenter/businesswire/release.html?guid=20090211005381en