coding beacon

[programming & visualization]

HPC tools and services

Distributed Computing in C++
-Combine the above two answers – MPI to communicate across clusters, OpenMP to parallelise for cores on clusters. If you have graphics cards, throw CUDA etc into the mix too. That’s what our distributed clusters do at work.-Checkout zNet is a C++ framework that is intended for multi-core and distributed core programming, supports streaming of build-in and custom types without any inheritance, transparently supports auto-discovery and load balancing and specifically oriented towards making application scalable on any hardware.
-CloudIQ Engine from Appistry. It allows you to distribute your C++ algorithms across any number of servers for processing. It also provides for process flow management for tasks. As part of the framework, failover is included, so if a task dies midstream (say someone pulls the plug on a machine), that task is automatically restarted on another node. And if that happens as part of a process flow, the whole flow does not have to be restarted, only the latest task. The framework automatically checkpoints your work at each step.

OpenMPI and/or OpenMP combinations work the best. We use OpenMPI on our supercomputing cluster to process large scientific jobs that require weeks of computing time. As an additional note, MPI has C++ bindings from Boost::MPI which supports lovely stuff like serialization of STL types (valarray, vectors, strings, etc.) to allow easier message-passing on your part.
Buying Cluster/Grid/Cloud Time?
-You might want to check out Amazon’s EC2 service:
Some people have already done some work in regards to clustering with EC2:*&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1
Additionally, Microsoft has offered Windows Azure, which has native hooks for .NET, allows you to run anything, really (Java, php), given that you are able to load a runtime and code from storage (or deployed with your app, but that has it’s own set of pros/cons).
-Amazon’s Elastic Compute Cloud is very interesting. You pay for what you use (Memory, CPU, Persisted storage) many OS options.

-There is a new service called Amazon Elastic MapReduce which runs on top of EC2 cluster. It has APIs in many of the programming languages including Ruby and PHP. Also, if you need more established service, checkoutGreenPlum


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: