Sharding with MangoDB

As per my earlier post we need to scale out our data store when the application data requirements outgrow a single instance capacity. This is in most cases achieved through sharding (a.k.a. partition of data).

With traditional RDBMS systems the application data model has to be modeled such from the very start to enable equalized partitioning of data or it may require a considerable change if we want to do it as an afterthought to scale up the application for using multiple database instances where each instance keep a balanced subset of the big data for an application. So in a way we have to plan for scaling out an application in typical RDBMS world but then with MongoDB it can actually occur as an afterthought as it provides automatic balancing and fail over of the partioned data

MongoDB as on 1.6 onwards supports production ready automated sharding architecture where we can convert a single instance into a cluster of shard whereby MongoDB manages the distribution of load of data across the shards (redistributing if the data on one node goes out of proportion with the rest) and also provides fail over support with each shard data being replicated in a replica set and all this with very minimal or no code change. The application needs to connect to a mongos process (which sits in front of the sharded cluster) which is responsible for managing the reads and writes across these shards giving a similar impression as we are working with a single node database systems.

Each shard as part of the cluster in itself consists of multiple servers where data is replicated either as a master slave configuration or as a replica set (v1.6 onwards). The data set or the collection of data is distributed through shard key pattern where we use one or more fields to define this key. These fields are to be choosen carefully and has to be granular enough to allow even distribution of data across the nodes and thus in most cases we end up using compound key with multiple fields.

Apart from the shard cluster we also have config servers and routing processes where config servers are the one with the metadata information about each node and the chunk of data each node has and the routing process being responsible for intercepting the client requests and processing them in orchestration with the different nodes of the cluster giving an impression that the request is being processed through a single server.

At the base minimum we can have just two or more shards, one config server and a mongos routing process to create a shard cluster. You can read more about configuring the shards here

About these ads

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

%d bloggers like this: