Managing MongoDB
Data Replication
Replication is the process of synchronizing data across multiple servers. Replication provides redundancy and increases data availability with multiple copies of data on different database servers. Replication protects a database from the loss of a single server. Replication also allows you to recover from hardware failure and service interruptions. With additional copies of the data, you can dedicate one to disaster recovery, reporting, or backup.
Why Replication?
- Replication has the following features:
- To keep your data safe
- High (24*7) availability of data
- Disaster recovery
- No downtime for maintenance (like backups, index rebuilds, compaction)
- Read scaling (extra copies to read from)
- The replica set is transparent to the application
How Replication Works in MongoDB
MongoDB achieves replication by the use of a replica set. A replica set is a group of MongoDB instances that host the same data set. In a replica, one node is the primary node that receives all write operations. All other instances, such as secondaries, apply operations from the primary so that they have the same data set. A replica set can have only one primary node.
- A replica set is a group of two or more nodes (generally minimum of 3 nodes are required).
- In a replica set, one node is the primary node and the remaining nodes are secondary.
- All data replicates from primary to secondary nodes.
- At the time of automatic failover or maintenance, the election is established for the primary and a new primary node is elected.
- After the recovery of the failed node, it again joins the replica set and works as a secondary node.
A typical diagram of MongoDB replication is shown in which client applications always interact with the primary node and the primary node then replicates the data to the secondary nodes.
Replica Set Features
- A cluster of N nodes
- Any one node can be primary
- All write operations go to primary
- Automatic failover
- Automatic recovery
- Consensus election of primary
How to setup MongoDB replica set/replication
Here are the steps to setup the MongoDB replica set cluster:
- Take the odd number of nodes (could be ec2) in AWS or VMs in (Azure) and a minimum of 3 nodes required and enable connectivity between a node and open port 27017 in the same subnet in which those created nodes belong.
- Install the same version of the MongoDB server on all the nodes based on the OS environment with the below command:
vim /etc/yum.repos.d/mongodb-org-6.0.repo
[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc
- Apply the below commands to install the MongoDB servers and their binary
sudo yum install -y mongodb-org
- Once its installed create the directory for storing data and its logs (in this example we are storing data in /data/mongo/ and logs in /data/logs/ otherwise data and its logs will be stored in its root space and if the server got crashed there are a high chance for data loss)
mkdir -p /data/
mkdir -p /data/logs
mkdir -p /data/key
mv /var/lib/mongo/ /data/
mv /var/log/mongodb/mongod.log /data/logs/
chown -R mongod:mongod /data/mongo
chown -R mongod:mongod /data/logs
chown -R mongod:mongod /data/key
- Open /etc/mongod.conf and change the config like below and tune variables as per your system like cacheSizeGB (70 to 80 % of the total ram), timeStampFormat as per your timezone, replSetName as per your name , bindIp as per your system’s IP.
storage:
dbPath: “/data/mongo/”
engine: “wiredTiger”
wiredTiger:
engineConfig:
cacheSizeGB: 1
collectionConfig:
blockCompressor: snappy
journal:
enabled: true
directoryPerDB: true
systemLog:
destination: file
path: “/data/logs/mongod.log”
logAppend: true
timeStampFormat: iso8601-utc
replication:
oplogSizeMB: 10240
replSetName: “test”
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
bindIp: 127.0.0.1,192.168.161.46
port: 27017
#security:
# keyFile: “/data/key/mongo.key”
# authorization: “enabled”
# javascriptEnabled: false
- Start the server with the help of below commands
systemctl start mongod
- Above commands will start the server in unauthorized mode. Once the server started we have to create the root user with strong password like below. To connect with DB just type mongo or mongosh like this: Mongo. Note: after connecting to DB run the below commands.
use admin;
db.createUser( { user: “<username>”,
pwd: “<password>”,
roles: [ { role: “root”, db: “admin” },
{ role: “userAdminAnyDatabase”, db: “admin” },
{ role: “readWriteAnyDatabase”, db: “admin” },
{ role: “dbAdminAnyDatabase”, db: “admin” },
{ role: “clusterAdmin”, db: “admin” }] } );
- Once an admin/root user gets created we have to restart the server in authorize mode after changing the parameter below in /etc/mongod.conf (we just have to uncomment the security variables like below).
storage:
dbPath: “/data/mongo/”
engine: “wiredTiger”
wiredTiger:
engineConfig:
cacheSizeGB: 1
collectionConfig:
blockCompressor: snappy
journal:
enabled: true
directoryPerDB: true
systemLog:
destination: file
path: “/data/logs/mongod.log”
logAppend: true
timeStampFormat: iso8601-utc
replication:
oplogSizeMB: 10240
replSetName: “test”
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
bindIp: 127.0.0.1,192.168.161.46
port: 27017
security:
keyFile: “/data/key/mongo.key”
authorization: “enabled”
javascriptEnabled: false
- After that, we have to restart the MongoDB server with the help of the below commands
systemctl restart mongod
- After this, we have to create the key file for MongoDB replication with the help of the below commands and change the permission to 400 like the below and transfer the same key file to all the nodes and change the file permission to 400 on all the nodes like below
openssl rand -base64 756 > /opt/key/mongo.key
Node:- has to run on one node and use the same key on all the nodes.
chmod 400 /opt/key/mongo.key
Note:- The above cmd has to run on all the nodes.
- Once the key got created on all the nodes and permission also changed. We have to pick one node from the cluster and connect with the admin user which we have created in step 7 and run the below commands to initiate the replica set. It will initiate the node and by default, this node will be primary
mongo -u<adminuser> -p –authenticationDatabase admin (to connect with db)
>rs.initiate()
- Once the primary node got initiated run the below commands to add other replicas in the cluster
> rs.add(‘<ip>:<port>);
For example : rs.add(‘192.168.1.1.:27017);
- Once all the nodes are added check the status of the node and see if all are working fine with the help of the below commands
rs.status();
rs.conf();
Conclusions
As with any technology, it is important to educate yourself on the best practices by reading the documentation, investing in training, or working with a trusted partner to get the best out of this fantastic database.
We look forward to posting more on MongoDB’s features, how to use and configure it, and a host of other topics.
