Migrating data from Mongo 4.2 to Atlas 5.0

This article demonstrates the usage of the tool called “Mongo-mirror” which can be used to migrate data from the old versions of MongoDB such as 4.2.25 to a newer version of MongoDB on Atlas.

We will exhibit how to set a single node replica set in Amazon AWS cloud EC2 instance, install MongoDB 4.2.25, load dummy data using the simrunner onto the source cluster, set an Atlas cluster (the destination), and use the aforementioned tool setup on a sync server (for demo purpose we will re-purpose the source machine as sync server, but it is advised to use a separate machine with higher resources as the sync server) to transfer the data from the source to destination.

Install Mongo on the EC2 machine

Launch an EC2 instance in AWS with Linux 2 with a t2.medium machine and then ssh into the machine once provisioned as shown below:

    
     ssh -i ~/.ssh/secrets.pem ec2-user@ip-172-31-29-64.us-east-2.compute.internal

Install Mongo 4.2.25 into the AWS EC2 machine:

    
     sudo tee /etc/yum.repos.d/mongo-org-4.2.repo<<EOF
[mongodb-org-4.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/4.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
EOF

sudo yum install -y mongodb-org

mongo --version
    
   
    
     MongoDB shell version v4.2.25
git version: 41b59c2bfb5121e66f18cc3ef40055a1b5fb6c2e
OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
allocator: tcmalloc
modules: none
build environment:
    distmod: amazon2
    distarch: x86_64
    target_arch: x86_64

Change ownership of log and db file to the user mongod

Change the ownership of the mongo log file and the data folder as shown below:

    
     sudo vi /etc/mongod.conf
sudo chown -R mongod:mongod /var/log/mongodb/mongod.log
sudo chown -R mongod:mongod /var/lib/mongo

sudo chmod 600 /var/log/mongodb/mongod.log
sudo chmod -R 755 /var/log/mongodb

Simulate dummy data into the source mongo deployment

Simulate data into the mongo database using the simrunner java application:

    
     # Install the required packages
sudo yum install -y git
sudo yum install -y java
sudo yum install -y maven
sudo yum install -y jq

git --version
java --version
mvn --version

# Clone git repo for the simrunner
git clone https://github.com/schambon/SimRunner.git

cd Simrunner
sudo cp characters.json /home/ec2-user/SimRunner/bin/
sudo cp locations.txt /home/ec2-user/SimRunner/bin/
mvn package

# Start the mongod service
sudo -u mongod mongod --config /etc/mongod.conf

# Add the connection string to the file Simrunner/templates/sample.json as shown below:
vi /home/ec2-user/SimRunner/templates/sample.json

"connectionString": "mongodb://admin:password@ip-172-31-29-64.us-east-2.compute.internal:27017/?authSource=admin&replicaSet=rs0",

# Start the simrunner by passing the following command:
java -jar /home/ec2-user/SimRunner/bin/SimRunner.jar /home/ec2-user/SimRunner/templates/sample.json

# Let the simrunner run for 10 minutes to add data to the simrunner.people collection.

Setup TLS on the source cluster

Setup self-signed TLS certificates (Follow Appendix A):

    
     sudo tee openssl-test-ca.cnf<<EOF
# NOT FOR PRODUCTION USE. OpenSSL configuration file for testing.

# For the CA policy
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 4096
default_keyfile = myTestCertificateKey.pem    ## The default private key file name.
default_md = sha256                           ## Use SHA-256 for Signatures
distinguished_name = req_dn
req_extensions = v3_req
x509_extensions = v3_ca # The extensions to add to the self signed cert

[ v3_req ]
subjectKeyIdentifier  = hash
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
nsComment = "OpenSSL Generated Certificate for TESTING only.  NOT FOR PRODUCTION USE."
extendedKeyUsage  = serverAuth, clientAuth

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default =
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = TestCertificateStateName
stateOrProvinceName_max = 64

localityName = Locality Name (eg, city)
localityName_default = TestCertificateLocalityName
localityName_max = 64

organizationName = Organization Name (eg, company)
organizationName_default = TestCertificateOrgName
organizationName_max = 64

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = TestCertificateOrgUnitName
organizationalUnitName_max = 64

commonName = Common Name (eg, YOUR name)
commonName_max = 64

[ v3_ca ]
# Extensions for a typical CA

subjectKeyIdentifier=hash
basicConstraints = critical,CA:true
authorityKeyIdentifier=keyid:always,issuer:always
EOF

Adjust the values of the signing certificates as shown in the box below:

    
     sed -i 's/countryName_default =.*/countryName_default = CA/' openssl-test-ca.cnf
sed -i 's/stateOrProvinceName_default =.*/stateOrProvinceName_default = Ontario/' openssl-test-ca.cnf
sed -i 's/localityName_default =.*/localityName_default = Toronto/' openssl-test-ca.cnf
sed -i 's/organizationName_default =.*/organizationName_default = Delbridge Solutions Inc/' openssl-test-ca.cnf
sed -i 's/organizationalUnitName_default =.*/organizationalUnitName_default = Data team/' openssl-test-ca.cnf
HOSTNAME_FQDN=$(hostname -f)
sed -i "/commonName_max = 64/i commonName_default = $HOSTNAME_FQDN" openssl-test-ca.cnf

openssl genrsa -out mongodb-test-ca.key 4096
openssl req -new -x509 -days 1826 -key mongodb-test-ca.key -out mongodb-test-ca.crt -config openssl-test-ca.cnf -batch
openssl genrsa -out mongodb-test-ia.key 4096
openssl req -new -key mongodb-test-ia.key -out mongodb-test-ia.csr -config openssl-test-ca.cnf -batch
openssl x509 -sha256 -req -days 730 -in mongodb-test-ia.csr -CA mongodb-test-ca.crt -CAkey mongodb-test-ca.key -set_serial 01 -out mongodb-test-ia.crt -extfile openssl-test-ca.cnf -extensions v3_ca
cat mongodb-test-ia.crt mongodb-test-ca.crt > test-ca.pem

Continued setup for self-signed TLS certificates (Follow Appendix B):

    
     sudo tee openssl-test-server.cnf<<EOF
# NOT FOR PRODUCTION USE. OpenSSL configuration file for testing.

[ req ]
default_bits = 4096
default_keyfile = myTestServerCertificateKey.pem    ## The default private key file name.
default_md = sha256
distinguished_name = req_dn
req_extensions = v3_req

[ v3_req ]
subjectKeyIdentifier  = hash
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
nsComment = "OpenSSL Generated Certificate for TESTING only.  NOT FOR PRODUCTION USE."
extendedKeyUsage  = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 =         ##TODO: Enter the DNS names. The DNS names should match the server names.
DNS.2 =         ##TODO: Enter the DNS names. The DNS names should match the server names.
IP.1 =          ##TODO: Enter the IP address.
IP.2 =          ##TODO: Enter the IP address.

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = TestServerCertificateCountry
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = TestServerCertificateState
stateOrProvinceName_max = 64

localityName = Locality Name (eg, city)
localityName_default = TestServerCertificateLocality
localityName_max = 64

organizationName = Organization Name (eg, company)
organizationName_default = TestServerCertificateOrg
organizationName_max = 64

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = TestServerCertificateOrgUnit
organizationalUnitName_max = 64

commonName = Common Name (eg, YOUR name)

commonName_max = 64
EOF

Adjust the values of the signing certificates as shown in the box below:

    
     sed -i 's/countryName_default =.*/countryName_default = CA/' openssl-test-server.cnf
sed -i 's/stateOrProvinceName_default =.*/stateOrProvinceName_default = Ontario/' openssl-test-server.cnf
sed -i 's/localityName_default =.*/localityName_default = Toronto/' openssl-test-server.cnf
sed -i 's/organizationName_default =.*/organizationName_default = Delbridge Solutions Inc/' openssl-test-server.cnf
sed -i 's/organizationalUnitName_default =.*/organizationalUnitName_default = Data team/' openssl-test-server.cnf
HOSTNAME_FQDN=$(hostname -f)
sed -i "/commonName_max = 64/i commonName_default = $HOSTNAME_FQDN" openssl-test-ca.cnf
HOSTNAME_IP=$(hostname -i)
sed -i "s|DNS.1 =.*|DNS.1 = $HOSTNAME_FQDN|" openssl-test-server.cnf
sed -i "s|IP.1 =.*|IP.1 = $HOSTNAME_IP|" openssl-test-server.cnf
sed -i "/DNS.2 =/d" openssl-test-server.cnf
sed -i "/IP.2 =/d" openssl-test-server.cnf

openssl genrsa -out mongodb-test-server1.key 4096
openssl req -new -key mongodb-test-server1.key -out mongodb-test-server1.csr -config openssl-test-server.cnf -batch
openssl x509 -sha256 -req -days 365 -in mongodb-test-server1.csr -CA mongodb-test-ia.crt -CAkey mongodb-test-ia.key -CAcreateserial -out mongodb-test-server1.crt -extfile openssl-test-server.cnf -extensions v3_req
cat mongodb-test-server1.crt mongodb-test-server1.key > test-server1.pem

Move the self-signed certificates into the required folder such as /etc/ssl/ and change the ownership of those certificates to “mongod” user as shown below:

    
     sudo mv test-server1.pem /etc/ssl/
sudo mv test-ca.pem /etc/ssl/
sudo chown -R mongod:mongod /etc/ssl/test-server1.pem
sudo chown -R mongod:mongod /etc/ssl/test-ca.pem

Add the TLS options into the mongo config file /etc/mongod.conf as shown below:

    
     sudo sed -i '/^net:/a \  tls:\n      mode: requireTLS\n      certificateKeyFile: /etc/ssl/test-server1.pem\n      CAFile: /etc/ssl/test-ca.pem\n      allowConnectionsWithoutCertificates: true' /etc/mongod.conf

Change the binding IP from 127.0.0.1 to 0.0.0.0 in the mongod config file as shown below:

    
     sudo sed -i 's/bindIp: 127.0.0.1  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting./bindIp: 0.0.0.0  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting./' /etc/mongod.conf

Add a user with the correct roles to the source cluster

Create the first user “admin” in the mongodb. 

  • First start the mongod process as a mongod user using the etc/mongod.config file.
  • Next enter into the mongo shell using the flags for TLS as shown.
  • Then create a user “admin” with “password” and add the required roles. Then exit from the shell.
    
     sudo -u mongod mongod --config /etc/mongod.conf
mongo --host $HOSTNAME_FQDN --port 27017 --tls --tlsCertificateKeyFile /etc/ssl/test-server1.pem --tlsCAFile /etc/ssl/test-ca.pem
    
   
    
     use admin
db.createUser({
  user: "admin",
  pwd: "password",
  roles: [
    { role: "userAdminAnyDatabase", db: "admin" },
    { role: "dbAdminAnyDatabase", db: "admin" },
    { role: "readWriteAnyDatabase", db: "admin" },
    { role: "root", db: "admin" }
  ]
})

Enable authorization in the /etc/mongod.config file:

    
     sudo sed -i 's/^#security:/security:\n  authorization: enabled/' /etc/mongod.conf

Enable replication in the /etc/mongod.config file, with appropriate replica set name:

    
     sudo sed -i 's/^#replication:/replication:\n  replSetName: rs0/' /etc/mongod.conf

Restart the mongod process by killing the current running mongod process and then restart the mongod process as shown below:

    
     sudo kill $(ps aux | grep -i "mongod --config" | grep -v grep | awk '{print $2}')

sudo -u mongod mongod --config /etc/mongod.conf

mongo --host $HOSTNAME_FQDN --port 27017 --tls --tlsCertificateKeyFile /etc/ssl/test-server1.pem --tlsCAFile /etc/ssl/test-ca.pem --username admin --authenticationDatabase admin

Enable replication for the source cluster

One of the conditions to use mongomirror is; the source cluster must be a replica set and not a standalone cluster. Initialize the Replica set by using the following command in the mongo shell logged in as admin:

    
     rs.initiate( {
   _id : "rs0",
   members: [
      { _id: 0, host: "ip-172-31-29-64.us-east-2.compute.internal:27017" }
   ]
})

Setup the Atlas destination cluster

Create an M10 cluster in Atlas, create a user called “migrationAdmin” with the “Atlas Admin” role and get the connections string from the connect tab and then select the driver from python 3.3 or earlier as shown here:

    
     cluster0-shard-00-00.6c8kk.mongodb.net:27017,cluster0-shard-00-01.6c8kk.mongodb.net:27017,cluster0-shard-00-02.6c8kk.mongodb.net:27017

Run Mongomirror to migrate data

Configure the mongomirror (version 0.12.8) with the options as shown below:

    
     # Mongomirror command to migrate data from source to destination:
nohup mongomirror \
--username admin \
--password password \
--authenticationDatabase "admin" \
--host rs0/ip-172-31-29-64.us-east-2.compute.internal:27017 \
--verbose \
--drop \
--destination "cluster0-shard-00-00.6c8kk.mongodb.net:27017,cluster0-shard-00-01.6c8kk.mongodb.net:27017,cluster0-shard-00-02.6c8kk.mongodb.net:27017" \
--destinationUsername "migrationAdmin" \
--destinationPassword migrationuseradmin \
--destinationAuthenticationDatabase=admin \
--oplogPath ./oplogs \
--bookmarkFile mongomirror.bookmark.json \
--oplogBatchSize 20000 \
--httpStatusPort 8080 \
--ssl \
--sslPEMKeyFile /etc/ssl/test-server1.pem \
--sslCAFile /etc/ssl/test-ca.pem
2>&1 | tee -a mongomirror.log &

Check the progress of the migration as shown below:

    
     curl http://localhost:8080 | jq

    
   
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   217  100   217    0     0   232k      0 --:--:-- --:--:-- --:--:--  211k
{
  "stage": "initial sync",
  "phase": "copying initial data",
  "details": {
    "copiedBytesAllColl": 4965790791,
    "simrunner.people": {
      "complete": false,
      "copiedBytes": 4965790791,
      "totalBytes": 7473563630
    },
    "totalBytesAllColl": 7473563630
  }
}

Perform cutover

After completion the progress will look like this and is ready for cutover:

    
     curl http://localhost:8080 | jq

    
   
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   193  100   193    0     0   225k      0 --:--:-- --:--:-- --:--:--  188k
{
  "stage": "oplog sync",
  "phase": "applying oplog entries",
  "details": {
    "currentTimestamp": 7388092787121455000,
    "latestTimestamp": 7388092787121455000,
    "lastWriteOnSourceTimestamp": 7388092185826034000
  }
}

For more details look at the nohup.out file (snapshot after migration shown here):

    
     2024-07-05T10:11:09.344+0000    Proceeding to tail oplog.
2024-07-05T10:11:09.344+0000    Start tailing the oplog from optime: {Timestamp: {1720174199 1}, Term: 4, Hash: 0}.
2024-07-05T10:11:09.345+0000    Current lag from source: 1m10s
2024-07-05T10:11:09.345+0000    starting TailAndApplyOplog loop iteration
2024-07-05T10:11:09.346+0000    Tailing the oplog on the source cluster starting at optime: {Timestamp: {1720174199 1}, Term: 4, Hash: 0}
2024-07-05T10:11:09.346+0000    Applying oplog entries occurring after {Timestamp: {1720174199 1}, Term: 4, Hash: 0}
2024-07-05T10:11:10.279+0000    buffering local.oplog.rs  11
2024-07-05T10:11:13.279+0000    buffering local.oplog.rs  11
2024-07-05T10:11:16.279+0000    buffering local.oplog.rs  11
2024-07-05T10:11:19.279+0000    buffering local.oplog.rs  11
2024-07-05T10:11:19.280+0000    Stopping temporary Oplog Buffer.
2024-07-05T10:11:19.280+0000    Stopping oplog buffering
2024-07-05T10:11:19.390+0000    Current lag from source: 10s
2024-07-05T10:11:20.281+0000    Buffered 12 oplog entries to oplogs/oplog-mongomirror.bson.sz
2024-07-05T10:11:20.281+0000    buffering local.oplog.rs  12
2024-07-05T10:11:20.281+0000    Removing compressed oplog file: oplogs/oplog-mongomirror.bson.sz
2024-07-05T10:11:29.348+0000    Current lag from source: 0s

As shown in the logs above once the lag reaches 0 seconds we can perform the cutover. Disconnect the old on-prem source mongo database and point the application to the new destination Atlas cluster by changing the connection string.

Conclusion

In this article, we demonstrated how to migrate data from a mongo deployment running the old version 4.2.25 to Atlas 5.0. We showed the important steps to secure the data packets when transferring the data from source to destination using TLS. We simulated data into the source cluster using the java tool called simrunner. Finally, we presented how to use mongomirror v0.12.8 to migrate data from the on-prem cluster to the Atlas.

Let's Connect!

Whether you’re looking to optimize your enterprise performance management or implement advanced data analytics, Delbridge Solutions brings unparalleled expertise and a client-focused approach to ensure your success. Contact us today to unlock the full potential of your business.
Get in Touch
Scroll to Top