How To Create a Sharded Cluster in MongoDB

Posted: 27th October 2016 by admin in all
Tags: , , , ,

MongoDB Sharding Topology

Sharding is implemented through three separate components. Each part performs a specific function:

  • Config Server: Each production sharding implementation must contain exactly three configuration servers. This is to ensure redundancy and high availability.

Config servers are used to store the metadata that links requested data with the shard that contains it. It organizes the data so that information can be retrieved reliably and consistently.

  • Query Routers: The query routers are the machines that your application actually connects to. These machines are responsible for communicating to the config servers to figure out where the requested data is stored. It then accesses and returns the data from the appropriate shard(s).

Each query router runs the "mongos" command.

  • Shard Servers: Shards are responsible for the actual data storage operations. In production environments, a single shard is usually composed of a replica set instead of a single machine. This is to ensure that data will still be accessible in the event that a primary shard server goes offline.

 

Implementing replicating sets is outside of the scope of this tutorial, so we will configure our shards to be single machines instead of replica sets. You can easily modify this if you would like to configure replica sets for your own configuration.

 

Initialize the Config Servers

The first thing we need to do is create a data directory, which is where the configuration server will store the metadata that associates location and content:

mkdir /mongo-metadata

Now, we simply have to start up the configuration server with the appropriate parameters. The service that provides the configuration server is called mongod. The default port number for this component is 27019.

We can start the configuration server with the following command:

(mongod –fork –logpath /opt/mongo-metadata/mongo1.log –configsvr –dbpath /opt/mongo-metadata –port 27019) – up Conf
Primary Conf Server
192.168.2.109:27019
192.168.2.109:27020
192.168.2.109:27021

Configure Query Router Instances

They query router service is called mongos. The default port number for this process is 27017 (but the port number in the configuration refers to the configuration server port number, which is 27019 by default).

The end result is that the query router service is started with a string like this:

mongos –fork –logpath /opt/mongos.log –configdb 192.168.2.109:27019,192.168.2.109:27020,192.168.2.109:27021

Script for this Procedure:

#!/bin/bash
#Check if no runing instances

if [-s /opt/MongoConf/mongoPID*]; then
killall mongod
killall mongos
fi


#Starting Conf Servers
/usr/bin/mongod --fork --logpath /opt/MongoConf/mongo1.log --configsvr --dbpath /opt/MongoConf/mongo-metadata/ --port 27019
echo -e "\033[31m Starting first Mongod Conf server"

/usr/bin/mongod --fork --logpath /opt/MongoConf/mongo1.log --configsvr --dbpath /opt/MongoConf/mongo-metadata1/ --port 27020
echo -e "\033[31m Starting second Mongod Conf server"

/usr/bin/mongod --fork --logpath /opt/MongoConf/mongo1.log --configsvr --dbpath /opt/MongoConf/mongo-metadata2/ --port 27021
echo -e "\033[31m Starting third Mongod Conf server"

echo -e "\033[31m Starting Mongos Balancer"
mongos --fork --logpath /opt/mongos.log --configdb 192.168.2.116:27019,192.168.2.116:27020,192.168.2.116:27021

echo -e "\033[31m Connect to Mongo with - mongo --host 192.168.2.116 --port 27017"

 

Add Shards to the Cluster

Shards Servers:
(mongod –fork –logpath /opt/data/instance2/monshard2.log –dbpath /opt/data/instance2 –port 27023)
(mongod –fork –logpath /opt/data/instance1/monshard1.log –dbpath /opt/data/instance1 –port 27022)

192.168.2.116:27022 ->sh.addShard( "192.168.2.116:27022" )
192.168.2.116:27023 ->sh.addShard( "192.168.2.116:27023" )

You can check your Sharding Status with – db.printShardingStatus()

--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "version" : 3,
    "minCompatibleVersion" : 3,
    "currentVersion" : 4,
    "clusterId" : ObjectId("53317aefca1ba9ba232b949e")
}
  shards:
    {  "_id" : "shard0000",  "host" : "127.0.0.1:27000" }
    {  "_id" : "shard0001",  "host" : "127.0.0.1:27001" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }

 

Thats it. Cluster is configured and ready to work.

#!/bin/bash
#Created by Denis P. June 2015.

#For debug unset #
#set -x

#Define VARs & check if SITELIST EXIST
if [ ! -d /tmp/ipchecks/ ] ;then
   mkdir /tmp/ipchecks/
fi
###
WORKDIR=/var/www
SITELIST=/tmp/ipchecks/sitelist
TODAY=`date +%m_%d_%y`
###
if [ ! -d /tmp/ipchecks/"$TODAY" ] ;then
   mkdir /tmp/ipchecks/"$TODAY"
fi
###
DEL='_'
SORTED="SORTED"
###
if [ ! -d /tmp/ipchecks/"$TODAY$DEL$SORTED" ] ;then
   mkdir /tmp/ipchecks/"$TODAY$DEL$SORTED"
fi

#parse all logs in WORKDIR
ls -h $WORKDIR | grep widgets | cut -d . -f 2 > $SITELIST
for SITENAME in `cat $SITELIST`
        do
          egrep -o "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"  $WORKDIR/widgets.$SITENAME.*/log/$SITENAME.*_access.log.1 |  sort | uniq -c  > /tmp/ipchecks/"$TODAY"/"$SITENAME"

          egrep "^[[:blank:]]*[0-9]{5,9} [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" /tmp/ipchecks/"$TODAY"/"$SITENAME" > /tmp/ipchecks/"$TODAY$DEL$SORTED"/"$SITENAME"

         if  [[ ! -s /tmp/ipchecks/"$TODAY$DEL$SORTED"/"$SITENAME" ]];then rm /tmp/ipchecks/"$TODAY$DEL$SORTED"/"$SITENAME"; fi
        done

#Creating list
cd /tmp/ipchecks/"$TODAY$DEL$SORTED"/
for x in `ls -1 /tmp/ipchecks/"$TODAY$DEL$SORTED"/| grep -v SENDING` ; do echo -e "Wensite $x\n$(cat $x)" > $x ; done
paste -d , /tmp/ipchecks/"$TODAY$DEL$SORTED"/*  | column -t -s "," >SENDING
#Clearing IPS from whitlisted (Not necessary step , here I checking whitlisted IPS in FW)
cp SENDING SENDINGBACK
while read x ; do grep -v "$x" SENDING > temp && mv temp SENDING ; done </tmp/ipchecks/AllowedIP

#Creating Template
cd /tmp/ipchecks/"$TODAY$DEL$SORTED"/
cp /tmp/ipchecks/mail.html /tmp/ipchecks/"$TODAY$DEL$SORTED"/
NUM=$[ $(cat mail.html|wc -l)-13 ]
headed=/tmp/ipchecks/"$TODAY$DEL$SORTED"/headed
tailed=/tmp/ipchecks/"$TODAY$DEL$SORTED"/tailed
head -13 mail.html >> headed && tail -$NUM mail.html >> tailed
while read x ;do  echo "$x" >> $headed  >> $headed; done</tmp/ipchecks/"$TODAY$DEL$SORTED"/SENDING
while read x ;do echo "$x" >> $headed ; done<$tailed
mv $headed /tmp/ipchecks/"$TODAY$DEL$SORTED"/sendrep.html

#Sending list
bash /usr/local/bin/telnetmail.sh (Any cript that can send email)
#GC
cd /tmp/ipchecks/"$TODAY$DEL$SORTED"/
rm mail.html headed tailed $SITELIST

2. Telnet Mail script ( telnetmail.sh in the cript)

#!/bin/bash
IFS='%'
WORKDIR=/var/www
SITELIST=/tmp/ipchecks/sitelist
TODAY=`date +%m_%d_%y`
DEL='_'
SORTED="SORTED"
count=1
while [[ $count=1 ]]
        do
(echo open SOMEMAILSRV
sleep 8
#echo helo mailsrv
echo helo srv
echo mail from:ipchecks@dom.com
sleep 2
echo rcpt to:denis@pesikov.tk
sleep 2
echo data
sleep 2
echo subject: IPSCHECKS  from `date`
while read line ; do echo "$line" ;done</tmp/ipchecks/$TODAY$DEL$SORTED/sendrep.html
sleep 2
echo .
sleep 1
echo quit)|telnet
count=2
exit
unset IFS
done

 

3.MAIL TEMLATE (mail.html in the script)

MIME-Version: 1.0
Content-Type: text/html; charset="ISO-8859-1"
&lt;html xmlns="http://www.w3.org/1999/xhtml"&gt;
&lt;head&gt;
&lt;meta content="text/html; charset=utf-8" http-equiv="Content-Type" /&gt;
&lt;meta name="viewport" content="width=device-width, initial-scale=1" /&gt;
&lt;title&gt;&lt;/title&gt;
&lt;/head&gt;
&lt;center&gt;&lt;font size="6" color="red"&gt;Detected suspicious IPs on websites:&lt;/font&gt;&lt;/center&gt;
&lt;body style="margin:0;padding:0;background-color:#c7c7c7"&gt;
&lt;pre&gt;
&lt;font size="3"&gt;

&lt;/font&gt;
&lt;/pre&gt;
&lt;/head&gt;
&lt;/html&gt;

 

Friends,

I noticed that SQL Server does a case-insensitive string match in a query. Hence in scenarios where passwords are to be validated, using a query as “WHERE Password =@Password” will give valid results if the user enter password as “admin” or “ADMIN” eventhough when the password is set as “aDmiN”. In this post, we will resolve this issue using a very simple method.

Let us consider below is our normal SQL procedure that validates a user from the tblUser table.

 
  1. CREATE PROCEDURE [dbo].[tblUserSelect_Authenticate]
  2. @Username nvarchar(50),
  3. @Password nvarchar(50)
  4. AS
  5. BEGIN
  6. SELECT * FROM tblUser WHERE Username = @Username AND Password = @Password
  7. END

To tell SQL do a case-sensitive search, we will modify the above procedure as below. Please note we added a “COLLATE SQL_Latin1_General_CP1_CS_AS” to the field we want an exact match for.

 
  1. CREATE PROCEDURE [dbo].[tblUserSelect_Authenticate]
  2. @Username nvarchar(50),
  3. @Password nvarchar(50)
  4. AS
  5. BEGIN
  6. SELECT * FROM tblUser WHERE Username = @Username AND Password = @Password COLLATE SQL_Latin1_General_CP1_CS_AS
  7. END

Keep learning and sharing! Cheers!

The OpenLDAP server is in Ubuntu's default repositories under the package "slapd", so we can

install it easily with apt-get. We will also install some additional utilities:

sudo apt-get update
sudo apt-get install slapd ldap-utils

You will be asked to enter and confirm an administrator password for the

administrator LDAP account.

Reconfigure slapd

When the installation is complete, we actually need to reconfigure the LDAP package.

Type the following to bring up the package configuration tool:

sudo dpkg-reconfigure slapd

You will be asked a series of questions about how you'd like to configure the software.

  • Omit OpenLDAP server configuration? No

  • DNS domain name?

    This will create the base structure of your directory path. Read the message to understand how it works.There are no set rules for how to configure this. If you have an actual domain name on this server, you can use that. Otherwise, use whatever you'd like.In this article, we will call it ldap.test

  • Organization name?

    Again, this is up to you.We will use example in this guide. 

  • Administrator password? – Use the password you configured during installation, or choose another one 

  • Database backend to use? HDB

  • Remove the database when slapd is purged? No

  • Move old database? Yes

  • Allow LDAPv2 protocol? No

 

Create Self Signed SSL

1. Generate a Private Key, after it we will sign all other clients certificates with this key too.

 # openssl genrsa -des3 -out ca.key 2048

2. Create Certificate

# openssl req -new -x509 -days 1825 -utf8 -key ca.key -out ca.cert

Here you need to answer to question SSL generate for you.The most! important question is

Common Name (eg, YOUR name) []: ldap.test(here give your LDAP FQDN name or an IP)

Creation of Self Signed Certificate

Generate private key

# openssl genrsa -out user.key 1024

Generate request for certificate signature

# openssl req -new -key user.key -out user.csr -utf8

And sign the certificate 

# openssl x509 -req -in user.csr -out user.cert \
            -CA ca.cert -CAkey ca.key -CAcreateserial -days 1095

Private key ca.key need to be stored a safely place, and a public ca.cert

we will publish along of LDAP clients.

Lets create on server machine a directory for storing the ceritificates 

# mkdir /usr/local/etc/openldap/ssl

Copy the cetificates to the directory

        # cp user.cert /etc/ldap/ssl/
        # cp user.key /etc/ldap/ssl/
        # cp ca.cert /etc/ldap/ssl/

Setting right permissions

        # chown openldap. * /etc/ldap/ssl/
        # chmod -R 750 /etc/ldap/ssl/

Now we need to change default LDAP conf file from ldap.conf to slapd.conf .

vim  /etc/default/slapd

Define the Path to your slapd.conf # SLAPD_CONF=/etc/ldap/slapd.conf

The slapd conf should be defined for your needs but here is mine , pretty basic – 

 

 

include         /etc/ldap/schema/core.schema
include         /etc/ldap/schema/cosine.schema
include         /etc/ldap/schema/nis.schema
include         /etc/ldap/schema/inetorgperson.schema

pidfile         /var/run/slapd/slapd.pid
argsfile        /var/run/slapd/slapd.args

#loglevel        none
loglevel        stats sync config conns

#TLSCipherSuite HIGH:MEDIUM:-SSLv2

#Working With ldap_tls_reqcert = allow(demand)
TLSCertificateFile    /etc/ldap/ssl/ldap_client.cert
TLSCertificateKeyFile /etc/ldap/ssl/ldap_client.key
TLSCACertificateFile  /etc/ldap/ssl/ldap_ca.cert

#NEW STARSSL CONF
#TLSCACertificateFile /etc/ssl/certs/ca_server.pem
#TLSCertificateFile /etc/ssl/certs/ldap_server.pem
#TLSCertificateKeyFile /etc/ssl/private/ldap_server.key
#TLS_REQCERT allow #for WebInterface not necessary

password-hash   {MD5}

modulepath      /usr/lib/ldap
moduleload      back_hdb
moduleload      syncprov.la
#moduleload     back_monitor

sizelimit 500
tool-threads 1

backend         hdb

database        hdb

suffix          "dc=ldap,dc=ldap-prod"

rootdn          "cn=admin,dc=ldap,dc=ldap-prod"
rootpw          Thi0kh0nuad
#rootpw                 {SSHA}UcCG********************

directory       "/var/lib/ldap"

dbconfig set_cachesize 0 2097152 0

# Sven Hartge reported that he had to set this value incredibly high
# to get slapd running at all. See http://bugs.debian.org/303057 for more
# information.

dbconfig set_lk_max_objects 1500
dbconfig set_lk_max_locks 1500
dbconfig set_lk_max_lockers 1500

index           objectClass eq

overlay         syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100

#lastmod         on

# Where to store the replica logs for database #1
# replogfile    /var/lib/ldap/replog

##Allow to change passwords
access to attrs=userPassword,shadowLastChange
        by self write
        by anonymous auth
        by users none

access to * by * read

access to attrs=shadowLastChange
        by * write
        by * auth

#access to dn.base="" by * read

#access to *
#        by self write
#        by * read

# Monitor Database

#database       monitor

#rootdn          "cn=Monitor"
#rootpw          {MD5}SomeMD5WasAlsoHere

#access to dn.subtree="cn=Monitor"
#        by dn.exact="cn=Monitor" write#
#       by users read#
#       by * read

#database config

#rootdn "cn=admin,cn=config"
#rootpw config

 

 

Ok,seems that we finished with Server configuration , lets start the OpenLDAP server

/etc/init.d/slapd start

Cheking it running: netstat -nap --tcp | grep -e 389 -e 636

tcp6       0      0 :::636        :::*          LISTEN      7392/slapd
tcp6       0      0 :::389        :::*          LISTEN      7392/slapd

Great !!

Client side configuration

Lets create on the clients machine a directory for storing a CA cetificate

mkdir /etc/ldap/ssl/

Copy the CA from the server to the client (scp or just copy paste)

Change the configuration file:  vim /etc/ldap/ldap.conf

# LDAP Defaults
# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE dc=ldap,dc=test
URI ldaps://ldap.test

timelimit 10
bind_timelimit 15
idle_timelimit 30
bind_policy soft

ssl on
tls_checkpeer yes

TLS_CACERT  /etc/ldap/ssl/ca.cert
TLS_REQCERT allow

pam_password md5

Lets restart NSLCD and NSCD daemons

 sudo /etc/init.d/nslcd restart &&  sudo /etc/init.d/nscd restart

And check if clinet start listennig

ldap# netstat -nap --tcp | grep -e 636 -e 389
tcp        0      0 X.X.X.X:40417      x.x.x.x:636       ESTABLISHED 25104/1
tcp        0      0 x.x.x.x:40419      x.x.x.x:636       ESTABLISHED 25186/-bash
tcp       70      0 x.x.x.x:40089      x.x.x.x:636       CLOSE_WAIT  3939/systemd-logind
tcp        0      0 x.x.x.x:40420      x.x.x.x:636       ESTABLISHED 23697/nscd

So we installed and configured working OpenLdap server with SSL Auth support

For diffirent systems there might be diffirent clinet configuration files:

Redhat: /etc/openldap/ldap.conf | /etc/pam_ldap.conf | /etc/nslcd.conf

Centos: /etc/sssd/sssd.conf

#Allow create HOME DIR from LDAP AUTH

Centos:
  grep sshd /var/log/audit/audit.log | audit2allow -M mypol
 semodule -i mypol.pp

Redhat:

Add to /etc/pam.d/system-auth:
session    required   pam_mkhomedir.so skel=/etc/skel/ umask=0022

Or
authconfig --enablemkhomedir --update

Get information from the LDAP (on client side)

ldapsearch -LL -H ldaps://ldap.test -b "dc=ldap,dc=test" -W -x -D "cn=admin,dc=ldap,dc=test"

For easy OpenLdap manage you can install PHPldapadmin (will be explaned in next Post)

Cheers, for any question leave a comments!

How to install ethtool from tar.gz

Posted: 27th October 2016 by admin in all
Tags: , , ,

1.Download latest ethtool archive from :https://www.kernel.org/pub/software/network/ethtool/

wget https://www.kernel.org/pub/software/network/ethtool/ethtool-3.15.tar.gz

2.Open the downladed archive

tar -zzfv ethtool-3.15.tar.gz

3. Go to unarchived folder

cd ethtool-3.15

Then type " ./configure –prefix=/usr " hit return should be screen fulls of more stuff.

Then type " make " more stuff

Then type " make install "

Hopefully this installs ethtool . Test by simply typing " ethtool " in a terminal , you should get the useage menu.