Install Elastic Search Logstash Kibana

 RHEL Setup

gpg encrypt and decrypt at other server

 First generate keys.
1. gpg --gen-key
      Type 1 and Enter for RSA and RSA (default)
      What keysize do you want? (2048) : --- Just Enter
      Key is valid for? (0): --- Just Enter
      Is this correct? (y/N): y --- Type y and Enter
      Real name: Dheeraj --- Type your Name
      Email address: dhirajkumar**5@gmail.com --- Type your email address
      Comment: First GPG --- Type any comment
      Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O --- Type O and Enter
      Enter passphrase: 12345678 --- Type any passphrase and remember this
      Repeat passphrase: 12345678 --- Type same as above
      Do any activities on your computer it will record and create keys.
     
2.  Optional Step:
     Check your keys

     gpg --list-secret-keys
sec   2048R/02551747 2020-12-15
uid                  Dheeraj (First GPG) <dhirajkumar055@gmail.com>
ssb   2048R/7C97C661 2020-12-15

3. Optinal Step:
    mkdir gpg_keys
    cd gpg_keys
    echo "12345" >textfile1
    echo "67890" >textfile2
    tar -cvf textfile.tar textfile1 textfile2
   
Note: We will encrypt textfile.tar and decrypt it.

4. Sign the file you want to encrypt like textfile.tar
    gpg --sign textfile.tar
    A new file with textfile.tar.gpg will be created.
    Send this generated file like textfile.tar.gpg at the destination server.

5. Optional Step: Verify the .gpg file
    gpg --verify textfile.tar.gpg
gpg: Signature made Tuesday 15 December 2020 02:34:20 PM IST using RSA key ID F085C77B
gpg: Good signature from "dheeraj <dhirajkumar055@gmail.com>"

6. Optional Step:
    mkdir publickeys
    cd publickeys

7. Export public key
    gpg --export -a dheeraj >dheerajpublic.key

8. Send the dheerajpublic.key to the destination server and import it at destination.
    gpg --import dheerajpublic.key

9. At destination
    gpg --output textfile.tar -d textfile.tar.gpg

Summary
At source :
    gpg --gen-key
    gpg --sign textfile.tar
    gpg --export -a dheeraj >dheerajpublic.key
    scp or cp textfile.tar.gpg and dheerajpublic.key
At Destination:
    gpg --import dheerajpublic.key
    gpg --output textfile.tar -d textfile.tar.gpg







 

Redis

#!/bin/bash
sudo amazon-linux-extras install redis4.0

sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker ec2-user
docker run -v redisinsight:/db -p 8001:8001 redislabs/redisinsight:latest

Install Redis on RedHat
#!/bin/bash
sudo yum install redis -y
sudo systemctl start redis
sudo systemctl enable redis

Error: Redis is running in protected mode because protected mode is enabled,
Solution: redis-server

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

for i in range(1,101):
    print("redis-cli {set} {i} {i}".format(set="set",i=i))

 

 

 

 

 

 

createKeysScript.sh
###########################################

#!/bin/bash
echo "#!/bin/bash" >insertKeysIntoRedis$1.sh
for i in `seq 1 $1`
do
echo "redis-cli set $i $i" >>insertKeysIntoRedis$1.sh
done

echo "redis-cli save" >>insertKeysIntoRedis$1.sh
echo "redis-cli info keyspace" >>insertKeysIntoRedis$1.sh

chmod +x insertKeysIntoRedis$1.sh

##########################################

importDump.sh
##########################################
#!/bin/bash
echo "Clearing all the keys"
#Uncomment below to clear all the keys
#redis-cli flushall
redis-cli save

echo "Stopping redis"
systemctl stop redis

echo "Replacing dump file"
sudo rm -rf /var/lib/redis/dump.rdb
sudo cp /home/ec2-user/dump.rdb /var/lib/redis/

systemctl start redis
redis-cli save

redis-cli info keyspace

##########################################

Redis Commands

set key value
get key
del key1 key2 key3
exists key1 key2 key3

Expire like for session of bank
set key value ex 10        #ex for seconds px for milliseconds
ttl key                             # -2 for expired or unavailable
ttl key                             # -1 no expiry date
Expire key 20 #(time in second)                   
set key value px 10000
pttl key
ttl key
pexpire key 20000
set key 1
set key 2
get key #will give 2
persist key #will persist the key
ttl key #will give -1 now

keys * #Get all the keys


KEYS Command:
? - one character - number of characters will be same
* - multiple character
[ae] - a or e
[^e] - e should not be there
[a-c] - anything between a to c can be there

keys *
keys h*llo
keys h[ae]llo
keys h[^e]llo
keys h[e-r]llo
keys *ll*
keys *e*
keys h???? - for 5 character long with first character h
keys ?????

shutdown nosave     #persist previously saved changes
shutdown save        #persist current changes


Randomkey
Rename key newkey       #replace newkey if already present
renamenx key newkey    #Do nothing newkey already present
touch key                         #Changes the access/modified time of key in DB
unlink key                    #Similar  to del
#unlink command creates new thread and delete the specified key
type key
dump key #you will get coded value
del key
restore key 0 <coded value> #Will give if key is already present or not deleted
restore key 0 <coded value> REPLACE #Forcefully restore even if key is present
restore otherkey 0 <coded value> #Get the value from stored data

set key value nx #Won't create key if already present #new key only
set key value xx #opposite of nx #existing key only

append key value   #gives length of value
incr key
incrby key "integerValue"
decr key
decrby key "integerValue"
incrbyfloat key "floatValue"
decrbyfloat key "floatValue"

getset key newvalue
mset key1 value1 key2 value2 key3 value3
mget key1 key2 key3
msetnx key1 value1 key2 value2 key3 value3   #All keys must be new

getrange key startinteger endinteger
setex key <expirySeconds> value #Set expiry time in seconds
psetex key <expiryMilliSeconds> value
setrange key <index> value
strlen key

lpush key value1 value2 value3
lrange ket 0 -1 #Get all elements
rpush key value5 value6
lpushx key value8 value9 #push only if key exists
rpushx key value0 value4
rpop key #pop last element
lpop key
ltrim key startIndex endIndex
lset key index newvalue #replace newvalue at index
lindex key index #get value at index
linsert key before|after existingvalue newvalue
llen listkey
lrem key removecount value #Remove value = count
#if removecount is negative Remove from end
#if removecount is 0 Remove all values equal to value

hset key field1 value1
hget key field1
hmset key field1 value1 field2 value2 field3 value3
hmget key field1 field2 field3
hgetall key
hvals key #Only values
hkeys key
hexists key field1
hlen key
hsetnx student field1 value1 #field1 should not be present before this command
#nx will not update the field
hdel key field
hincrby key field <integervalue> #hincrby will not work for float
hincrbyfloat key field <floatvalue> #hincrbyfloat will work for both float/integer
hstrlen key field
#hash will return positive value if new field gets added and not value is updated


Sets
sadd key value1 value2 value3 value4
smembers key
sismember key field #check is field is member of key
scard key #Cardinality or total members
smove sourcekey destkey member
spop key <count> #pop random members
srem key member1 member2 member3 #Remove members
sdiff key1 key2 #key1 - key2
sdiffstore key key1 key2 #key=key1-key2
sinter key1 key2
sinterstore key key1 key2   #key=key1&key2
sunion key1 key2
sunionstore key key1 key2 #key=key1|key2
srandmember key
del key

Sorted Set
zadd key score1 member1 score2 member2 #Score is float only
zrange key <startInteger> <endInteger>
zrange key <startInteger> <endInteger> withscore
zrange key nx <startInteger> <endInteger>
#You can have multiple values mapped to a single score
zadd key ch score1 member1 score2 member2 #prints no. of members added
zadd key incr <integer> member1 ##Score will be incremented
zcard key
zrem key member1 member2
zscore key member #Get score of member
zrevrange key <startInt> <endInt> #Prints in reverse order of range
zrank key member #Give the index of member in sorted set
zrevrank key member
zincrby key <integer> member #Increment score
zcount key <min> <max> #how many members have score between min and max
zcount name -inf +inf #All members of sorted set
zpopmax key <count> #popped maximum scored members
zpopmin key <count>
zinterstore <destinationkey> <numberOfKeys> <key1> <key2> ... weights <integer1> <integer2> ... sum|min|max
zunionstore <destinationkey> <numberOfKeys> <key1> <key2> ... weights <integer1> <integer2> ... sum|min|max
#Default weight : 1
#Default aggregator : sum
zrangebyscore key <min> <max> #Get members with scores b/w min and max
zrangebylex  key <min>(member1 <max>[member2 #get lexicographically sorted members between min and max
zrangebylex  key - + #- start + end
zlexcount key (member1 [member2 #count between min and max laxicographically
zrevrangebylex key [member2 [member1 #must give mem2 and then mem1
zremrangebylex key [member2 [member1
zremrangebyrank key <integer> <integer> #Rank is like index
zremrangebyscore key <integerMin> <integerMax> #Delete b/w min and max


Transaction
multi exec discard watch
Transaction consists of block of commands.

multi
set key1 value1
set key2 value2
exec

multi
incrby key1 500
decrby key 500
exec

multi

...
discard   #will abort the transaction

watch key
multi
...

watch from one client cann't be unwatched by another client


publish <channel> Hello
subscribe <channel1> <channel2>
Ctrl+C for unsubscribe

Patterned-Subscription
psubscribe ch* | ch? | ch[12e]
#administration pubsub
pubsub Numsub <channel>#no. of subscriber but not for regex subscriber
pubsub numpat #No. of patterned subscription client

#Geospatial good for low latency
#GeoHash 52bit integer value
#Stored in sorted key
#longitude and lattitude is stored as score
#Ranges
#Longitude : -180 to 180
#Lattitude : -85.0511 to 85.0511

geoadd geopoints longitude latitude member
zrange geopoints 0 -1 withscores
geohash geopoints member
www.geohash.org/<geohash>
geopos geopoints member #Get longitude & latitude
geodist geopoints member1 member2 mi|km|ft|m(default)
georadiusbymember geopoints member <integerRadius> mi|km withdist|withcoord|withhash Count <integer> desc|asc
georadiusbymember geopoints member <integerRadius> mi|km store <member> #Store hashes
georadiusbymember geopoints member <integerRadius> mi|km storedist <member> #Store distances
#In georadis command we have to provide longitude and latitude both instead of members
georadius geopoints latitude longitude <integer> mi|km|m|ft withdist

Hyperloglog
#:Unique elements with less time complexigy
#Only store counts
pfadd key element1 element2 element3
pfadd key element3 #will return 0
pfcount element1
pfmerge destkey1 srckey2


Redis With Python
import redis
r=redis.Redis('host')
r.set("key","value")
r.get("key")

String Integer Float Byte only allowed
today=datetime.date.today()
#stoday=today.isoformat() #Python3.7+
stoday=(str)(today)
visitors={"ss","dd","dda"}
r.sadd(stoday, *visitors)
values=r.smembers(stoday)
print(values)
cardinality=s.scard(stoday)
print(cardinality)

Install nginx and configure ssl

 On centos

Createa repo
[nginx]
name=nginx repo
baseurl=https://nginx.org/packages/centos/$releasename/$basearch/
gpgcheck=0
enabled=1

replace $releasename with 6 or 7 depending on your /etc/*release* file

Install epel-release for fedora

For

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

sudo mkdir /etc/ssl/private
sudo chmod 700 /etc/ssl/private
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt 

etcd cluster

 

{
  export ETCD_VER=v3.4.10
  wget -q "https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz"
  tar zxf etcd-v3.4.10-linux-amd64.tar.gz
  mv etcd-v3.4.10-linux-amd64/etcd* /usr/local/bin/
  rm -rf etcd*
  PATH=$PATH:/usr/local/bin
}

Change below name and IPs for 1,2,3 accordingly

ETCD_NAME="etcd3" 

NODE_IP=$(hostname -i)

ETCD1_IP="172.31.8.194"
ETCD2_IP="172.31.12.116"
ETCD3_IP="172.31.13.204"


cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --initial-advertise-peer-urls http://${NODE_IP}:2380 \\
  --listen-peer-urls http://${NODE_IP}:2380 \\
  --advertise-client-urls http://${NODE_IP}:2379 \\
  --listen-client-urls http://${NODE_IP}:2379,http://127.0.0.1:2379 \\
  --initial-cluster-token etcd-cluster-1 \\
  --initial-cluster etcd1=http://${ETCD1_IP}:2380,etcd2=http://${ETCD2_IP}:2380,etcd3=http://${ETCD3_IP}:2380 \\
  --initial-cluster-state new
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

{
  systemctl daemon-reload
  systemctl enable --now etcd
}

ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 member list

ETCD Cluster is ready now

How to use cluster in kubeadm?

 {

ETCD1_IP="172.31.8.194"
ETCD2_IP="172.31.12.116"
ETCD3_IP="172.31.13.204"

cat <<EOF > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  podSubnet: "192.168.0.0/16"
etcd:
    external:
        endpoints:
        - https://${ETCD1_IP}:2379
        - https://${ETCD2_IP}:2379
        - https://${ETCD3_IP}:2379
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "172.16.16.100"
EOF

} 

kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=all

kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

kubeadm token create --print-join-command

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config  

How delete etcd data in cluster?
# rm -Rf /etc/etcd/*
# rm -Rf /var/lib/etcd/*

login reboot records

 /var/run/utmp (virtual file)
who, w and uptime commands

who command
<username> <devicefile for this terminal> <date> <time when user logged in> <Ip from where user logged in>
vvdn     tty7         2020-11-15 15:02 (:0)

 

What is difference between pts and tty?
tty- Virtual Terminals
https://www.youtube.com/watch?v=vAr9PM9dEtE
Used for executing commands and offering inputs
You cannot use mouse with virtual terminal.
Enables different users to work on different program at the same time on same computer.
Enter into virtual terminal : Ctrl+Alt+F1
There are 6 virtual terminal : Ctrl+Alt+F1 to F6
To go back to main screen : Ctrl+Alt+F7
tty is the teletype number

Some useful commands
reset: reset the terminal
history: list of commands executed by the user
Ctrl+d: logout of terminal
Ctrl+Alt+del : Reboot the system

Virtual IP:
VRRP : Virtual Router Redundancy Protocol
keepalived is the software which allows us to do VRRP

/etc/keepalived/keepalived.conf
vrrp_instance httpd2{
 state BACKUP
 interface eth0
 virtual_router_id 101
 priority 100
 authentication{
  auth_type PASS
  auth_pass 1234
 }
 virtual_ipaddress{
  192.168.254.100
 }
}

Installing pcs cluster...
Check the selinux status: sestatus

yum repo:
[rhel]
name=redhatrepo
baseurl=file:///directory
enabled=1
gpgcheck=0

createrepo /directory

yum install pcs pacemaker fence-agents lvm2-cluster resource-agents psmisc policycoreutils-python gfs2-utils -y

Check password expiry of a user:
chage -l <username>

pcs cluster auth ip-172-31-14-80.ap-south-1.compute.internal ip-172-31-1-112.ap-south-1.compute.internal
give===>
username: hacluster
password: redhat

Create cluster
pcs cluster setup --start --name dheerajPCScluster ip-172-31-14-80.ap-south-1.compute.internal ip-172-31-1-112.ap-south-1.compute.internal --force


pcs cluster start --all

Checking status
systemctl status pacemaker
systemctl status corosync
pcs status
crm_mon -r1


pcs cluster destroy

Log Rotation

Delete files older then n days
n=1  ---- Number of days
filePattern="abc.log.day*"   --- Pattern of file
find . -mtime +$n -name "$filePattern" -exec rm {} \;  --- Command delete files older then n days


Move files older then n days to directory
dir=




Zip files older then 1 day
find . -mtime +1 -name "fileName*" -exec gzip {} \;