AWs basics: S3

Local Setup

• npm install aws-sdk
• create a .aws/credentails file in the ${user_home} directory ( True for both MAC, LInux and Windows )
• Copy the below contents to the ${user_home}/.aws/credentials file:

AWSAccessKeyId=agahaha
AWSSecretKey=ahahahae

Also create .aws/config file and add the following two lines:

[default]

region = us-west-2

How to work with pre-signed URLs

Bucket name: my.test
Region: us-west-2

Content Folder: mycontent

I have tested the above by creating pre-signed URLs for upload content:
My logs:

The URL is on bucket = my.test,region = us-west-2 is = https://s3-us-west-2.amazonaws.com/my.test/content/hello-upload.txt?AWSAccessKeyId=%22nitesh%22&Content-MD5=lSlzR14%2FTZkv5IV4CGweFw%3D%3D&Expires=1460725754&Signature=bcVxdTE44JkYPFjS77yLn4D2KNw%3D
Success generating pre signed url= https://s3-us-west-2.amazonaws.com/my.test/content/hello-upload.txt?AWSAccessKeyId=%22nitesh%22&Content-MD5=lSlzR14%2FTZkv5IV4CGweFw%3D%3D&Expires=1460725754&Signature=bcVxdTE44JkYPFjS77yLn4D2KNw%3D

Public and Private Folder for Content on S3

1. Log into the AWS console
2. Browse to my.test/content/public folder
3. Right click and choose 'public'.
4. Well that makes it all public.

Set CORS at AWS

AWS basics: EC2

Note: Draft document, needs review.

Ec2 instance	Username and password to connect to AWS console: D: akon@gmail.com PW: boomboom Ec2 instance type : m4.large instance with 2 vCPU and 8GiB memory. Availability Zone: Oregon Elastic IP :  a.b.c.d Public DNS : ec2-a-b-c-d.us-west-2.compute.amazonaws.com Instance name : alpha-z Instance ID is :  i-1884a1c0) Private IP is : k.n.m.z EC2 instance image :  Amazon Linux AMI 2016.03.0 x86_64 HVM GP2 Opened port: Port # 22 (ssh) and 3000 (nodeJS)
Users	Root - root (Super user creds are unknown.Need to check in the image document) Main user : ec2-user
MongoDB	Version  - MongoDB 3.2 MongoDB is running on port 27017 MongoDB can be connected via browser using url - http://a.b.c.d:27017/ (Currently port 27017 is blocked for any incoming call from external IP. Only app server running on localhost can access it) MongoDB service starts automatically when instance is booted MongoDB instance stores its data files in /var/lib/mongo and its log files in /var/log/mongodb by default. Config file of mongoDB “mongod.config” lies on /etc. All mongo executable in /usr/bin folder. stop mongoDB service using command  - “sudo service mongod stop” start mongoDB service using command “sudo service mongod start”
Conect to Mongo	Mongo port 27017 is blocked for all external access. Change /etc/mongod.conf bind_id and inboudn/outbound of aws console security group. But one can connect to mongo shell after login to the ec2 instance. Strange: localhost, a.b.c.d and ec2-.... public DNS does not work. Only thing that works is : sudo mongo 127.0.0.1:27017/mydb -u my-admin -p admin-pwd Incase of error 'BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly' Run the command > export LC_ALL=C
Node JS installation	version v6.0.0-pre Installation directory : /home/ec2-user/node NodeJS service does not start when instance is started. Need to work on that.
Node JS Application server	Webserver location - /opt/myserver NodeJS app server port : 3000 URL to connect the server: http://a.b.c.d:3000/
Ec2 instance connection with putty (Windows)	Open Puttygen and load the private key (.pem file). This file can be downloaded from AWS. Then save the private key (.ppk file) on your disk Then open putty, and load this private (.ppk file) in  SSH->AUTH section. Then connect to instance with host as ec2-user@ipaddress Now you will be remotely access the EC2 instance.
Ec2 instance connection ssh client ( Mac or Linux )	Get the private key (.pem or ppk ) file from the admin. Contact the admin. Store it at your local machine, at a secure path, say, /path/my-private-key.pem Change permissions of the my-private-key.pem file: chmod 400 /path/my-private-key.pem command to connect: ssh -i /path/my-private-key.pem {user}@{public_dns} common user: ec2-user public dns for the ec2 instance: ec2-a-b-c-d.us-west-2.compute.amazonaws.com
Setup Git repo	Commands below : cd /opt sudo mkdir "myserver" cd /opt/myserver/ sudo git init sudo git clone https:///myrepo.git

Some Unix date operations

input_date_format="%m/%d/%Y"
target_date_format="%m%d%Y"


Inbdate=`date +"$input_date_format" --date="12/11/2011"`
Incdate=`date +"$input_date_format" --date="10/31/2011"`


bdate=`date +"$target_date_format" -d "$Inbdate"`
cdate=`date +"$target_date_format" -d "$Incdate"`
cdate1=`date +"$target_date_format" -d "$Incdate - 1 day"`

-----------------------------------------------------------------------
Some usage of if with regex
--------------------------------

if [[ "$app_id" = Nitesh* ]]
then
app="NiteshApp"
elif [[ "$app_id" = Kumar* ]]
then
app="Kumarapp"
fi

----------------------------------------------------------------------------
Exception handling
------------------------

a.)
sqlcommand=`echo "insert ignore into impressed_$value ( select * from impressed_""$value""_""$app_id"" )"`
echo "executing : $sqlcommand"
mysql -u$root -p$password -h$db_server -P$db_port -D$db_name -e"$sqlcommand"
if [ $? -ne 0 ]
then
echo "Exiting failed : $sqlcommand"
exit 2;
fi

b.) Debgging
#!/bin/sh -ex

--------------------------------------------------------------------------------
use of '\' to ignore new line
-------------------------------------

sqlcommand=`echo "insert into adx_adenginelog_tmp \
( select \
reqType, sessionID, timestamp, appID, phoneNum, numResults \
from log_$ldate where appid='$app_id' and timestamp >= '$ltimestamp' and timestamp <= '$ltimestamp_eod' );"`
echo "Executing: $sqlcommand"
mysql -u"$db_username" -p"$db_password" -h${db_server} -P${db_port} -D${db_name} -e"$sqlcommand"

Some common usable Unix commands

Simple linux command

Find out the PID of the program on a given port

sudo lsof -i :8080 | grep LISTEN

1. grep -R "search string" "file"
2. find "filename" -type d -iname "string" ---- i is to ignore case

3. ###### to find all newlines and replace it with comma, use AWK ##########
Nice awk tutorial : http://www.savs.hcc.edu.tw/~chuavv/awk/nawk_23.html

http://bogdan.org.ua/2010/11/16/how-to-replace-newlines-with-commas-tabs-etc-merge-lines.html


awk '{print $1}' result.txt | tr '\n' ',' > sq.sql

awk uses file field to be separated by space.

if a different fs is required then
awk 'BEGIN { FS = "," } ; { print $2 }' sq.sql

4. SED ### this is to replace a particular regex with another regex
suppose a file has one line as
a,b,c,d

but i want then line by line like
a
b
c
d

command : sed 's/,/\n/g' file.txt > new.txt

so nw.txt now has
a
b
c
d

SOME EXAMPLES:
Most of the characters used in SED do not need backslashing like the below example
sed 's/-h${db_server}/-h${db_server} -P${db_port}/g' init.sh

4. TR ### this command also replace a separator with another separator
Like in the previous example new.txt has
a
b
c
d

we want out as a,b,c,d
then use awk to parse it line by line and then tr to replace

awk '{print $1}' new.txt | tr '\n' ',' > new1.txt

so new1.txt has
a,b,c,d

5. CUT ###########
a.) http://en.kioskea.net/faq/1451-sed-delete-one-or-more-lines-from-a-file
b. ) http://linux.die.net/man/1/cut

Some Common Unix editor errors

Problem:

1. The script test.sh was zipped copied from one linux server to another and then unzipped
2. The script test.sh wwas given right permissions
3. Scripts were read and modified using vim editor on both servers
4. Both the servers ( to and from ) have the same version of linux and red-hat
>>dmesg
dmesg command gives system details and showed that linux versions were same

While test.sh execution , below error occured

Error
/bin/sh^M: bad interpreter: No such file or directory

Cause of Error: Still not sure how ^M char could be found.
However using dos2unix test.sh solved the problem
for all the scripts use >> dos2unix *.sh ( but 'dos2unix *' does not work ?? weird !! )
b.) another solution is open using vi and then :set fileformat=unix also solved the problem

Links:
http://pbxinaflash.com/forum/showthread.php?t=3738

http://www.gizmola.com/blog/archives/87-Linux-shell-scripting-bad-interpreter-No-such-file-or-directory.html

http://www.edaboard.com/thread219178.html

some more array operations

Some help links
arithmetic operations :
http://bash.cyberciti.biz/guide/Perform_arithmetic_operations

for arrays:
http://www.arachnoid.com/linux/shell_programming.html
http://tldp.org/LDP/abs/html/arrays.html

master_slaves.txt is a file having lines separrated by ':' that needs to converted to array and then processed

>>cat master_slaves.txt
master_config:root:venableroot:3306:ml1
slave_config:ml2:ml3

master= array of values from line 1
slave = array of values from line 2


---------------------------------- script -----------------------------------
!/bin/bash -ex
p_r="test"
filename="$p_r/master_slave.txt"
master=()
slaves=()

while read line
do
table=$line

splits=(`echo "$table" | tr ':' ' '`)

if [ "${splits[0]}" = "master_config" ]
then
master=(${splits[*]})
elif [ "${splits[0]}" = "slave_config" ]
then
slave=(${splits[*]})
else
continue
fi

if [ "$master" ] && [ "$slave" ]
then

lenm="${#master[*]}"
echo "lenm=$lenm"
lens="${#slave[*]}"
echo "lens=$lens"

## append master to slave
slave[$lens]=${master[4]}
lens=$(( lens + 1 ))

i=0;
while [ $i -lt $lenm ]
do


echo "master:${master[$i]}"
i=$(( i + 1 ))
done

i=0;
while [ $i -lt $lens ]
do

echo "master:${slave[$i]}"
i=$(( i + 1 ))
echo "i=$i"

done

else
echo "master and slave config improper. check $filename"
fi

done < ${filename}
echo "over"

split a string using 'tr' and convert into array

Below script reads a file line by line
Each line is a string delimited by underscore ( '_' ).Script splits each string by this delimiter and extracts the last index,which is the date part of the string
and produces a new string

if the lines of status.txt are

impression_1_10142011
impression_1_10152011

then output of the below script is

impression_2_10142011
impression_2_1015201
-----------------------------------------------------------
#!/bin/sh

filename="status.txt"

while read line
do
table=$line
splits=(`echo "$table" | tr '_' ' '`)
tabletmp=`echo "impression_2_""${splits[2]}"`
echo "table tmp=$tabletmp"
done < ${filename}

echo "over"

-------------------------------------------------------------

Other ways of splitting a string by delimiters
a.
>> echo "nitesh:kumar:jag:gita" | awk -F: '{print $3}'
>> jag

b.
>> echo "nitesh:kumar:jag:gita" | cut -d: -f4
>> gita

c.
>> echo "nitesh" | awk '{print substr($0,2)}'
>> itesh

d.
>> echo "nitesh" | cut -c2
>> i

e.
>> echo "nitesh" | cut -c2-
>> itesh

f.
>> echo "nitesh" | cut -c2,4
>> ie

g.
>> echo "nitesh" | cut -c2-4
>> ite

h.
>> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 10321208 2687988 7108932 28% /
/dev/sdb 423135208 203084 401438084 1% /mnt
none 8963788 0 8963788 0% /dev/shm
/dev/sdg 1032123136 953849496 25844840 98% /data

then
>> df | grep "\ /data" | awk '{print $5}'
>> 98%

i. TO get process id of a process 'crond'
>> ps -ef | grep "crond" | grep -v grep | awk '{print $2}'
>> 988