Wednesday, January 8, 2020

Linux - Interactive, Non-interactive, login and non-login shells

A shell is the generic name for any program that gives you a text-interface to interact with the computer. You type a command and the output is shown on screen.
Many shells have scripting abilities: Put multiple commands in a script and the shell executes them as if they were typed from the keyboard. Most shells offer additional programming constructs that extend the scripting feature into a programming language.
On most Unix/Linux systems multiple shells are available: bash, csh, ksh, sh, tcsh, zsh just to name a few. They differ in the various options they give the user to manipulate the commands and in the complexity and capabilities of the scripting language.
Interactive: As the term implies: Interactive means that the commands are run with user-interaction from keyboard. E.g. the shell can prompt the user to enter input.
Non-interactive: the shell is probably run from an automated process so it can't assume if can request input or that someone will see the output. E.g Maybe it is best to write output to a log-file.
Login: Means that the shell is run as part of the login of the user to the system. Typically used to do any configuration that a user needs/wants to establish his work-environment.
Non-login: Any other shell run by the user after logging on, or which is run by any automated process which is not coupled to a logged in user.

For Bash, they work as follows. Read down the appropriate column. Executes A, then B, then C, etc. The B1, B2, B3 means it executes only the first of those files found.
+----------------+-----------+-----------+------+
|                |Interactive|Interactive|Script|
|                |login      |non-login  |      |
+----------------+-----------+-----------+------+
|/etc/profile    |   A       |           |      |
+----------------+-----------+-----------+------+
|/etc/bash.bashrc|           |    A      |      |
+----------------+-----------+-----------+------+
|~/.bashrc       |           |    B      |      |
+----------------+-----------+-----------+------+
|~/.bash_profile |   B1      |           |      |
+----------------+-----------+-----------+------+
|~/.bash_login   |   B2      |           |      |
+----------------+-----------+-----------+------+
|~/.profile      |   B3      |           |      |
+----------------+-----------+-----------+------+
|BASH_ENV        |           |           |  A   |
+----------------+-----------+-----------+------+
|                |           |           |      |
+----------------+-----------+-----------+------+
|                |           |           |      |
+----------------+-----------+-----------+------+
|~/.bash_logout  |    C      |           |      |
+----------------+-----------+-----------+------+
In more detail is this excellent flowchart from http://www.solipsys.co.uk/new/BashInitialisationFiles.html :

Wednesday, February 8, 2017

Linux PORTs and PIDs

Get PID of a process running on a given PORT



down voteaccepted

$ sudo netstat -nlp | grep :80
tcp  0  0  0.0.0.0:80  0.0.0.0:*  LISTEN  125004/nginx
You can also use lsof:
$ sudo lsof -n -i

Nginx and AWS


To Install Nginx from Source

cd /tmp #so we can clean-up easily
wget http://nginx.org/download/nginx-1.10.0.tar.gz
tar zxvf nginx-1.10.0.tar.gz && rm -f nginx-1.10.0.tar.gz
cd nginx-1.10.0
sudo yum install pcre-devel openssl-devel #required libs, not installed by default
./configure \
  --prefix=/etc/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --pid-path=/var/run/nginx.pid \
  --lock-path=/var/run/nginx.lock \
  --with-http_ssl_module \
  --with-http_v2_module \
  --user=nginx \
  --group=nginx
make
sudo make install
sudo groupadd nginx
sudo useradd -M -G nginx nginx
rm -rf nginx-1.10.0

You'll then want a service file, so that you can start/stop nginx, and load it on boot.
Here's one that matches the above config. Put it in /etc/rc.d/init.d/nginx:
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  NGINX is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/etc/nginx/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/run/nginx.lock

make_dirs() {
   # make required directories
   user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
   if [ -z "`grep $user /etc/passwd`" ]; then
       useradd -M -s /bin/nologin $user
   fi
   options=`$nginx -V 2>&1 | grep 'configure arguments:'`
   for opt in $options; do
       if [ `echo $opt | grep '.*-temp-path'` ]; then
           value=`echo $opt | cut -d "=" -f 2`
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value && chown -R $user $value
           fi
       fi
   done
}

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac
Set the service file to be executable:
sudo chmod 755 /etc/rc.d/init.d/nginx
Now you can start it with:
sudo service nginx start
To load it automatically on boot:
sudo chkconfig nginx on
Finally, don't forget to edit /etc/nginx/nginx.conf to match your requirements and run 
sudo service nginx reload 
to refresh the changes.

To disable Nginx 


sudo /etc/init.d/nginx stop
Or 

sudo service nginx stop

sudo chkconfig nginx off
then confirm with
chkconfig --list | grep nginx

To list all the autostart services or processes 

sudo chkconfig 

To Reload Nginx


sudo nginx -s reload

or

 sudo service nginx reload

Reload is required whenever nginx.conf file changes

Monday, January 30, 2017

MongoDB on MAC

Some Basic Commands

Start and Stop a Mongod server


brew services start mongodb


brew services stop  mongodb


Restore and Dump database


To restore a single database you need to provide the path to the dump directory as part of the mongorestore command line.
For example:
# Backup the training database
mongodump --db training

# Restore the training database to a new database called training2
mongorestore --db training2 dump/training
The --db option for mongodump specifies the source database to dump.
The --db option for mongorestore specifies the target database to restore into.

Mongo DB shell Commands

Start mongo shell : {path_to_mongo}/bin/mongo
If path already there then type
> which mongo
> mongo

Help: https://docs.mongodb.com/manual/reference/mongo-shell/

Login with username and pwd


$ mongo mydb -u uname -p 'password'

mongo Shell Command History

You can retrieve previous commands issued in the mongo shell with the up and down arrow keys. Command history is stored in ~/.dbshell file. See .dbshell for more information.

Command Line Options

The mongo shell can be started with numerous options. See mongo shell page for details on all available options.
The following table displays some common options for mongo:
OptionDescription
--helpShow command line options
--nodb
Start mongo shell without connecting to a database.
To connect later, see Opening New Connections.
--shell
Used in conjunction with a JavaScript file (i.e. <file.js>) to continue in themongo shell after running the JavaScript file.
See JavaScript file for an example.

Command Helpers

The mongo shell provides various help. The following table displays some common help methods and commands:
Help Methods and CommandsDescription
helpShow help.
db.help()Show help for database methods.
db.<collection>.help()Show help on collection methods. The <collection> can be the name of an existing collection or a non-existing collection.
show dbsPrint a list of all databases on the server.
use <db>Switch current database to <db>. The mongo shell variable db is set to the current database.
show collectionsPrint a list of all collections for current database
show usersPrint a list of users for current database.
show rolesPrint a list of all roles, both user-defined and built-in, for the current database.
show profilePrint the five most recent operations that took 1 millisecond or more. See documentation on the database profiler for more information.
show databasesPrint a list of all available databases.
load()Execute a JavaScript file. See Write Scripts for the mongo Shell for more information.

Basic Shell JavaScript Operations

The mongo shell provides a JavaScript API for database operations.
In the mongo shell, db is the variable that references the current database. The variable is automatically set to the default database test or is set when you use the use <db> to switch current database.
The following table displays some common JavaScript operations:
JavaScript Database OperationsDescription
db.auth()If running in secure mode, authenticate the user.
coll = db.<collection>
Set a specific collection in the current database to a variable coll, as in the following example:
coll = db.myCollection;
You can perform operations on the myCollection using the variable, as in the following example:
coll.find();
db.collection.find()
Find all documents in the collection and returns a cursor.
See the db.collection.find() and Query Documents for more information and examples.
See Iterate a Cursor in the mongo Shell for information on cursor handling in the mongo shell.
db.collection.insert()Insert a new document into the collection.
db.collection.update()Update an existing document in the collection.
db.collection.save()Insert either a new document or update an existing document in the collection.
db.collection.remove()Delete documents from the collection.
db.collection.drop()Drops or removes completely the collection.
db.collection.createIndex()Create a new index on the collection if the index does not exist; otherwise, the operation has no effect.
db.getSiblingDB()Return a reference to another database using this same connection without explicitly switching the current database. This allows for cross database queries.
For more information on performing operations in the shell, see:

Keyboard Shortcuts

The mongo shell provides most keyboard shortcuts similar to those found in the bash shell or in Emacs. For some functions mongo provides multiple key bindings, to accommodate several familiar paradigms.
The following table enumerates the keystrokes supported by the mongo shell:
KeystrokeFunction
Up-arrowprevious-history
Down-arrownext-history
Homebeginning-of-line
Endend-of-line
Tabautocomplete
Left-arrowbackward-character
Right-arrowforward-character
Ctrl-left-arrowbackward-word
Ctrl-right-arrowforward-word
Meta-left-arrowbackward-word
Meta-right-arrowforward-word
Ctrl-Abeginning-of-line
Ctrl-Bbackward-char
Ctrl-Cexit-shell
Ctrl-Ddelete-char (or exit shell)
Ctrl-Eend-of-line
Ctrl-Fforward-char
Ctrl-Gabort
Ctrl-Jaccept-line
Ctrl-Kkill-line
Ctrl-Lclear-screen
Ctrl-Maccept-line
Ctrl-Nnext-history
Ctrl-Pprevious-history
Ctrl-Rreverse-search-history
Ctrl-Sforward-search-history
Ctrl-Ttranspose-chars
Ctrl-Uunix-line-discard
Ctrl-Wunix-word-rubout
Ctrl-Yyank
Ctrl-ZSuspend (job control works in linux)
Ctrl-H (i.e. Backspace)backward-delete-char
Ctrl-I (i.e. Tab)complete
Meta-Bbackward-word
Meta-Ccapitalize-word
Meta-Dkill-word
Meta-Fforward-word
Meta-Ldowncase-word
Meta-Uupcase-word
Meta-Yyank-pop
Meta-[Backspace]backward-kill-word
Meta-<beginning-of-history
Meta->end-of-history

Queries

In the mongo shell, perform read operations using the find() and findOne() methods.
The find() method returns a cursor object which the mongo shell iterates to print documents on screen. By default, mongo prints the first 20. The mongo shell will prompt the user to “Type it” to continue iterating the next 20 results.
The following table provides some common read operations in the mongo shell:
Read OperationsDescription
db.collection.find(<query>)
Find the documents matching the <query> criteria in the collection. If the <query> criteria is not specified or is empty (i.e {} ), the read operation selects all documents in the collection.
The following example selects the documents in the users collection with the name field equal to "Joe":
coll = db.users;
coll.find( { name: "Joe" } );
For more information on specifying the <query>criteria, see Specify Equality Condition.
db.collection.find(<query>,<projection>)
Find documents matching the <query> criteria and return just specific fields in the <projection>.
The following example selects all documents from the collection but returns only the name field and the _idfield. The _id is always returned unless explicitly specified to not return.
coll = db.users;
coll.find( { }, { name: true } );
For more information on specifying the <projection>, see Project Fields to Return from Query.
db.collection.find().sort(<sort order>)
Return results in the specified <sort order>.
The following example selects all documents from the collection and returns the results sorted by the namefield in  ascending order (1). Use -1 for descending order:
coll = db.users;
coll.find().sort( { name: 1 } );
db.collection.find(<query>).sort(<sortorder>)Return the documents matching the <query> criteria in the specified <sort order>.
db.collection.find( ... ).limit( <n> )Limit result to <n> rows. Highly recommended if you need only a certain number of rows for best performance.
db.collection.find( ... ).skip( <n> )Skip <n> results.
db.collection.count()Returns total number of documents in the collection.
db.collection.find(<query>).count()
Returns the total number of documents that match the query.
The count() ignores limit() and skip(). For example, if 100 records match but the limit is 10,count() will return 100. This will be faster than iterating yourself, but still take time.
db.collection.findOne(<query>)
Find and return a single document. Returns null if not found.
The following example selects a single document in the users collection with the name field matches to "Joe":
coll = db.users;
coll.findOne( { name: "Joe" } );
Internally, the findOne() method is the find()method  with a limit(1).
See Query Documents documentation for more information and examples. See Query and Projection Operators to specify other query operators.

Error Checking Methods

Changed in version 2.6.
The mongo shell write methods now integrates the Write Concern directly into the method execution rather than with a separate db.getLastError() method. As such, the write methods now return a WriteResult() object that contains the results of the operation, including any write errors and write concern errors.
Previous versions used db.getLastError() and db.getLastErrorObj() methods to return error information.

Administrative Command Helpers

The following table lists some common methods to support database administration:
JavaScript Database Administration MethodsDescription
db.cloneDatabase(<host>)Clone the current database from the <host> specified. The <host> database instance must be in noauth mode.
db.copyDatabase(<from>, <to>, <host>)
Copy the <from> database from the <host> to the <to> database on the current server.
The <host> database instance must be in noauth mode.
db.fromColl.renameCollection(<toColl>)Rename collection from fromColl to <toColl>.
db.repairDatabase()Repair and compact the current database. This operation can be very slow on large databases.
db.getCollectionNames()Get the list of all collections in the current database.
db.dropDatabase()Drops the current database.
See also administrative database methods for a full list of methods.

Opening Additional Connections

You can create new connections within the mongo shell.
The following table displays the methods to create the connections:
JavaScript Connection Create MethodsDescription
db = connect("<host><:port>/<dbname>")
Open a new database connection.
conn = new Mongo()
db = conn.getDB("dbname")
Open a connection to a new server using newMongo().
Use getDB() method of the connection to select a database.
See also Opening New Connections for more information on the opening new connections from the mongo shell.

Miscellaneous

The following table displays some miscellaneous methods:
MethodDescription
Object.bsonsize(<document>)Prints the BSON size of a <document> in bytes