Single/Multiple SOLR Cloud 7.5 instances and Zookeeper cluster install on Linux

Install Java (Oracle Open JDK)

Centos

sudo yum install java-1.8.0-openjdk.x86_64
sudo java -version

Ubuntu

sudo apt-get update && apt-get upgrade

sudo apt-get install software-properties-common

sudo add-apt-repository ppa:linuxuprising/java

sudo apt-get update

sudo apt-get install oracle-java11-installer

Install LSOF

Centos

sudo yum install lsof

Ubuntu

sudo apt-get install lsof

Disable Firewall

Centos

sudo firewall-cmd --state
sudo systemctl stop firewalld
sudo systemctl disable firewalld

Ubuntu

sudo ufw disable

Install WGET (WGET is installed by default on Ubuntu)

Centos

yum install wget

Edit File and Process limits

Due to the file handling requirements of SOLR, the number of open process and open file limits need to be increased.

Edit the following file by typing:

sudo vi /etc/security/limits.conf

and add this text to the bottom of the file:

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

Additionally on Centos the 20-nproc.conf file needs to be edited to change the following file  - 4096 to 535535

sudo vi /etc/security/limits.d/20-nproc.conf

Stage 1 - Install Zookeeper (quorum cluster mode)

https://lucene.apache.org/solr/guide/6_6/setting-up-an-external-zookeeper-ensemble.html

Create the Zookeeper user:
sudo groupadd zookeeper
sudo useradd -g zookeeper -d /opt/zookeeper -s /sbin/nologin zookeeper



Download a defined version of Zookeeper from an approved mirror:

sudo wget http://apache.mirror.anlx.net/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz

Extract the archive and copy it to /opt/zookeeper/:
sudo mv zookeeper-3.4.13.tar.gz /opt
cd /opt
sudo tar xzf zookeeper-3.4.13.tar.gz
sudo mv zookeeper-3.4.13 zookeeper

cd zookeeper

sudo mkdir data

sudo chown -R zookeeper:zookeeper /opt/zookeeper/*

Edit the Zookeeper config file and add all of the servers that will be running Zookeeper (this typically needs to be an odd number):

sudo vi /opt/zookeeper/conf/zoo.cfg

and add the following (replacing your server IP's in the red text):

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/data
clientPort=2181
server.1=10.10.10.10:2888:3888
server.2=10.10.10.11.5:2888:3888



Each node in the Zookeeper cluster needs to have an identifier, to do this, we create a file called myid in the zookeeper/data folder on each server. Just create the file and ensure that each server has a unique number that matches the zoo.cfg file. So for example. server.1=10.10.10.10:2888:3888 must have 1 as it's myid:


cd /opt/zookeeper/data

sudo vi myid

modify firewall to allow all port traffic between your Zookeeper/SOLR nodes


Once the firewall rules are in place Zookeeper can be started 
Start Zookeeper on each server:
cd /opt/zookeeper/bin/
sudo ./zkServer.sh start

The zkServer.sh scripts accepts other arguments, as well as start. These are stop, restart and status

(Optional) If you wish to turn this Zookeeper installation into a service and you are running init.d

Setup systemd

vi /usr/lib/systemd/system/zookeeper.service



[Unit]
Description=Zookeeper Service

[Service]
Type=simple
WorkingDirectory=/opt/zookeeper/
PIDFile=/opt/zookeeper/data/zookeeper_server.pid
SyslogIdentifier=zookeeper
User=zookeeper
Group=zookeeper
ExecStart=/opt/zookeeper/bin/zkServer.sh start
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
Restart=always
TimeoutSec=20
SuccessExitStatus=130 143
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then...

systemctl daemon-reload
systemctl enable zookeeper.service
systemctl start zookeeper.service

Test again

cd /opt/zookeeper/bin/
sudo ./zkCli.sh



Stage 2 - Install SOLR

Create a data folder in the root on your drive to store indexes

Create a user called solradmin
sudo mkdir /data

copy your SOLR config files onto your linux servers. Extract the compressed files into /home/solradmin/.

copy your solr.xml file (in this case from /home/solradmin) into /data:
sudo cp /home/solradmin/solr.xml /data

create a solrconfig folder in the root of your system:
sudo mkdir /solrconfig

Download and install SOLR
cd /

sudo wget http://apache.org/dist/lucene/solr/7.5.0/solr-7.5.0.tgz

sudo tar xzf solr-7.5.0.tgz solr-7.5.0/bin/install_solr_service.sh --strip-components=2

sudo ./install_solr_service.sh solr-7.5.0.tgz

sudo service solr restart

We need to customise the Solr startup configuration in /etc/default/solr.in.sh and add the following

Note - add all the Zookeeper nodes in the ZK_HOST parameter by IP or name (these are the same servers you added in the Zookeeper config setup, zoo.cfg):

#Replace these IP's below with your own IP's

ZK_HOST="10.10.10.10:2181,10.10.10.11:2181"

SOLR_OPTS="$SOLR_OPTS -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60"

SOLR_HOME=/data

SOLR_HEAP="30g"

SOLR_JAVA_MEM="-Xms30g -Xmx30g"

Copy the rest of your custom config files and folders (listed below) into /solrconfig (in this case from /home/solradmin/config/):

field-aliases.yml
schema.xml
solrconfig.xml
solr.xml
velocity folder

sudo cp -r /home/solradmin/config/* /solrconfig

sudo chmod -R 777 /data



If all has gone well you should be able to access the default SOLR instance via a browser on port 8983.



Single or multiple SOLR instances

By default, this installation will run one SOLR instance per server on the default port of 8983. If you wish to run additional instances per server on their own port you will need to use a script similar to the following to force an instance on a new port. Create two relevant shell script text files, called solrstart.sh and solrstop.sh, make them executable and use the following to stop and start:







#In this instance we are using incremental port numbers, up from the default instance. This creates an additional two SOLR instances on every server you run the script

 

#solrstart.sh

sudo ./solr start -force -c -p 8984 -s /data/solr2

sudo ./solr start -force -c -p 8985 -s /data/solr3

 

 

#solrstop.sh

sudo ./solr stop -force -c -p 8984 -s /data/solr2

sudo ./solr stop -force -c -p 8985 -s /data/solr3

CentOS 7 - Automatic security updates

In order to enable automatic download and install of critical security updates, carry out the following:

Log on as root and manually update the server:

yum check-update yum update

Now install cron and edit conf:

yum update -y && yum install yum-cron -y   vi /etc/yum/yum-cron.conf   # Edit the following lines to match these values   update_cmd = security update_messages = yes download_updates = yes apply_updates = yes   emit_via = email email_from = root@localhost email_to = root

Please change the values for email to something relevant

Now we start the service:

systemctl start yum-cron systemctl enable yum-cron

Cron jobs on CentOS

To run a cron job every minute you can create a new file in the folder:

/etc/cron.d

The entry in this file should be in the format:

* * * * * root /home/xxadmin/project/ascript.sh

You need a Line Feed (LF) at the end to make sure that cron can read the entry.  Make sure that it is not a Windows line ending (CRLF)

Centos 7 minimal install with basic puppet setup

 Download the CentOS7 minimal ISO and use it to build at least two VM's that can be configured using Puppet


Set and enable the network and host name - you can also edit the IP settings etc at this point as well, or later with the nmtui command

Set root password

Reboot

Update server
sudo yum -y update

Set IP configuration
nmtui
Set IP address to manual and also specify the bitmask to set the gateway - i.e. 10.0.1.201/21
Set both internal and external Google DNS - 8.8.8.8

Restart networking
service network restart

Check hostname
hostname

Check disks
df

Networking options
Add hostnames to DNS (FQDN to be confirmed)
Add resolves to /etc/hosts

10.0.1.201  stage-db
10.0.1.202  stage-solr
10.0.1.250  puppet

Add and install puppet agent or puppet server (depending on role - most will be puppet agent) repos
yum -y install http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
ll /etc/yum.repos.d/

Disable software firewall on all servers
systemctl stop firewalld; systemctl disable firewalld

For server
yum -y install puppet-server

Edit the puppet conf file on master server to set parameters
vi /etc/puppet/puppet.conf

Make changes to the main section and change the highlighted line
[main]
dns_alt_names = puppet, puppet.test.com
certname = puppet
# The Puppet log directory.
# The default value is '$vardir/log'.
logdir = /var/log/puppet
# Where Puppet PID files are kept.
# The default value is '$vardir/run'.
rundir = /var/run/puppet
# Where SSL certificates are kept.
# The default value is '$confdir/ssl'.
ssldir = $vardir/ssl
[agent]
# The file in which puppetd stores a list of the classes
# associated with the retrieved configuratiion. Can be loaded in
# the separate ``puppet`` executable using the ``--loadclasses``
# option.
# The default value is '$confdir/classes.txt'.
classfile = $vardir/classes.txt
# Where puppetd caches the local configuration. An
# extension indicating the cache format is added automatically.
# The default value is '$confdir/localconfig'.
localconfig = $vardir/localconfig

Generate Certs on the master puppet server
sudo puppet master --no-daemonize --verbose
PRESS CTRL + C

Start the puppet server
systemctl start puppetmaster

and ensure service starts automatically
systemctl enable puppetmaster

For agent
yum -y install puppet

Edit the puppet conf file on agent servers to talk to puppet master server
vi /etc/puppet/puppet.conf

Make changes to the agent section and change the highlighted line
[main]
# The Puppet log directory.
# The default value is '$vardir/log'.
logdir = /var/log/puppet
# Where Puppet PID files are kept.
# The default value is '$vardir/run'.
rundir = /var/run/puppet
# Where SSL certificates are kept.
# The default value is '$confdir/ssl'.
ssldir = $vardir/ssl
[agent]
server = puppet
# The file in which puppetd stores a list of the classes
# associated with the retrieved configuration. Can be loaded in
# the separate ``puppet`` executable using the ``--loadclasses``
# option.
# The default value is '$confdir/classes.txt'.
classfile = $vardir/classes.txt
# Where puppetd caches the local configuration. An
# extension indicating the cache format is added automatically.
# The default value is '$confdir/localconfig'.
localconfig = $vardir/localconfig

Sign Certs on the puppet agent server
puppet agent -t

Once all servers (apart from puppet master, have signed, then return to the puppet master server to acknowledge the signing requests
puppet cert list

Then, one by one, sign each cert with - puppet cert sign , for example
puppet cert sign stage-db

start and enable the pupper agents on each server
systemctl start puppet
systemctl enable puppet

Check connectivity with
puppet agent -t

Create a puppet manifest file on the puppet server to make sure changes are pushed to all servers
cd /etc/puppet/manifests
vi site.pp
Paste in this basic config
node default {
file {'/etc/purple':
content => 'This is an Purple test',
}
}
Log into another server and check whether the content has been pulled
puppet agent -t

Install Docker Community Edition on CentOS7

# Install Docker Community Edition on CentOS7

# Install docker dependencies and daemon

sudo yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce

# Set Docker to start on boot

sudo systemctl enable docker

# Add a non-root user to the Docker group

sudo usermod -aG docker guser # guser is my user in this case

# UPGRADE DOCKER CE
# To upgrade Docker CE, download the newer package file and repeat the installation procedure, using yum -y upgrade instead of yum -y install, and pointing to the new file.

# Test Docker is working

sudo docker run hello-world

# Install Docker compose

sudo yum install epel-release

sudo yum install -y python-pip

sudo pip install docker-compose

sudo yum upgrade python*

docker-compose -v

# Disable the firewall

systemctl disable firewalld

Install SOLR 3.5 on Centos

Install Centos 32 or 64 bit on the relevant hardware or VM accepting all default installation options

Install Tomcat 6.x from the Centos package manager

Modify startup items to make sure Tomcat 6 runs at boot time. Please note startup items are user specific, so if logged in as administrator, root would not have Tomcat startup automatically

Reboot the server

Test if Tomcat is working by opening Firefox and navigating to http://localhost:8080/manager/html

Download the TGZ binary (non src) version of SOLR 3.5 from http://www.mirrorservice.org/sites/ftp.apache.org//lucene/solr/3.5.0/

Decompress file to /OPT/SOLRBINARY (you will need to create this folder)

Copy the whole SOLR folder from /OPT/SOLRBINARY/apache-solr-3.5.0/example/  to /OPT/

Running as root, open a terminal and run the following - CHMOD 777 –R /OPT

Copy the Tomcat SOLR webapp (the .WAR file) from /OPT/SOLRBINARY/apache-solr-3.5.0/dist to /OPT/SOLR/ and rename it to solr.war

Now copy this solr.war file to /USR/SHARE/TOMCAT6/WEBAPPS

From the command line terminal run (as root): service tomcat6 restart

Now, to fix library bugs copy the following from /OPT/SOLRBINARY/apache-solr-3.5.0/contrib./extraction/lib: (where x is version number)

Tika-core-x.jar

Xmlbeans-x.jar

Xml-apis-x.jar

Xercesimpl-x.jar

Copy to /USR/SHARE/TOMCAT6/WEBAPPS/SOLR/WEB-INF/LIB

From the command line terminal run (as root): service tomcat6 restart

Set the solr/home value in the /USR/SHARE/TOMCAT6/WEBAPPS/SOLR/WEB-INF/web.xml file by editing and un-commenting the following and changing the paths to those shown below:

   solr/home

   /OPT/SOLR

   java.lang.String

There is a bug in the 3.5.x velocity writer module and to fix this edit OPT/SOLR/CONF/SOLRCONFIG.XML and comment out line 1554 (which starts ‘Query Response Writer Velocity’)

Copy your custom schema.xml file to /OPT/SOLR/CONF/SCHEMA.XML (if you have one)

From the command line terminal run (as root): service tomcat6 restart

Reboot the server

Test if Solr is running by going to: http://localhost:8080/solr/admin

Copy or create a PDF file locally on the SOLR server and place it in /OPT/SOLR (make sure the PDF is a ‘text’ PDF containing searchable strings

Open a terminal and navigate to /OPT/SOLR

Test an indexing of a file by using curl to upload the PDF file:

curl “http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true –F “myfile=@testpdf.pdf”

Navigate back to http://localhost:8080/solr/admin and in the ‘Query String’ box enter:

id:smoketest

Then press ‘search’

An XML results page should be displayed with your PDF file as the search result