Ubuntu 12.04 and SOLR 3.6.2 install

Solr installation

Hi, I thought I would revisit my solr install to try and clarify a few points where people were having problems loading the solr-cell class and getting binary files to index, with errors such as these:

Error loading class    
    'org.apache.solr.handler.dataimport.DataImportHandler'

So, hopefully by following this tutorial you should get a working solr install:

Install base OS, 64 bit variant, in this case Ubuntu 12.04 64 bit

create a solr user and log in with that user

Go to 'Software centre' and install Synaptic Package Manager
install tomcat6 + tomcat6-common + tomcat6-admin

sudo apt-get install tomcat6
sudo apt-get install tomcat6-admin

Modify the Tomcat admin configuration to allow a new manager login.

sudo gedit /etc/tomcat6/tomcat-users.xml

Add the following to the file, commenting out other similar lines

Restart tomcat

sudo service tomcat6 restart


Check the root of Tomcat works by visiting: http://localhost:8080
Then log into the Tomcat manager page by visiting http://localhost:8080/manager/html and entering your Tomcat username and password that you setup previously

Prepare the solr environment

Create a temp folder for the solr binary

cd /home/solr/
mkdir tmp/solr/

Navigate to that folder

cd tmp/solr/

download the relevant  Apache Lucene solr binary using WGET (you will need to get the  URL for the version you wish to install)

wget http://apache.mirrors.timporter.net/lucene/solr/3.6.2/apache-solr-3.6.2.tgz

Decompress the binary file, copy files into their relevant locations and give the tomcat user rights to the working folder

tar xzvf apache-solr-3.6.2.tgz
cd apache-solr-3.6.2
sudo cp dist/apache-solr-3.6.2.war /var/lib/tomcat6/webapps/solr.war
sudo cp -fr example/solr /var/lib/tomcat6/
sudo chown -R tomcat6:tomcat6 /var/lib/tomcat6/solr

Restart tomcat6

sudo service tomcat6 restart

Set the solr home directory

sudo gedit /var/lib/tomcat6/webapps/solr/WEB-INF/web.xml

Modify the file accordingly, by uncommenting the solrhome env entry and adding the line below:

/var/lib/tomcat6/solr/

Restart tomcat6

sudo service tomcat6 restart

Check you can start solr

http://localhost:8080/solr

Setup solr-cell for binary file (word, pdf etc) indexing

First we need to load the solr-cell module in solrconf.xml

sudo gedit /var/lib/tomcat6/solr/conf/solrconfig.xml

Find and change the solr-cell line to match the following:

lib dir="/var/lib/tomcat6/webapps/solr/WEB-INF/lib/" regex="apache-solr-cell-\d.*\.jar"

Save the file

Now we must copy all the relevant .jar files into the classpath library

cd /home/solr/tmp/solr/apache-solr-3.6.2/

sudo cp contrib/extraction/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/
sudo cp contrib/dataimporthandler/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/
sudo cp dist/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/


Restart tomcat6

sudo service tomcat6 restart

Test our solr installation

First we must install the curl command to send binary files via http to the request handler

sudo apt-get install curl

Create a text (non image) PDF file by creating a text file in Gedit with a keyword, such as 'otter' and printing the result to the file test.pdf (save this in your /home/documents/ folder)
Now we will take the test file and 'curl' it into solr

cd /home//Documents/

curl "http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true" -F "myfile=@test.pdf"

Now test to see if you can search the index for the id

From a remote PC enter the following URL:
http://:8080/solr/admin
In the query string box enter the following:

id:smoketest

Press search and the results should show a match for the 'numFound' variable. This shows 1 matching hit for 'smoketest' so we can confirm the system is working correctly.

Once the basic system is running it is time to apply the custom solr schema (if you have/need one). The schema resides in the following location:
/var/lib/tomcat6/solr/conf/schema.xml

More misc linux stuff

cool linux apps

lmms
handbrake
audacity
xbmc
hugin
htop - awesome coloured version of top

how to make a bitnami stack executable (bin file)
chmod 755 filename.bin
./filename.bin

check packet flow on linux with
iptstate
this uses the libnetfilter_conntrack part of linux kernel
It analyses all nics
If you can see traffic on a particular port and you want to see if it is coming from your machine check your /etc/services file to see if a known service is listed for it.
If you need more info use netstat:
netstat -nap
(you can pipe this to grep to analyse specific ports)
netstat -nap | grep ":22\s"

Look at devices
/sbin/lspci
or
/sbin/lspci -vv
for more detail

look for usb only

/sbin/lspci | grep -i usb

you can also run

/sbin/lsusb

Kernel modules are in
/lib/modules//kernel
find the loaded ones using
lsmod

To find out what has been plugged or unplugged query the kernel with
dmesg

check disk space
df

Check io
iostat
or
iostat 5 5 (try 5 times with 5 second pause)

check processes
ps (plus switches)

create a folder
mkdir
or create nested folders
mkdir -p top/onedown/twodown

copy files
cp cheese.txt /folder/cheese.txt


send mail from the command line in linux (ubuntu)

sudo apt-get install ssmtp

Then edit the ssmtp configuration file:

sudo gedit /etc/ssmtp/ssmtp.conf

at the very least set the mailhub
Mailhub=foo.bar.thing.com

to enable 'mail' from the command line install the mail utils:
sudo apt-get install mailutils

then from the command line, any command can be piped to mail:

ls | mister@cheese.com

Compile a linux application from source:
Download source files
unpack to a working directory:
tar xzvf nameoffile.tgz
cd to this folder
To compile:
./configure
make
su -c "make install"

Apps to configure Linux iptables with a GUI:
www.fwbuilder.org
www.simonzone.com/software/guarddog
www.shorewall.net

APT commands

apt-get update
apt-get upgrade
apt-get install
apt-get remove
apt-get clean
apt-get autoclean
apt-get autoremove

Ubuntu 12.04 and SOLR 3.6 Install + TIKA

Ubuntu 12.04 LTS 64 bit Apache solr Tomcat 6 install with Tika and other JARS

We have chosen this particular Linux variant for it's long term support options for updates etc.

• Install base OS, 64 bit variant
• install tomcat6 + tomcat6-common + tomcat6-admin

sudo apt-get install tomcat6
sudo apt-get install tomcat6-admin

• Modify the admin configuration to allow a new login.

sudo service tomcat6 restart

• Check Tomcat works by visiting: http://localhost:8080
• Prepare the solr environment
• Create a temp folder for the solr binary

mkdir -p ~/tmp/solr/

• Navigate to that folder

cd ~/tmp/solr/

• download the latest Apache Lucene solr binary using WGET (you will need to get the latest URL for the version you wish to install)

wget http://apache.ziply.com/lucene/solr/3.6.0/apache-solr-3.6.0.tgz

• Decompress the binary file

tar xzvf apache-solr-3.6.0.tgz

• Create a folder for the main solr files (this is where the main SOLR config files are, not the expanded webapp)

sudo mkdir -p /var/solr 

• Copy the compressed solr webapp into the /var/solr folder

sudo cp apache-solr-3.6.0/dist/apache-solr-3.6.0.war /var/solr/solr.war

• copy the main solr files into /var/solr (you can either copy the default examples SOLR or multicore)

sudo cp -R apache-solr-3.6.0/example/solr/* /var/solr/

• Give the tomcat6 service owner rights to the /var/solr folder

sudo chown -R tomcat6 /var/solr/ 

• set the SOLR home environment variable (solr needs to know where its config is) and change JAVA memory allocations, set on a per webapp basis

sudo gedit /etc/init.d/tomcat6
JAVA_OPTS="$JAVA_OPTS -Dsolr.home=/var/solr -Xms1g -Xmx4g" 

• Restart tomcat6

sudo service tomcat6 restart

• Tell Tomcat catalina where to get everything and the solr webapp is extracted and run from:

/var/lib/tomcat6/webapps/solr/ 

echo -e '\n\n' | sudo tee -a /etc/tomcat6/Catalina/localhost/solr.xml
echo 'TOMCAT6_SECURITY=no' | sudo tee -a /etc/default/tomcat6

• As we need to index binary files such as Word and PDF we need to copy extra libraries from your extracted solr binary folders

cd ~/tmp/solr/

sudo cp apache-solr-3.6.1/contrib/extraction/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/

sudo cp apache-solr-3.6.1/contrib/dataimporthandler/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/

• Restart the server

sudo reboot

• We now need to test our solr installation
• First we must install the curl command to send binary files via http to the request handler

sudo apt-get install curl

• Create a text (non image) PDF file by creating a text file in Gedit and printing the result to the file test.pdf (save this in your /home/documents/ folder)
• Now we will take the test file and 'curl' it into solr

cd /home//Documents/

curl "http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true" -F "myfile=@test.pdf"

• If all goes well you should receive output similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">124</int></lst>
</response>

• Now test to see if you can search the index
• From a remote PC enter the following URL:

http://:8080/solr/admin

• In the query string box enter the following:

id:smoketest

• Press search
• If all works well the id will be discovered in the index and results similar to the following will be displayed:

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="indent">on</str>
<str name="start">0</str>
<str name="q">id:smoketest</str>
<str name="version">2.2</str>
<str name="rows">10</str>
</lst></lst>
<result name="response" numFound="1" start="0">
<doc>
<arr name="content_type">
<str>application/pdf</str>
</arr><str name="id">smoketest</str>
</doc>
</result>
</response>

• the 'numFound' variable shows 1 matching hit for 'smoketest' so we can confirm the system is working correctly
• Once the basic system is running it is time to apply the custom solr schema (if you have/need one). The schema resides in the following location:

/var/solr/conf/schema.xml

How to use the GREP command

I recently had to search for a particular string in a bunch of PHP and CGI bin files, other search methods would not have worked so grep came to the rescue!

grep command
grep “text string to search” directory-path

Examples

To search for a string called CHEESE in all text files located in /home/purpleotter/*.txt directory:


$ grep "CHEESE" /home/purpleotter/*.txt

To search all subdirectories recursively

To search for a text string in all files under directories, recursively use the -r option:

$ grep -r "CHEESE" /home/purpleotter

Install SOLR 3.5 on Centos

Install Centos 32 or 64 bit on the relevant hardware or VM accepting all default installation options

Install Tomcat 6.x from the Centos package manager

Modify startup items to make sure Tomcat 6 runs at boot time. Please note startup items are user specific, so if logged in as administrator, root would not have Tomcat startup automatically

Reboot the server

Test if Tomcat is working by opening Firefox and navigating to http://localhost:8080/manager/html

Download the TGZ binary (non src) version of SOLR 3.5 from http://www.mirrorservice.org/sites/ftp.apache.org//lucene/solr/3.5.0/

Decompress file to /OPT/SOLRBINARY (you will need to create this folder)

Copy the whole SOLR folder from /OPT/SOLRBINARY/apache-solr-3.5.0/example/  to /OPT/

Running as root, open a terminal and run the following - CHMOD 777 –R /OPT

Copy the Tomcat SOLR webapp (the .WAR file) from /OPT/SOLRBINARY/apache-solr-3.5.0/dist to /OPT/SOLR/ and rename it to solr.war

Now copy this solr.war file to /USR/SHARE/TOMCAT6/WEBAPPS

From the command line terminal run (as root): service tomcat6 restart

Now, to fix library bugs copy the following from /OPT/SOLRBINARY/apache-solr-3.5.0/contrib./extraction/lib: (where x is version number)

Tika-core-x.jar

Xmlbeans-x.jar

Xml-apis-x.jar

Xercesimpl-x.jar

Copy to /USR/SHARE/TOMCAT6/WEBAPPS/SOLR/WEB-INF/LIB

From the command line terminal run (as root): service tomcat6 restart

Set the solr/home value in the /USR/SHARE/TOMCAT6/WEBAPPS/SOLR/WEB-INF/web.xml file by editing and un-commenting the following and changing the paths to those shown below:

   solr/home

   /OPT/SOLR

   java.lang.String

There is a bug in the 3.5.x velocity writer module and to fix this edit OPT/SOLR/CONF/SOLRCONFIG.XML and comment out line 1554 (which starts ‘Query Response Writer Velocity’)

Copy your custom schema.xml file to /OPT/SOLR/CONF/SCHEMA.XML (if you have one)

From the command line terminal run (as root): service tomcat6 restart

Reboot the server

Test if Solr is running by going to: http://localhost:8080/solr/admin

Copy or create a PDF file locally on the SOLR server and place it in /OPT/SOLR (make sure the PDF is a ‘text’ PDF containing searchable strings

Open a terminal and navigate to /OPT/SOLR

Test an indexing of a file by using curl to upload the PDF file:

curl “http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true –F “myfile=@testpdf.pdf”

Navigate back to http://localhost:8080/solr/admin and in the ‘Query String’ box enter:

id:smoketest

Then press ‘search’

An XML results page should be displayed with your PDF file as the search result

Linux find iname

If you need to quickly find a file or directly but have no idea of case or path, use the following:

find -iname "somefile.txt"

More Linux Permissions

If you need to set folders to 755 and files to 644

sudo chmod -R 644 * ; sudo chmod -R ugo+X *

Determining free disk space in Linux

Just use the df command:

df /

or

df -P

will generate a usage table.

To just extract the 'used' portion:

df=($(LC_ALL=C df -P /)); echo "${df[11]}"

Email the report from a shell script:

df -h | mail -s “disk space report” fromage@cheese.com

If you don't have mail installed, you can use sendmail, which would be:

df-h | sendmail fromage@cheese.com

OR if you want to get the results of this (or any other linux command) in a pop up x window then first redirect the output of a command to a file:

df -h > otterlog.txt

Then open this file in a pop up:

xmessage -file otterlog.txt

Linux APT commands

After initial install run:
sudo apt-get update

To update installed applications:
sudo apt-get upgrade

To install the 'cheese' package:
sudo apt-get install cheese

To remove the 'cheese' package:
sudo apt-get remove cheese

To clean the APT database:
sudo apt-get clean   or sudo apt-get autoclean

To have a good clear up:
sudo apt-get autoremove

Postfix - prevent certain domains from sending to your server

Add the following line in /etc/postfix/main.cf

• smtpd_sender_restrictions = hash:/etc/postfix/access

then modify the access text file in /etc/postfix/ to include the list of rejected senders (similar to below)

• aol.com     REJECT

Then open a terminal window and ‘re-hash’ the access file with the following command

• postmap hash:/etc/postfix/access

Now restart Postfix

Download a website using WGET on Linux

wget --wait=20 --limit-rate=20K -r -p -U Mozilla http://asite.com

Noatime and atime attribute on Linux

Linux records information about when files were created and last modified as well as when it was last accessed. There is a cost associated with recording the last access time. The ext2 file system of Linux has an attribute that allows the super-user to mark individual files such that their last access time is not recorded. This may lead to significant performance improvements on often accessed frequently changing files such as the contents of the /var/spool/news directory. 

Linux has a special mount option for file systems called noatime that can be added to each line that addresses one file system in the /etc/fstab file.

If a file system has been mounted with this option, reading accesses to the file system will no longer result in an update to the atime information associated with the file like we have explained above.

The importance of the noatime setting is that it eliminates the need by the system to make writes to the file system for files which are simply being read.

Since writes can be somewhat expensive, this can result in measurable performance gains. Note that the write time information to a file will continue to be updated anytime the file is written to.

In our example below, we will set the noatime option to our /chroot file system. 

Edit the fstab file vi /etc/fstab and add in the line that refer to /chrootfile system the noatime option after the defaults option as show below:

/dev/sda7          /chroot          ext2          defaults,noatime          1  2

         

You need not reboot your system for the change to take effect, just make the Linux system aware about the modification you have made to the /etc/fstab file. This can be accomplished with the following commands:            

/#mount -oremount /chroot/

 

Linux - split and rejoin large files

In Linux it is possible to split big files, say ISO's into smaller, user defined chunks. For example, to split a 5Gb ISO into 250Mb chunks, open a bash terminal and type:


split -b250m filename.ISO iso-split


When run the filename>ISO file is split into 250 MegaByte chunks and iso-split is prefix of generated file(s).
To restore the file(s) back into one chunk type:


cat iso-split* > NEWFILENAME 


If you need to rejoin the file in windows then go to a command prompt and type:


type filepart1 filepart2 > filefull


Where filepart1, 2 etc are the chunks made by the linux split command

Linux - Run multiple commands together

Just create your script or from command line put && between commands. This logical operator 'and' will run the first command, then if successful will run the second. If the first fails the command chain will stop. i.e.

./configure && make && make install

The other logical operator available is || this means 'or' for example, run a script and if it fails print a warning, i.e.

./some_script || echo "The script isn't working!"

Linux - export system info

Just open up a terminal and do the following:

uname -a >system.txt
lspci >>system.txt
lspci -vv >>system.txt