Ubuntu 12.04 and SOLR 3.6.2 install

Solr installation

Hi, I thought I would revisit my solr install to try and clarify a few points where people were having problems loading the solr-cell class and getting binary files to index, with errors such as these:

Error loading class    
    'org.apache.solr.handler.dataimport.DataImportHandler'

So, hopefully by following this tutorial you should get a working solr install:

Install base OS, 64 bit variant, in this case Ubuntu 12.04 64 bit

create a solr user and log in with that user

Go to 'Software centre' and install Synaptic Package Manager
install tomcat6 + tomcat6-common + tomcat6-admin

sudo apt-get install tomcat6
sudo apt-get install tomcat6-admin

Modify the Tomcat admin configuration to allow a new manager login.

sudo gedit /etc/tomcat6/tomcat-users.xml

Add the following to the file, commenting out other similar lines

Restart tomcat

sudo service tomcat6 restart


Check the root of Tomcat works by visiting: http://localhost:8080
Then log into the Tomcat manager page by visiting http://localhost:8080/manager/html and entering your Tomcat username and password that you setup previously

Prepare the solr environment

Create a temp folder for the solr binary

cd /home/solr/
mkdir tmp/solr/

Navigate to that folder

cd tmp/solr/

download the relevant  Apache Lucene solr binary using WGET (you will need to get the  URL for the version you wish to install)

wget http://apache.mirrors.timporter.net/lucene/solr/3.6.2/apache-solr-3.6.2.tgz

Decompress the binary file, copy files into their relevant locations and give the tomcat user rights to the working folder

tar xzvf apache-solr-3.6.2.tgz
cd apache-solr-3.6.2
sudo cp dist/apache-solr-3.6.2.war /var/lib/tomcat6/webapps/solr.war
sudo cp -fr example/solr /var/lib/tomcat6/
sudo chown -R tomcat6:tomcat6 /var/lib/tomcat6/solr

Restart tomcat6

sudo service tomcat6 restart

Set the solr home directory

sudo gedit /var/lib/tomcat6/webapps/solr/WEB-INF/web.xml

Modify the file accordingly, by uncommenting the solrhome env entry and adding the line below:

/var/lib/tomcat6/solr/

Restart tomcat6

sudo service tomcat6 restart

Check you can start solr

http://localhost:8080/solr

Setup solr-cell for binary file (word, pdf etc) indexing

First we need to load the solr-cell module in solrconf.xml

sudo gedit /var/lib/tomcat6/solr/conf/solrconfig.xml

Find and change the solr-cell line to match the following:

lib dir="/var/lib/tomcat6/webapps/solr/WEB-INF/lib/" regex="apache-solr-cell-\d.*\.jar"

Save the file

Now we must copy all the relevant .jar files into the classpath library

cd /home/solr/tmp/solr/apache-solr-3.6.2/

sudo cp contrib/extraction/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/
sudo cp contrib/dataimporthandler/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/
sudo cp dist/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/


Restart tomcat6

sudo service tomcat6 restart

Test our solr installation

First we must install the curl command to send binary files via http to the request handler

sudo apt-get install curl

Create a text (non image) PDF file by creating a text file in Gedit with a keyword, such as 'otter' and printing the result to the file test.pdf (save this in your /home/documents/ folder)
Now we will take the test file and 'curl' it into solr

cd /home//Documents/

curl "http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true" -F "myfile=@test.pdf"

Now test to see if you can search the index for the id

From a remote PC enter the following URL:
http://:8080/solr/admin
In the query string box enter the following:

id:smoketest

Press search and the results should show a match for the 'numFound' variable. This shows 1 matching hit for 'smoketest' so we can confirm the system is working correctly.

Once the basic system is running it is time to apply the custom solr schema (if you have/need one). The schema resides in the following location:
/var/lib/tomcat6/solr/conf/schema.xml

Ubuntu 12.04 and SOLR 3.6 Install + TIKA

Ubuntu 12.04 LTS 64 bit Apache solr Tomcat 6 install with Tika and other JARS

We have chosen this particular Linux variant for it's long term support options for updates etc.

• Install base OS, 64 bit variant
• install tomcat6 + tomcat6-common + tomcat6-admin

sudo apt-get install tomcat6
sudo apt-get install tomcat6-admin

• Modify the admin configuration to allow a new login.

sudo service tomcat6 restart

• Check Tomcat works by visiting: http://localhost:8080
• Prepare the solr environment
• Create a temp folder for the solr binary

mkdir -p ~/tmp/solr/

• Navigate to that folder

cd ~/tmp/solr/

• download the latest Apache Lucene solr binary using WGET (you will need to get the latest URL for the version you wish to install)

wget http://apache.ziply.com/lucene/solr/3.6.0/apache-solr-3.6.0.tgz

• Decompress the binary file

tar xzvf apache-solr-3.6.0.tgz

• Create a folder for the main solr files (this is where the main SOLR config files are, not the expanded webapp)

sudo mkdir -p /var/solr 

• Copy the compressed solr webapp into the /var/solr folder

sudo cp apache-solr-3.6.0/dist/apache-solr-3.6.0.war /var/solr/solr.war

• copy the main solr files into /var/solr (you can either copy the default examples SOLR or multicore)

sudo cp -R apache-solr-3.6.0/example/solr/* /var/solr/

• Give the tomcat6 service owner rights to the /var/solr folder

sudo chown -R tomcat6 /var/solr/ 

• set the SOLR home environment variable (solr needs to know where its config is) and change JAVA memory allocations, set on a per webapp basis

sudo gedit /etc/init.d/tomcat6
JAVA_OPTS="$JAVA_OPTS -Dsolr.home=/var/solr -Xms1g -Xmx4g" 

• Restart tomcat6

sudo service tomcat6 restart

• Tell Tomcat catalina where to get everything and the solr webapp is extracted and run from:

/var/lib/tomcat6/webapps/solr/ 

echo -e '\n\n' | sudo tee -a /etc/tomcat6/Catalina/localhost/solr.xml
echo 'TOMCAT6_SECURITY=no' | sudo tee -a /etc/default/tomcat6

• As we need to index binary files such as Word and PDF we need to copy extra libraries from your extracted solr binary folders

cd ~/tmp/solr/

sudo cp apache-solr-3.6.1/contrib/extraction/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/

sudo cp apache-solr-3.6.1/contrib/dataimporthandler/lib/*.jar /var/lib/tomcat6/webapps/solr/WEB-INF/lib/

• Restart the server

sudo reboot

• We now need to test our solr installation
• First we must install the curl command to send binary files via http to the request handler

sudo apt-get install curl

• Create a text (non image) PDF file by creating a text file in Gedit and printing the result to the file test.pdf (save this in your /home/documents/ folder)
• Now we will take the test file and 'curl' it into solr

cd /home//Documents/

curl "http://localhost:8080/solr/update/extract?literal.id=smoketest&commit=true" -F "myfile=@test.pdf"

• If all goes well you should receive output similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">124</int></lst>
</response>

• Now test to see if you can search the index
• From a remote PC enter the following URL:

http://:8080/solr/admin

• In the query string box enter the following:

id:smoketest

• Press search
• If all works well the id will be discovered in the index and results similar to the following will be displayed:

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="indent">on</str>
<str name="start">0</str>
<str name="q">id:smoketest</str>
<str name="version">2.2</str>
<str name="rows">10</str>
</lst></lst>
<result name="response" numFound="1" start="0">
<doc>
<arr name="content_type">
<str>application/pdf</str>
</arr><str name="id">smoketest</str>
</doc>
</result>
</response>

• the 'numFound' variable shows 1 matching hit for 'smoketest' so we can confirm the system is working correctly
• Once the basic system is running it is time to apply the custom solr schema (if you have/need one). The schema resides in the following location:

/var/solr/conf/schema.xml

Password protect folder in Apache

If you need to setup basic authentication and a protected folder, use the following procedure:

• Setup a basic Apache build with document root set to /htdocs/
• Create the folder you want to protect under /htdocs/ - for example /htdocs/secret/
• Create a text file in your 'secret' folder called - '.htaccess' (no quotes)
• Populate this file with the settings (directives) that you want you 'secret' folder to have, in this case:

AuthType Basic
AuthName "Secret Web"
AuthUserFile /srv/www/.htpasswd
Require valid-user

• Now create a text file called '.htpasswd' (again no quotes) in /srv/www or somewhere other than your Apache document root
• Edit this and using a password encryption generation tool (http://www.tools.dynamicdrive.com/password/) create a username/password combination. It will look something like this:

cheese:y4E7Ep8e7EYV

If you now try to access your 'secret' URL you will be prompted for your username and password.