SILPA¶
SILPA is an acronym of Swathanthra(Mukth, Free as in Freedom) Indian Language Processing Applications. Its a web framework written using Flask micro framework for hosting various Indian language computing algorithms written in python. It currently provides JSONRPC support which is also used by web framework itself to input data and fetch result.
The modules work as standalone python packages which will serve their purpose and also they plug into the silpa-flask webframewok so that they can be accessed as web services also, or become just another webapp like the dictionary module.
Contents:
Install Instructions¶
Note that this is currently a work in progress. So things may not work as you expected it to work. If you want to test this you need to have following installed on your system
- pip
- Virtualenv
Test deployment
Installing VirtualEnv¶
If you are on Mac OS X or Linux, chances are that one of the following two commands will work for you:
$ sudo easy_install virtualenv
or even better:
$ sudo pip install virtualenv
One of these will probably install virtualenv on your system. Maybe it’s even in your package manager. If you use Ubuntu, try:
$ sudo apt-get install python-virtualenv
Once you have virtualenv installed, just fire up a shell-session and create your own environment.
$ virtualenv silpa
New python executable in silpa/bin/python
Installing distribute............done.
Note
You may want to use --system-site-packages option while creating virtualenv as some of dependencies like pango and cairo will not install cleanly from pip.
Now, whenever you want to work on a project, you only have to activate the corresponding environment. On OS X and Linux, do the following:
$ . silpa/bin/activate
If you are a Windows user, the following command is for you:
$ silpa\scripts\activate.bat
Either way, you should now be using your virtualenv (notice how the prompt of your shell-session has changed to show the active environment).
Get the Code¶
$ git clone https://github.com/Project-SILPA/Silpa-Flask.git
$ cd Silpa-Flask
$ pip install -r requirements.txt
$ python silpa.py
If you want to Install all modules:
$ pip install -r requirement-modules.txt
Note
Previously we were suggesting use of modules.txt but now we are unable properly update the pypi modules in time so we suggest use of head of git repo. But note that this might lead to some breakage.
To enable module a line should be added to silpa.conf. By default all modules will be enabled if you don’t want this behavior mark no infront of module name under modules section. modules_display section is used to display a text in the side bar of SILPA main page. Tweak it if required.
Warning
normalizer and silpa_common modules are helper modules which is required by the current modules. Do not add a line to silpa.conf for this module. Its not a web module pure python module
Now you can just enter the following command to get Flask activated in your virtualenv:
$ pip install Flask
A few seconds later and you are good to go.
You can start the silpa application by
python silpa.py
Running on http://127.0.0.1:5000/
Hosting the SILPA on Webserver¶
SILPA can also be hosted on webserver like apache or nginx. Here we utilize the uwsgi containers to contain our application. It is also possible to use other methods which are commonly used to host WSGI application. But here we give example of uWSGI for hosting which we use on our servers.
For installing uWSGI on Debian based derivatives use following command.
$ sudo aptitude install uwsgi uwsgi-plugin-python
Now we need to create separate user for running silpa, its not recommended to run uWSGI as root. For creating a user for SILPA run following command
$ sudo adduser --disabled-login --disabled-password silpa
Note
You can use different user name. Above is just an example for creating new uid.
Once installed and uid is created you need to place following configuration file under /etc/uwsgi/appas-available/silpa.ini
[uwsgi]
single-interpreter = True
plugins = 0:python
chmod = 766
uid = silpa
wsgi-file = /path/to/silpa-flask/dispatch.wsgi
virtualenv = /path/to/virtuanenv/created
docroot = /path/to/silpa-flask
touch-reload = %(docroot)/dispatch.wsgi
touch-reload = %(docroot)/silpa.py
touch-reload = %(docroot)/webbridge.py
touch-reload = %(docroot)/loadconfig.py
touch-reload = %(docroot)/silpa.conf
callable = application
backtrace-depth = 4
master = True
workers = 4
cheaper = 2
threads = 2
Now we need to enable this configuration for doing this we need to symlink above file to /etc/uwsgi/apps-enabled. Run following commands to do this.
cd /etc/uwsgi/apps-enabled/
sudo ln -s ../uwsgi/apps-available/silpa.ini
Now to start the application container for SILPA run following comand.
sudo service uwsgi start silpa
Now you can use apache2 or nginx as front end for your uWSGI container. A example apache2 conf which we use is below.
NameVirtualHost *:80
<VirtualHost *:80>
ServerName dev.silpa.org.in
ServerAlias www.dev.silpa.org.in
ServerAlias silpa.org.in
ServerAlias dev.silpa.org.in
ServerAdmin silpa-discuss@nongnu.org
DocumentRoot /path/to/silpa-flask
<Location />
SetHandler uwsgi-handler
uWSGISocket /var/run/uwsgi/app/silpa/socket
uWSGImaxVars 512
</Location>
ErrorLog ${APACHE_LOG_DIR}/dev.silpa.org.in/error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/dev.silpa.org.in/access.log combined
</VirtualHost>
Note
All above hosting documentation is suited for Debian derivatives. If you are using some other distribution you might need to tweak it suit your distribution files.
Module structure for SILPA¶
To host your existing code on SILPA you need to do small changes to allow SILPA framework to get some information out of your module. The important information needed by SILPA from module is its instance and template for rendering the web page.
Returning module instance¶
First information that every module which is to be hosted using SILPA framework requires to be able to provide its own instance using a function called getInstance.
An example code from soundex module is shown below
def getInstance:
return Soundex()
Template for Rendering on Webpage¶
All modules to be hosted on SILPA should also provide a template which will be rendered to user who can utilize the module from web. For this each module should include templates folder with a html file with same name as module. i.e. module.html.
If you consider soundex module as an example, you will notice templates folder contains soundex.html content.
{% extends "silpa.html" %}
{% block modulescript %}
<script type="text/javascript">
$(document).ready(function() {
$("form").submit(function(event)
{
event.preventDefault();
id_result = $('#result');
id_error = $('#errormessage');
id_progress = $('#progress');
var word1 = $("input[name=word1]", 'form').val();
var word2 = $("input[name=word2]", 'form').val();
var jsonRequest = {
"method" : "soundex.compare",
"params" : [word1,word2],
"id" : ""
};
$.ajax({
type: "POST",
contentType: "application/json; charset=utf-8",
url: "{{ url_for(main_page + '/JSONRPC' if not main_page == '/' else main_page + 'JSONRPC') }}",
data: JSON.stringify(jsonRequest),
dataType: "json",
beforeSend:function(){
id_result.hide();
id_error.hide();
id_progress.html("Processing. Please Wait..").show();
},
success: function(msg) {
id_progress.hide();
if(msg.result==0)
id_result.html("Both words are same");
if(msg.result<0)
id_result.html("Words does not match");
if(msg.result==1)
id_result.html("Both words are from same language and sounds alike");
if(msg.result==2)
id_result.html("Words are from different languages but sounds alike");
id_result.show();
},
error: function(msg) {
id_progress.hide();
id_error.html("Something went wrong. Please try again!").show();
}
});
});
$("input[type=button]", 'form').click(function(e){
var id_clicked = e.target.id;
if(id_clicked=="button1")
var word= $("input[name=word1]", 'form').val();
else
var word= $("input[name=word2]", 'form').val();
var jsonRequest = {
"method" : "soundex.soundex",
"params" : [word],
"id" : ""
};
id_error = $('#errormessage');
id_progress = $('#progress');
$.ajax({
type: "POST",
contentType: "application/json; charset=utf-8",
url: "{{ url_for(main_page + '/JSONRPC' if not main_page == '/' else main_page + 'JSONRPC') }}",
data: JSON.stringify(jsonRequest),
dataType: "json",
beforeSend:function(){
id_error.hide();
id_progress.html("Processing. Please Wait..").show();
},
success: function(msg) {
id_progress.hide();
// Render it
if(id_clicked=="button1")
$('#soundex1').html(msg.result).show();
else
$('#soundex2') .html(msg.result).show();
},
error: function(msg) {
id_progress.hide();
id_error.html("Something went wrong. Please try again!").show();
}
});
});
});
</script>
{% endblock %}
{% block content %}
<div class="well">
<h3>Indic Soundex</h3>
<p>Soundex Phonetic Code Algorithm Demo for Indian Languages. Supports all indian languages and English. Provides intra-indic string comparison</p>
<ul>
<li><a href="http://thottingal.in/blog/2009/07/26/indicsoundex/">Read More about the algorithm </a></li>
<li><a href="apis.html#soundex">Read about the JSON-RPC based APIs of SILPA for this service </a></li>
</ul>
<form id="soundex_form" action="" method="post">
<span class="help-block">To get the soundex code for a word enter a word in the text box and press Soundex button.</span>
<input type="text" name="word1"/>
<br/>
<input type="button" value="Soundex" class="btn" id="button1" />
<br/>
<br/>
<div id="soundex1" class="alert alert-info hide"></div>
<input type="text" name="word2"/>
<br/>
<input type="button" value="Soundex" class="btn" id="button2" />
<br/>
<br/>
<div id="soundex2" class="alert alert-info hide"></div>
<p class="help-block">To compare two words, enter the words in the below text boxes and press Compare button.</p>
<input type="submit" id="compare" value="Compare" class="btn"/>
</form>
<div id="progress"></div>
<div id="successmessage" class="alert alert-success hide"></div>
<div id="errormessage" class="alert alert-error hide"></div>
<div id="result" class="alert alert-info hide"></div>
</div>
<hr/>
<div class="well">
<h3 name="soundex">Python Soundex API</h3>
This service provides indic soundex algorithm based soundex codes for a word
<ul>
<li>Method: modules.Soundex.soundex
<ul>
<li>arg1 : the word</li>
<li>Return : The soundex code for the word</li>
</ul>
</li>
<li>Method: modules.Soundex.compare
<ul>
<li>arg1 : first word</li>
<li>arg2 : second word</li>
<li>Return : 0 if both strings are same, 1 if both strngs sounds alike and from same language, 2 if strings are from different langauges but sounds alike</li>
</ul>
</li>
</ul>
Sample usage is given below.
<pre class="code">>>>print silpaService.soundex.soundex("കാര്‍ത്തിക്")
കAPKKBF0
>>>print silpaService.soundex.compare("കാര്‍ത്തിക്","கார்திக்")
2</pre>
</div>
{% endblock %}
You will notice that above template uses Jinja templating system and extends silpa.html it also contains javascript function to utilize JSONRPC exposed by SILPA framework to post data and fetch result. For data input a form is provided.
Note
SILPA framework uses Bootstrap3 for UI designing so consider using same elements and structure as SILPA to get uniformity in design. Also make sure this template data gets included when you package your module for pypi.
SILPA Webservice APIs¶
Note
The old listMethods API call for JSONRPC is not yet implemented in new version see issue #15.
Introduction¶
Silpa provides a set of webservice APIs over json-rpc protocol. Silpa services can be used from any programming language which has an RPC implementation. The request and response formats of APIs are in json. It is recommended to read the json-rpc documentation if you are not familiar with that.
This page explains the available usage and sample usage from python application. Implementing this in other programming languages should not be difficult and should be a matter of changing the programming language syntax.
For using the APIs you need the jsonrpc library of python. You can get this from here.
Concepts¶
JSON-RPC wraps an object, allowing you to call methods on that object and get the return values. It also provides a way to get error responses. The specification goes into the details (though in a vague sort of way). Here’s the basics:
All access goes through a POST to a single URL.The POST contains a JSON body that looks like
{
"method": "methodName",
"id": "arbitrary-something",
"params": [arg1, arg2, ...],
}
The id parameter is just a convenience for the client to keep track of which response goes with which request. This makes asynchronous calls (like an XMLHttpRequest) easier. We just send the exact same id back as we get, we never look at it.
The response is JSON. A successful response looks like
{
"result": the_result,
"error": null,
"id": "arbitrary-something",
}
The error response looks like
{
"result": null,
"error": {"name": "JSONRPCError",
"code": (number 100-999),
"message": "Some Error Occurred",
"error": "whatever you want\n(a traceback?)"},
"id": "arbitrary-something",
}
It doesn’t seem to indicate if an error response should have a 200 response or a 500 response. So as not to be completely stupid about HTTP, we choose a 500 resonse, as giving an error with a 200 response is irresponsible.
Usage¶
A sample usage using python can be seen below
from jsonrpclib import Server
proxy = Server("http://silpa.org.in/JSONRPC")
print proxy.scriptrender.render_text("Some text to render", "svg",
100,200)
A sample usage using PHP can be seen below
define ('HOSTNAME', 'http://silpa.org.in/JSONRPC');
$url = HOSTNAME;
// Open the Curl session
$session = curl_init($url);
// If it's a POST, put the POST data in the body
$postvars = '{"method": "scriptrender.render_text",
"params": ["Your text goes here", "png", 100,100],
"id":"jsonrpc"}';
curl_setopt ($session, CURLOPT_POST, true);
curl_setopt ($session, CURLOPT_POSTFIELDS, $postvars);
// Don't return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
// Make the call
$json = curl_exec($session);
// The web service returns json. Set the Content-Type appropriately
header("Content-Type: application/json");
echo $json;
$obj = json_decode($json,false);
$result = $obj->{"result"};
echo $result;
curl_close($session);
Frequently Asked Questions¶
I am interested in contributing to this application. To whom should I contact?¶
Thanks. Just drop a mail to silpa-discuss@nongnu.org. Please do some homework on SILPA before coming up with questions.
Can I use this application in my machine without having internet access?¶
Yes you can. See the installation instructions.
Can I use this application in windows?¶
Yes you can. See the installation instructions.
Can I host this application in my domain?¶
Yes you can and you are encouraged to do so. Note that you should follow the AGPL licensing terms and conditions.
Who is sponsoring the development of this project?¶
Nobody.
When did this project development start?¶
March 2009.
Is this application available in any GNU/Linux distributions?¶
No. But we would be happy if somebody help to get it packaged.
I found a bug in one module. How can I report?¶
Please report bug against the module / framework repository found on our organization page.
Credits¶
Many people contributed in direct and indirect way in the development of silpa. This page attempts to list their names.
- Baiju. M, Swathanthra Malayalam Computing for his mlsplit program for using it as a base for syllabalizer for many languages.
- Laxminarayan Kamath for testing and feature suggestions.
- Rajeesh Nambiar and Nishan Naseer of SMC for their contributions for Font converter.
- Guess Language module is based on the python implementation by Kent S Johnson of guesslanguage.cpp by Jacob R Rideout for KDE which itself is based on Language::Guess by Maciej Ceglowski.
- Spellcheck dictionaries for many languages are contributed by hunspell dictionaries maintained by language communities.
- The python based spellchecker is a highly customized version of basic python spellchecker by Peter Norvig.
- The python implementation of tex hyphenation algorithm is by Wilbert Berendsen, and it is based on Text::Hyphen of Ruby.
- The sort module uses Python UCA implementation The thirukkural quotes collection for Random Quote module is by Manvendra Bhangui and the original text is taken from Shakthi Kannan’s collection.
- The chanakya quotes is from Chanakya Niti website and prepared by Girish Venkatachalam.
- The fontmap of ASCII Tamil font, Valluvar for encoding converter was contributed by Kevin.
- Shantanu Oak for Rs 10K donation to meet expenses of project development (March 1, 2011)
- Netdotnet.com sponsors our server! Silpa Project thanks netdotnet.com for sponsoring server for silpa.smc.org.in instance.
- SILPA Mirror instances Silpa is mirrored in the following addresses.
- Santhosh Thottingal runs a mirror instance at http://thottingal.in/silpa
- Ashik Salahudeen runs a mirror instance at http://aashiks.in/silpa