whats the plot

i like the site for twitter, ‘friendorfollow,’ which tells me who i am following and isn’t following me back, who are following me and i am not following back, and so on. given that i got access to google buzz today, i decided to write a little script to help me determine the breakdown of my friends (specifically, who i am following and isn’t following me back, since the rest of the information is easy to read off of the website anyway).

this script will tell you who your friends are (lol, i wish), who your fans are, and who you are following but isn’t following you back. unfortunately, there’s a manual step involved (getting the server response with your friends and followers). there’s really no point in trying to automate this right now, because soon enough, i expect that one of the google apis will expose this functionality.

download it from github here.

after the nexus one came out, i became a little more interested in android. while i had written a simple ’salam world’ app some time ago, i figure it would be nice to delve a little deeper and try to write something remotely useful.

so without further adieu, introducing a (very simple) quran for android:

source code on github

for the most part, android is fairly fun and easy to develop for. however, one of the most frustrating parts is the extremely limited memory for apps. since apps can’t be on the sd card (due to some security issues), you are limited to the very small amount of memory on the device. consequently, in order to display the images for the quran in this app, i have to require the user to download the images to their sd card (or do it for them over the web).

note – try it at your own risk, i only tested it on the emulator…

so i had a little bit of time before i depart on my journey for hajj, so i wanted to play around with the firefox jetpack extension a little bit. so without further adieu, this post is here to introduce a simple quran plugin for jetpack

what is it
this is a quran script for jetpack, a plugin for firefox which allows customizing the browser just through javascript.

when you are typing in a textarea (in wordpress, gmail, google talk, etc), if you highlight any text in the format of sura:ayah (ex 1:1) and right click on it, you will be given a menu that allows you to replace that with the actual verse (in arabic, transliteration, or translation).

this is very nifty for taking notes, chatting, sending emails, etc.

how does it look like?
here is a screenshot:

known issues
- sometimes, the server appears to take long to reply so it doesn’t return and you are forced to try again.
- the plugin doesn’t work on google docs and other sites that overwrite right click functionality.
- the plugin also doesn’t seem to be working on gmail when rich formatting is on.

how to install it
- install jetpack for firefox.
- go to this page. on the top right corner of the webpage, you will find a button that says, “Install…”. click it.
- scroll to the very bottom, click the “auto-update this feature” checkbox, and then click on “i know what i am doing, install it!”

source code
the source is also available as on github in this gist.

changelog
november 16th, 2009
- fixed a bug in which the appended (sura:ayah) to the text was put at the end of the textarea rather than right after the ayah.
- added a little favicon for the menu.

often times while working with databases, you find the need to do some simple tasks (adding a column and populating it with a value from another table, etc). rather than write a script to do this, use mysql’s native subquery (and temporary table) functionality – it makes life much easier.

consider the case where you have two tables:

mysql> describe colors;
+-------+-------------+------+-----+---------+-------+
| Field | Type        | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| id    | int(11)     | YES  |     | NULL    |       |
| color | varchar(10) | YES  |     | NULL    |       |
+-------+-------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

and

mysql> describe color_mapping;
+------------+---------+------+-----+---------+-------+
| Field      | Type    | Null | Key | Default | Extra |
+------------+---------+------+-----+---------+-------+
| picture_id | int(11) | YES  |     | NULL    |       |
| color_id   | int(11) | YES  |     | NULL    |       |
+------------+---------+------+-----+---------+-------+
2 rows in set (0.00 sec)

let’s say you want to update the colors table to add a frequency column such that you know how often a given color is used. it turns out this is really easy using mysql:

alter table colors add frequency int;
update colors set frequency = (select count(*) from color_mapping \
   where color_mapping.color_id = colors.id);

and that’s it. also useful are temporary tables:

create temporary table color_frequencies select color_id, count(*) as cnt \
   from color_mapping group by color_id;

if you then attempt to desc color_frequencies, you’ll see a table with two columns – a color_id and a cnt column.

hopefully this will save some people some efforts writing scripts next time some simple database updates are needed

i’ve done a quick port of the quran ubiquity plugin to work under the new version of ubiquity (0.5). as you may know, ubiquity 0.5 and beyond use a new parser (parser 2) that isn’t compatible with the old parser (parser 1 for 0.1.x versions of ubiquity).

you can get it here. make sure to select “automatically update this feed” so that you get any updates i may get around to making.

this version is pretty much identical to the older one, except that now, you can use “get-ayah,” “get ayah,” or “ayah” to get an ayah, and “search-quran” or “search quran” to do a search. i hope to support some of the new stuff from the pre-alpha version of the quran api soon insha’Allah (other translations, etc).

today, i took the arabeyes php extension of itl, the yahoo geocoding api, and the geonames api and put together waqt.org.

it’s a fairly minimalistic prayertimes site. the code is available on github. note that the calculation method is currently hardcoded to use the isna method, but this is fairly easy to change.

updated and released the first version of the quran ubiquity plugin! you can go here to install it.

essentially, it contains two commands -
1. search-quran – takes a parameter of what to search for and will show the results that match that particular query. hitting enter will bring up the search results page.
2. get-ayah – takes a parameter of which ayah (ex 2:2) and an optional parameter of the language/translation you want the ayah in (in english – muhsin khan, for example – note that ubiquity will provide suggestions for these). hitting enter will insert the text into the selection space.

this is uber-useful for muslims imho :p perhaps i will try to provide a screencast later on that shows how to use this for those who are still afraid to try it

*update* – rather than make my own screencast, i’ve decided to record a set of audio instructions on how to use it.

by the way – if you haven’t used ubiquity before, i highly recommend that you watch this video first. it explains what ubiquity is and gives you an idea of what it is useful for. to put it quite simply, ubiquity is amazing. it’s an indispensable tool for your firefox. watch the video

and here is the audio tutorial on the quran plugin for ubiquity.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

enjoy!

i recently decided to move my tech tips microblog to identi.ca (the original copy was on jaiku), as i felt it was a little more befitting, actively developed, etc (although jaiku is now open source).

anyway… so i wanted to migrate my posts over – so i wrote a php script to do it (it assumes your jaiku is public and reads it without hassling with oauth).

<?php
$sleepTime = 5;
$jaikuSource = "http://username.jaiku.com/json";

$mode = 'identi.ca';
$baseStatusUrl = 'http://identi.ca/api/statuses/update.json';

// thanks, php-twitter
if ($mode == 'twitter'){
   $baseStatusUrl = 'http://twitter.com/statuses/update.json';
   $headers = array('Expect:', 'X-Twitter-Client: ',
      'X-Twitter-Client-Version: ','X-Twitter-Client-URL: ');
}

$ctr = 0;
$entries = array();

print "destination account username: ";
$username = trim(fgets(STDIN));
print "password: ";
system('stty -echo');
$password = trim(fgets(STDIN));
system('stty echo');
print "\n";

$done = false;
$params = '';
while (true){
   $count = 0;
   $posts = fetchUrl($jaikuSource . $params);
   $json = json_decode($posts, true);
   $stream = $json['stream'];
   $lastEntry = null;
   foreach ($stream as $entry){
      if (isset($entry['comment_id'])) continue;
      $lastEntry = $entry;
      $count++;
      $entries[$ctr++] = $entry['title'];
   }
   if ($count == 0) break;
   $lastPostTime = $lastEntry['created_at'];
   $ts = split('-', $lastPostTime);
   $hd = split('T', $ts[2]);
   $min = split('Z', $ts[4]);
   $gmtime = gmmktime($hd[1], $ts[3], $min[0], $ts[1], $hd[0], $ts[0]) - 1;
   $params = "?offset=$gmtime";
}

for ($i = $ctr-1; $i>=0; $i--){
   $params = array('status' => $entries[$i]);
   if ($i != ($ctr-1)){
      print "sleeping $sleepTime seconds\n";
      sleep($sleepTime);
   }
   twitterApiCall($baseStatusUrl, $params);
   print "updated status to: " . $entries[$i] . "\n";
}

function fetchUrl($url){
   $ch = curl_init($url);
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
   $resp = curl_exec($ch);
   curl_close($ch);
   return $resp;
}

function twitterApiCall($url, $args = null){
   global $username, $password, $headers;

   // thanks, php-twitter
   $ch = curl_init($url);
   if (!is_null($args)){
      curl_setopt($ch, CURLOPT_POST, true);
      curl_setopt($ch, CURLOPT_POSTFIELDS, $args);
   }
   if ((!empty($username)) && (!empty($password)))
      curl_setopt($ch, CURLOPT_USERPWD, $username . ':' . $password);
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
   if (!empty($headers))
      curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
   $resp = curl_exec($ch);
   $info = curl_getinfo($ch);
   curl_close($ch);
   if ($info['http_code']!=200)
      print "error - got an http code of: " . $info['http_code'] . "\n";
}

make sure you edit $baseStatusUrl and $mode as necessary. enjoy!

today, i felt like playing some more with ubiquity, which i had installed for a while now but had not played around with sufficiently. i decided to try to write a simple plugin that will search the quran for a particular set of words. to do this, i felt obliged to expose an api for the alpha version of quranicrealm first, which was good because i needed to do it eventually anyway.

and here’s the mandatory screenshot:

it still needs a lot of work… things i still want to do if i get around to it:

• add a favicon (for the site and for the plugin)
• more options (ex, “search english,” or “search transliteration,” etc)
• replace the current text with a link (or translation). this would be useful in im conversations or while writing blog posts.
• a “get-ayah” command (to say, “get ayah 1 of sura fatiha in arabic,” for example).

anyway, i’ll post up the code when i’ve added some improvements insha’Allah. if you want it before then, post a comment.

important update (august 5th, 2009) – so i realized that some of the results here (specifically, the java lucene ones) were incorrect. the reason is because as is mentioned on the lucene wiki, the first search has to initialize the caches. as thus, the results aren’t accurate. this seems to be very true. for example, if i run a test query, discard the results, and then run the real query, results for the three classes of queries are now 17, 8, and 4 ms respectively, which is very comparable (if not sometimes better) than that of sphinx. i will probably need to re-run this benchmark to do a better job of giving the backend systems a level playing field to test on.

update (jan 26th, 2009) – as mentioned in the comments, the mysql results aren’t very accurate either because i was probably not properly searching against the index.

i have a set of ~6000 quotes (verses, if you will), along with a multiple set of translations for each of those verses. before, i was searching across these verses using mysql. while this seemed to work, it was very limiting, and i began looking into alternatives.

so i did a little bit of research and tried out lucene and sphinx. for lucene, i specifically used the zend version (i’ll discuss standard lucene (java) towards the end of this post.)

i’ll show the results first, and then explain them after.

the graph above shows a quick overview of the tests run. a set of 3 different queries were run against 4 different backends. the numbers were generated using apache bench (ab) using 100 requests with a concurrency of 1.

backends:
lucene: this was the first implementation. in it, each verse was a “document.” each translation was a property of the document. the total number of documents was thus equivalent to the number of verses.

sphinx: this was the second sphinx implementation (see sphinx alt below for the first implementation). this implementation was just done to make the data model similar to that of lucene, which is exactly what it is. although this ended up being the fastest (by < 5ms in the tests run), i prefer the sphinx alt implementation because it’s closest to that of the database schema.

sphinx alt: although it is named “sphinx alt” in the graph above, this is really the initial sphinx implementation. in this model, a translation of one verse was a document. consequently, the total number of documents was (number of translations) * (number of verses). i sort of like this one most (even though it’s not the fastest) because it is the closest to the current database schema.

mysql: this is sort of the baseline, and, to be honest, it’s not fair either. the query used here is something in the nature of getting the row where the text is like ‘%word1%word2%’; the number of results returned by this are far fewer (and less valuable) than those returned by either lucene or sphinx. one would need to do “where text like ‘%word1%word2%’ or text like ‘%word2%word1%’” to get a more accurate estimate, but for baseline purposes, i simply ran the first query. note that the query cache size is 0 (ie query cache is on but effectively off for this set of tests). note that the text field has a fulltext index on it.

results:
sphinx wins hands down. however, although it seems that lucene comes in last, this is not really accurate because of the type of mysql query being used. from my limited tests (using a more complicated sql query), lucene and mysql have comparable performance, but lucene of course has the added benefit of more advanced query options, etc.

sphinx times were 8.030 ms, 7.542 ms, 8.324 ms, sphinx alt times were 8.304 ms, 7.898 ms, 11.131 ms, lucene times were 285.759 ms, 116.222 ms, 224.381 ms, and mysql times were 106.254 ms, 106.561 ms, 108.747 ms for queries 1, 2, and 3, respectively. query 1 contained three words (+term1 +term2 +term3), query 2 contained one word (+term4), and query 3 contained two words (+term5 +term6).

additional details:
plain vanilla java lucene is usually faster than zend’s lucene implementation. the largest difference can be noted in indexing times (a few seconds for java versus 15+ minutes in php). if i had to index frequently, i’d use java lucene or sphinx because they are insanely faster.

for example, the first query takes 179.84 ms on average in java (over 100 queries) versus about 272.61 ms on average for php. the second query takes 173.22 ms on average in java versus about 103.30 ms in php. the third query takes 178.98 ms on average in java versus about 214.78 ms in php.

php only won at the second query, which also happens to be the simplest query. two things to note – first, the times here don’t include the jvm or php interpreter start times. these are times reported by taking the time before and after the search call and displaying them. second, unlike the first test, this was all run from the command line and not directly via web (didn’t want to bother setting up tomcat or solr, etc).

just for fun, i implemented the “sphinx alt” data scheme in java lucene as well and re-ran the 3 tests 100 times each. the results were 178.54 ms, 160.20 ms, and 172.72 ms – very much comparable to the results with the alternate schema.

the summary of this very long post in 2 words: sphinx rocks.