Dave Dash2012-01-17T21:54:19-08:00http://davedash.com/Dave Dashdd+atom1@davedash.compy vs php: stemming2008-02-16T00:00:00-08:00http://davedash.com/2008/02/16/py-vs-php-stemming<p>I've been porting some PHP to python during SuperHappyDevHouse and was amazed at how little code I needed to write since python makes list manipulation a breeze.</p>
<p>Today I was working on stemming (ala <a href="http://tartarus.org/martin/PorterStemmer/">Porter Stemming algorithm</a>). <a href="http://reviewsby.us/">reviewsby.us</a> uses stemming in the search engine to make queries:</p>
<p>Stemming turns <code>hello everybody how are you guy's</code> into a collection <code>'everybodi', 'gui', 'hello'</code>. To produce this in php I do the following:</p>
<!--more-->
<div><textarea name="code" class="php">
public static function stemPhrase($phrase)
{
// remove apostrophe's and periods
$phrase = strtolower(str_replace(array('\'', '.'), null, $phrase));
// split into words
$words = str_word_count($phrase, 1);
// ignore stop words
$words = array_diff($words, STOP_WORDS_ARRAY);
// stem words
$stemmed_words = array();
foreach ($words as $word)
{
$stemmed_words[] = PorterStemmer::stem($word, true);
}
return $stemmed_words;
}
</textarea></div>
<p>With some magic python:</p>
<div><textarea name="code" class="python">
def stem_phrase(phrase):
words = phrase.lower().replace('.','').replace("'",'').split()
# ignore stop words
words = list(set(words)-set(STOP_WORDS))
p = PorterStemmer()
return [p.stem(word,0,len(word)-1) for word in words]
</textarea></div>
<p>The magic here is list mappings. Learning about them, they don't seem that great, but as soon as you start coding you stop using a lot of for loops.</p>
<p>I'm sure my PHP can be cleaned up and reduced as well, but its fun exploiting the magic of languages.</p>