Daily IT Readings

Behind the time when dogwood starts to flower
I work and dance inside long changing days
to find the taste, the marrow of the hour
and twist it like a snake into a phrase
that stings with all the passion of a kiss
and smiles with anger in a lying mask
behind your back and turning in your wrist:
I give you back in blood the thing you ask.

-- Kathleen Raine

Labels: poetry

A group of Uruguay-based P2P researchers recently released an open-source P2P streaming application,GoalBit. It is available for Linux and Windows.

GoalBit is a peer to peer distribution system, capable of distributing high-bandwidth live-content to all network peers preserving its quality. This project follows a bittorrent-like approach where the stream is decomposed into several flows sent by different peers to each client. In order to meassure the peers perceived quality, it is used the recently proposed PSQA (Pseudo-Subjective Quality Assessment) technology.

I just learned that FriendFeed will be acquired by Facebook. So, I will post the backup script, reworked, so you can back up your items.

class FriendFeed {
 
public $mongo;
public $collection;
public $mondo_db_name           = 'ff';
public $mongo_db_collection     = 'items';
public $ff_username             = 'YOUR_FRIENDFEED_USERNAME';

function __construct(){
 //Set up connection to Mongo
 $this->mongo = new Mongo();
 $this->collection = $this->mongo->selectDB( $this->mondo_db_name )->selectCollection( $this->mongo_db_collection );
 $this->collection->ensureIndex( array( "id" => 1 ) );
}

/**
* This function constructs the FF url which reads the user's feed.
* It reads $num messages starting from the $start message.
*/
private function getUrl($start, $num){
 return "http://friendfeed-api.com/v2/feed/".$this->ff_username."?start=$start&num=$num&fof=1&maxcomments=100";
}

/**
* Reads $num messages starting from the $start message.
* It seems like $num=100 is the maximum supported by FF.
*/
public function read($start=0, $num=100){
 $json = file_get_contents($this->getUrl($start,$num));
 return self::conv_obj(json_decode($json));
}

/**
* This function converts the FF json to array.
*/
public static function conv_obj($data){
 if(!is_object($data) && !is_array($data)) return $data;
 if(is_object($data)) $data = get_object_vars($data);
 return array_map(array('FriendFeed','conv_obj'), $data);
}

/**
* This function reads $num messages and inserts them in mongodb
*/
public function update($start=0,$num=100){
 $data = $this->read($start,$num);
 //Process the FF data, updating Mongo with each item
 if (!empty($data))
  foreach ($data['entries'] as $entry){
   //upsert the item in mongo
   $this->collection->update(array("id" => $entry['id']), $entry, true);
  }
}

}

$ff = new FriendFeed();

/*//UNCOMMENT BELOW TO DO BACKUP
$max_num_items = 1600; //change this if you have more items
foreach (range(0,intval($max_num_items/100)) as $i) $ff->update(100*$i);
//*/


/*//UNCOMMENT BELOW TO LIST YOUR 10 LATEST ENTRIES
$cursor = $ff->collection->find()->sort(array( "date" => -1 ))->limit(10);

foreach ($cursor as $value) {
    echo $value['date'].' ('.$value['via']['name'].'): '.html_entity_decode($value['body'])."\n";
}
//*/
?>

I wanted to backup my FriendFeed (FF), because... well, because it backs up a lot of my social activities. So this is the first attempt. I will use MongoDB because it works naturally with JSON, which suits my needs just fine. I will also use PHP.

What follows is dead simple. First a crude wrapper for accessing FF.

class FriendFeed {
   private $user;

   function __construct($user){
       $this->user=$user;
   }

   /**
    * This function constructs the FF url which reads the user's feed.
    * It reads $num messages starting from the $start message.
    */
   private function getUrl($start, $num){
       return "http://friendfeed-api.com/v2/feed/".$this->user."?start=$start&num=$num";
   }

   /**
    * Reads $num messages starting from the $start message.
    * It seems like $num=100 is the maximum supported by FF.
    */
   public function read($start=1, $num=100){
       $json = file_get_contents($this->getUrl($start,$num));
       return self::conv_obj(json_decode($json));
   }

   /**
    * This function converts the FF json to array.
    */
   public static function conv_obj($data){
       if(!is_object($data) && !is_array($data)) return $data;
       if(is_object($data)) $data = get_object_vars($data);
       return array_map(array('FriendFeed','conv_obj'), $data);
   }
}

This wrapper only reads the FF entries and decodes them to array. Now lets use it to read 100 entries and insert them in MongoDB.

//CONFIGURATION
$mondo_db_name = 'YOUR_MONGO_DB_NAME';
$mongo_db_collection = 'YOUR_MONGO_COLLECTION_NAME';
$ff_username = 'FriendFeed_USERNAME';

//Set up connection to Mongo
$m = new Mongo();
$collection = $m->selectDB( $mondo_db_name )->selectCollection( $mongo_db_collection );
$collection->ensureIndex( array( "id" => 1 ) );

//Read data from FF
$ff = new FriendFeed($ff_username);
$data = $ff->read();

//Process the FF data, updating Mongo with each item
if (!empty($data))
   foreach ($data['entries'] as $entry){
       //output new items
       if (!$collection->findOne(array("id" => $entry['id'])))
           echo $entry['date'].":".$entry['body']."\n";

       //upsert the item in mongo
       $collection->update(array("id" => $entry['id']), $entry, true);
   }


//Output number of items in collection
$collcnt = $collection->count();
echo $collcnt."\n";

Make this run once per hour or once per day and you will have a local copy of your FF stream. Of course, this will not download photos and files. I may add an extension for this in the future, in case I need it.

If you need to cycle through all your local FF items:

$cursor = $collection->find();
$cursor = $cursor->sort( array( "date" => 1 ) );

foreach ($cursor as $id => $value) {
   var_dump( $value );
}

If you need to backup all your items, you need to replace $data = $ff->read(); and the code that follows with something like this:

while ($data = $ff->read($start)) {
   //processing code
   $start+=100;
}

This code (without the while loop) has been tested and works both on Linux and Windows, with PHP 5.3 and MongoDB 0.9.7.

The following story by Herodotus reminds me so much of current politics.

According to Herodotus, the Medes lived independently (autonomon) in scattered villages after they won their freedom from the Assyrians. Ambitious to unite the Medes under his rule, Deioces set about gaining a reputation for honesty. The men in his village grew to trust him and invited him to settle their disputes. As his reputation grew, more and more people submitted their disputes to him until he finally declared he had had enough and would judge no more lawsuits. His withdrawal plunged the country into lawlessness (anomia) and forced the Medes to make him king. Once in office, Deioces demanded that his subjects build him a vast palace at Ecbatana. When the palace was complete, Deioces remained inside to keep himself safe from plots and communicated with his people through messengers. He continued to judge lawsuits, but all cases were now submitted to him in writing so that he could keep his distance from the people.

excerpt from "Solon of Athens New Historical and Philological Approaches", Mnemosyne, Supplements, 272, Brill Publishing, 2006