Dev Status - 20 August 2007

From: Ben Grodsky <>
Date: Mon, 20 Aug 2007 12:34:20 -0700

It's been a while since I've sent one of these. Jay's trying to re-focus me on development, so I should be sending these out more regularly after the weekly status meetings. These notes are meant to help you remember what was discussed during the meeting: both your own project(s); and your fellow team members' projects.


* proxmaster - have something for QA to test

* java console

        * integrating http get for podcast downloading

        * web services already setup for using URLs as sources (from Dylan)

        * atty general

                * needs to coordinate with Ivan mysql encryption connection

                * will get passed parameters from Ivan.

                * Just give certificate path for connection


* miivi

        * podcast

                * filters out anything w/o thumbnail

                * if no channel thumb and no episode thumb, then skipped

                * currently 1500 videos in system from Colin's last list version (20-40 podcast feeds)

                * timer every ?? hr indexes

                * some feeds have url redirect and have to deal with this condition (they create black thumbnails)

        * working on recategorize tool for Colin

* soulseek

        * logging on, but didn't see own searches

        * so no update there

* ed2k - no update


* data collection moving along well

* new IPs for BT (not being used on mininova/tpb b/c of active moderating)

* meeting w/ Dr Hahn Thursday & Friday to discuss his findings thus far

* postmaster banned from ticketing system (may resolve mailbombing issue)


* added another site to searcher

* fixed problem for .torrent downloading

* migrating db file


* shareaza

        * problem passing msgs between hubs and leaves

        * changed code to do buffer checks ???

        * socket times out from leaf

                * regular shareaza works - so IP doesn't sound banned

                * possibly mistake in handshake - checking in wireshark

        * no current stability problems with jeff's legacy system


* UMG data

        * storing filenames and track number and dumped a couple weeks of data

        * still processing data automatically for Ares (9-10 hrs to process 1 day's data); some issue with Kad has it behind; gnutella is almost caught up

                * let Jay know if it doesn't seem the data processing will catch up to that day's data

                * may want to split up data processing to 2+ computers

        * set up another computer now (get from Ivan) that can start processing data from yesterday. We should have 2 systems working on ares data processing: one catching up and one working on current data.

* downloading

        * getting 10% of new hashes

        * similar to discussions with Ivan, Sujay and Abishek, download by hashes with most sources first, but keep track of failed download attempts so you can re-try those later.

        * for now, follow Ivan's format of directory structure how to upload to UMG ftp (filename = hash.extension; real filename from p2p will be in database)


* inflation load balanced now for multiple computers

* removed redundancies from Nutter's BT scraped torrent files db (57mm --> 7mm rows)

* redid AP (removing Nutter code and replacing with new Jed code)



        * added logout button to site

        * adding projects to system

        * adding functionality to pass-off project input system to PM team

        * working on errors in geo IP data

        * added Universal tracks (waiting to activate from Sujay's ok - he's adding more machines)


* finished reactiontorrents tracker


* moving to QA

* worked with Sujay to optimize


* UMG downloader

        * try to redownload hashes that may have accumulated more sources over time

        * track how many successful downloads out of hashes tried, so that we can figure out whether more system need to be added

        * working on optimizations with Sujay

        * successfully downloaded 500+ hashes (sorted by sources) out of data set of ~3000 hashes (with >1 source)

        * for now, follow Ivan's format of directory structure how to upload to UMG ftp (filename = hash.extension; real filename from p2p will be in database)


* some servers on DCa aren't working right, fixing those

* made sponsor link change


* atty general

        * has to coordinate with Sergio mysql encryption for atty general investigation

        * will test downloader on local computer and then let Jay know to move-along w/ atty general

* umg ares downloader

        * trying to download everything newly found that day

        * currently downloading a few 100 out of the several thousand found

        * just have to change u/l searcher

* piolet

* looking into mysql encryption with Sergio
Received on Fri Sep 14 2007 - 10:55:59 BST

This archive was generated by hypermail 2.2.0 : Sun Sep 16 2007 - 22:19:47 BST