Dev Status - 24 August 2007

From: Ben Grodsky <>
Date: Fri, 24 Aug 2007 17:47:17 -0700

I forgot to send this straight-away after the meeting.


* please escalate IT issues to Jay/Grodsky, if you're not hearing back about something and it's holding you back


* proxy

        * tracking down bug QA saw

* atty general

        * mysql encryption

                * done

                * gave Ivan a version yesterday

                * tested w/ audio, image, and video

* miivi

        * working on http download

        * maybe worthwhile to consider this http download stuff for podcast downloads so that http downloader is made more generically

                * for publish stuff, may be worthwhile to use servers as Quality of Service thing

                * tries to go p2p, but if it can't then it goes to http

                * possibly gives bw savings; doesn't help with hd costs


* miivi podcast

        * finished RSS

                * gets all from feeds

                * follows redirects

                * checks for thumbnail + video sizes for validity

                * now all that Colin gives works, except things that are mp3s, flvs, etc.

        * search/query generator working again

        * working on tools for Colin to recategorize

        * Gerald got thumbnails to nonstro??? when it's a feed ???

* slsk windows

        * Linus making much progress, so abandoning slsk win fix

        * giving Linus a couple networks of resources for slsk linux

* ed2k

        * adding 1 more network

* torrents

        * made secondary torrent generator

        * only does autogenerator stuff

        * primary torrent generator prioritizes for BT team and is available at idle for autogenerator

        * this should help alleviate resource issues

Gerald (absent)

* miivi podcast

* adding sites to BT searcher


* shareaza linux

        * memory error resolved

        * controller code changed to add more than 1 ???

        * ??? update insertions

        * fixed issue with remote ???

        * know which packets to send, but not in what order to send them

                * comparing against Jeff code

                * this is leaf-hub protocol


* umg DC

        * uploading audio files for UMG

        * downloading 10% of new hashes seen daily

                * false stat b/c not really comparing a daily thing, should be tracking as sources aggregate for same file, then it should become downloadable....

        * getting ~80 files/day

        * make a list of files, sources and see with counts (similar to data you're giving UMG), if there's a way to compare that list to what's been downloaded to-date, make sure you're getting most of the files with plenty of sources

                * at first, the lists won't match closely ???

                * but the ones with 5+ sources you should be getting

                * if you're not getting those top sourced ones, then we need to figure out why

                        * may be bug that you think it's downloading, but it's not

        * processing ares hashes

                * 2 sets going now: today; and older data

                * more data is getting collected now by Ty, and now the data processes more slowly than it is collected!

                * Wrote multi-threaded data processor, so may build-out to multi-computer processing system

                * may want to create system with 2+ computers handing off data

                        * if 1 computer takes longer than a day and is still processing at the beginning of the next day, then it hands off the new day's data to a second machine

                        * and then the 2nd machine is working on data for 24+ hours, while the first computer is finishing it's 1st day of data

                * consider otherwise optimizing processing code

* atty gen DC

* Univ Studs data feeds


* marketing intel 1.0

        * handed off UMG+ logins

        * dropped amount of processing from MI machines (took that machine's load down from 50% to 5%)

        * wrote script to check what's protected/not-protected so those tracks are displayed/collecting data in MI

        * add note about the 3 weird days of data

* marketing intel 2.0

        * learning curve for Flex/Flash

        * waiting on license


* slsk linux (w/ QA)

        * got more machines from Dylan

        * some of the machines have issues - ticketing w/ IT

        * running on 40 machines now, working pretty well

                * issue with 1 particular track

                * check the length of keywords and make special case for numbers

* Dylan will show you how to make RPMs

* will need new project once QA is satisfied


* BitComet flaw

* BT inflation

        * working on balancing this now

        * adding features

        * torrents successfully posted manually will be inflated (when made on public tracker and marked for

* BT autoposting

        * finished rebalancing AP (nuked Nutter code)

        * did whole project load each night --- way too much work

        * now system is scaled back to do 3 torrents/project

                * constant stream should be better than batches

                * this is where Dylan's help to make another generator came in

* BT auto interdiction

* waiting on ~1000 IPs from Ebert ?!


* global data (AD hooks)

        * intelligent grouping

        * giving precedence to meta tags

        * some tracks repeat, but that's because of meta tag issues

        * have flat data table, just need schema/formatting from Potts

* atty gen

* MI

        * gnutella data weird for 3 days

* umg DC

* real decoy distribution system


* BT real tracker

        * socket issue resolved

        * reactiontorrent

                * still working on website frontend

                * working fine for now

* adding BT sites for poster

        * added monova and extratorrent

        * resolved issue

        * working on torrentmatrix now

        * then working on btjunkie


* umg DC

        * gnutella downloader done

        * get ~9000 hashes daily

        * first 1000 d/l fine

        * then have issue in which can only download 1/8

        * working on optimizing this

        * just make sure on trying again the hashes that you weren't able to d/l, as those grow in sources (theoretically) over time

        * outstanding issue is automating hash downloads every 24 hrs

                * have to update list of hash, # sources every 24 hrs to enter newer hashes in list

        * running on 1 computer now

                * consider adding another machine to get the lesser popular hashes

        * if there's a substantial drop-off at a specific # of sources, what order of magnitude of sources does that drop-off happen at

        * Sujay has been helping


* atty general

        * software ready to e-mail AG about and give them software to run on their computer

* UMG dc

        * 20k files that it's re-trying

        * currently just on 1 computer

        * have to consider some way of trimming the list, adding computers won't solve the problem

        * look at history of successfully downloaded hashes to see what limit should set for cut-off (maybe 3?)

        * sorted by sources primarily, maybe worthwhile to sort secondarily by when hash last seen

* torrent interdiction parser crashing every 10 hrs

        * this issue is causing a problem removing dead torrents

        * might be limptorrent library issue???

        * klugey fix: inserting to db states


* working on bringing SLA up

* working closely with Linus on slsk now

* hiring another team member soon
Received on Fri Sep 14 2007 - 10:55:57 BST

This archive was generated by hypermail 2.2.0 : Sun Sep 16 2007 - 22:19:46 BST