Twitter developer announces advanced people search upgrades ahead of SXSW2010

Announcement via Twitter

Alex Payne, a developer @Twitter announced today via tweet that work has been completed on a project to enhance the scalability, resiliency and quality of people search results at Twitter. While the engineer tried to downplay the announcement, the impact could be huge for how people find each other on Twitter. I’m going on a hunch, but I am betting that Twitter made the changes in advance of a looming swarm or location-crazed socialites that will be descnding upon Austin TX in just a few days.

SXSW is coming up. Twitter counting on not going down

Sidestepping all the hype surrounding the upcoming festival, most, if not all the attendees are avid Twitter users following hundreds and thousands of other users. What do twitter users do when they drink? Well, besides have sex? They tweet. And tweet. And look up followers. And tweet. The tweeting alone has been known to bring the service down before (hello @FailWhale!) but what about people search? I asked the developers and engineers and have not had a comment back yet, but I would wager again that if it is a priority at Twitter, it has to do with reliability and uptime. These guys are serious about it now.

Web app scalability experts

If you don’t run in global web domination circles, you’re unaware thatAlex Payne (who also caught some heat for a cryptic tweet that could mean HUGE things for the Twitter.com web interface) is also the author of Programming Scala (O’Reilly):

With this book, you’ll discover why Scala is ideal for highly scalable, component-based applications that support concurrency and distribution.

Six months in the making, the upgrades are all behind the scenes according to Evan Weaver, infrastructure team manager for Twitter:

@joelgibby The changes are all behind the scenes. It’s been like a six-month project, I think. We are very proud.

Alex shared that he had been working with Steve Jenson on the project for some time and that the:

Thing I’ve been working on with @stevej for a while is now out to 100% of users: highly scalable, slightly better quality people search!

He also shared:

Lots of room for improvement in people search quality, but our main goal of making it hella shared/replicated was accomplished with GUSTO.

Indeed.

Social information sharing – better than a press release?

Twitter has been vibing quite hard on it’s own sidechannels with no doubt triple-vetted tweets from developers hinting at upcoming features along with announcements of enhancements to twitter profiles and some new partnerships. This post was different though, because he announced that it is fully rolled out now. Before making his announcement, he retweeted @evan (Evan Weaver, infrastructure team manager @Twitter), who said “scalable people search is scalable.” We sure hope so.

Scaling to keep pace with realtime

Building scalable infrastructure is key in Twitter keeping the talking stick  in the realtime web conversation.  As social sites like Twitter grow by the thousands and millions of users uptime becomes increasingly critical as those users begin to rely on the service for everyday convenience connections like meeting friends for drinks or getting advice on local food carts, or for gathering life-saving information (via Mashable) on friends or family who may be in harm’s way. Remember the June 2009 Iranian Election? The US State Department suggested that Twitter perform routine maintenance such that Twitter would only briefly be down during the middle of the night, Iranian time. This is not just some web2.0 toy or for teens any longer. It’s a core web service, like Google Search, Reddit or Digg. Sidenote: Reddit and Digg have been tremendous community forces of good in their own right, with Reddit and Digg users together raising over $185,000 in 12 hours for Haiti relief efforts.

Dynamic development team

After this post was published, @stevej informs me that only the three mentioned in the post were responsible for the site changes, though “frontend-type folks are ofter working on that page, though.” Thanks, Steve!
I was not able to find out if the only developers responsible were just the three mentioning it on Twitter, but whether 3 or 300, Twitter really has some talented people working for them. Just check out their blogs for evidence. These are some of the brightest folks the web has to offer. As Twitter continues to open up to the web at large with announcements going out via Twitter employees rather than traditional press releases, expect to see excitement about new features and realtime developments in the unfolding Twitter story and its players.

A users-eye view of location and activity in the stream

Location is Big

But it’s only a small part of the larger social landscape. Take a step back and think about what sites like Foursquare and Facebook are asking everyone: Who are you? Where are you? What are you doing? Who are your friends? What do you like? What do you DISLIKE?  (I’m looking at you, Facebook.) Where are you going? What will you be doing? Who do you want to do it with? (You can take that to mean event invitations or something entirely different.) Do you want to see what else is around you? How about a menu? Do you need to do some shopping here today? Here are our specials. Want to pre-order something? Want to see what others thought was worth buying? (Square will be able to use location and purchase history to see hot spots.) No cash? No problem! This location accepts Square payments!  Your friend’s birthday is today and he’s two streets over at the pub with the great New York style pizza for $1.25 and $2.50 pints ’til closing (Captain Ankeney’s in SW Portland). Want to send him a discounted Brightkite / Foursquare birthday pint and a slice? (recipient may claim gift up to 5 days after purchase, must present phone to cashier for redemption)?

…and it’s heating up.

I could tell you that no one knows what’s going to happen at SXSW Interactive this year, but I’d be lying. There will be game-changers erupting from the swarthy loins of sweet, sweet, Austin TX in a little over a week. There’s bound to be a billion startups and more pitching than spring training, but these are products, services and artists that were not born, trained nor tweeted for the first time yesterday. Take for example Brightkite: a global location-based social hub ready for action, with the users to prove it. MG Siegler’s timely article over @TechCrunch has a great rundown of Brightkite. The pure excitement of everything that is coming together right now is overwhelming, and it just keeps heating up. Rather than try to segue cleanly into this next bit I’ll just let you have it:

Action streams may help us share across services.

Adrian Chan is a social interaction designer helping to seed discussions that extend all of these topics into business scenarios that are likely to change the way business is done locally. Remember my scenarios at the top of the article? (think Square) Check out his brief on Action Streams, an evolution of the Activity Streams format . These standalone services are awesome and super-dope. They represent huge steps forward in what is possible even on their own. Start gluing them together and watch the heat build. Stay with me here: Right now, the Action Streams format is a “blue sky” envisioning of the possible requirements, implications and lots more than I can go into detail on, just click through and read Adrian’s brief when you have 2 minutes. Until it’s an agreed standard, or gets adoption, we need something now.

As Twitter web goes, so go the apps.

Twitter has been constantly adding features to their web interface since the service launched. When Twitter.com implements a feature, you can bet that it will be showing up in clients soon (@replies, retweets, lists) but some features remain API-only. Back in August 2009, they first demoed the location API, and it was rapidly adopted by Twitter clients almost across the board. Until Twitter.com adopts location however, the third-party apps are shooting in the dark. A post by one of Twitter’s developers, picked up and run through the rumor mill by MG may hint at what may be coming down the pike (maybe in May?), but until there’s solid word from Twitter or a confirmed source spills, we’re stuck with lone startups and developers doing the best they can to connect us and bring us together in new and exciting ways. I’m not too worried about it. They seem to be doing a pretty good job so far.

Social Inspiration: Prime time for location, actions and beyond…

Author’s Note: I began this post at midnight after waking with what seemed like hundreds of ideas swimming in my head. This is what spilled out. It is raw. It is not “bloggy” It may still have errors even after I’ve revised it a few times. I’m not going to edit it any more though. It will stay as raw and n00b and desperate as when it was first posted. For the boiled down, web-friendly, chunked-out version, see http://bit.ly/location-activity-stream (http://joelgibby.net/2010/03/02/a-users-eye-view-of-location-and-activity-in-the-stream/) This post is more just for posterity.

Inspiration is trending right now. Events like SXSW and the Refresh series are sparking ideas and connections at a blistering pace, leaving the rest of the world shouting “Hey, wait for us! That looks cool! iPhone only? Oh nice, a mobile web app! So I can get to it on my Droid phone?”

Yes, I wrote “Droid phone” – From personal experience, folks don’t understand that Android is a Mobile OS and not a Verizon Brand. That’s another story though. Ask me about Scarborough Research sometime.

With location the thermite ready to ignite SXSW this year, people are literally Buzzing ™ with excitement over what could be the event of the realtime web decade. And yes, I used thermite as a metaphor, for good reason:

While location is keeping developers up for days on end, hacking APIs, forging conduits for activity streams, hammering out their own duct work for realtime data, it’s only a small part of the larger social landscape. Take a step back and think about what sites like Foursquare and Facebook are asking everyone: Who are you? Where are you? What are you doing? Who are your friends? What do you like? What do you DISLIKE?  (I’m looking at you, Facebook.) Where are you going? What will you be doing? Who do you want to do it with? (You can take that to mean event invitations or social and location-based personals, big in Japan, not so much stateside yet.) Do you want to see what else is around you? How about a menu? Do you need to do some shopping here today? Here are our specials. Want to pre-order something? Want to see what others thought was worth buying? (Square, you genius devils, you!) No cash? No problem! This location accepts Square payments!  Your friend’s birthday is today and he’s two streets over at the pub with the great New York style pizza for $1.25 and $2.50 pints ’til closing (Captain Ankeney’s in SW Portland). Want to send him a discounted brightkite / foursquare birthday pint and a slice? (recipient may claim gift up to 5 days after purchase, must present phone to cashier for redemption)? Like Penny Lane said: “It’s all happening.”

I could tell you that no one knows what’s going to happen at SXSW this year, but I’d be lying. There will be game-changers erupting from the swarthy loins of sweet, sweet, Austin TX in a little over a week. While there’s bound to be a billion startups and more pitching than spring training, these are  products, services and artists that were not born, trained nor tweeted for the first time yesterday. Take for example Brightkite: a global location-based social hub ready for action, with the users to prove it. MG Siegler’s timely article over  @TechCrunch has a great rundown of @brightkite that takes all the excitement of the aforementioned thermite bomb (trust me, thermite is exciting; just don’t stare at the white core or you’ll go blind, kid) and compresses it into a 762-word WMD. If this blog post (mine, not MG’s) ever gets legs, I owe it all to him. The pure excitement of everything that is coming together right now is overwhelming, and it just keeps heating up. Rather than try to segue cleanly into this next bit I’ll just let you have it:

Adrian Chan is a social interaction designer helping to seed discussions that extend all of these topics into business scenarios that are likely to change the way business is done locally. Remember my scenarios at the top of the article? (think Square) Check out his brief on Action Streams, an evolution of the Activity Stream format .  These standalone services are awesome and super-dope. They represent huge steps forward in what is possible even on their own. Start gluing them together however, and well, Daft Punk just showed up and is now playing at your house. Stay with me here: Right now, the Action Streams format is a “blue sky” envisioning of the possible requirements, implications and lots more than I can go into detail on, just click through and read it when you have 2 minutes. Until it’s an agreed standard, or gets adoption, here’s what I came up with (was awoken from bed at 12:00AM as the calendar flipped from Feb to Mar, with the full moon shining into my bedroom. Yeah, I know. Weird.) Check it out:

I’ll update my twitter status if I’m feeling command-line-ish today; maybe tomorrow I use Brightkite mobile web or the native Android Gowalla or Foursquare apps (still in need of lots of work, the both of ya)someday, with the following status:

Eating a Big Ass Burger. !@BrunchBox

It contains all the elements needed for basic social location: Who I am (my twitter username), Where I am (new “bang-at” tag (you heard it here first, folks) puts me at BrunchBox on SW 5th and Stark in downtown Portland OR), and because BrunchBox has a named twitter account and their location is set, API apps can see both named places (think Foursquare and Gowalla) and geotagged coordinates (think Flickr or Picasa). With the right amount of API glue, I can even see which of my friends (on whatever service has open APIs for friend location) are nearby using both the named location, and/or the coordinates. These open APIs will allow anyone with a bit of programming experience and inspiration to mashup and remix these services in new, possibly awesome ways. Sure, it will take a bit of math to calculate search radius and .. oh what’s that you say? There’s already APIs from Google or just about any other map provider to do that? Oh, OK, cool. I will also wager that Twitter will be making some huge location announcements very soon.

And that’s just using Twitter. There’s a sea (well more like a lake right now, but it’s a big lake, mind you) of apps, services, developers and local folks like you that are focused on more than just location. It’s going to be huge, trust me, but there’s definitely more to it than you might think.

Brightkite on CrunchBase

MG Siegler

PS: I’m broke and want to go to SXSW. If any industry (tech/journalism/K12 Education) folks want to take a bet on a dark horse that could pay off big, I need airfare, housing and an SXSW Interactive Badge.  Hollaz.

#now-at #location #bang-at #exclamation #foursquare #gowalla #activity #stream #action #brightkite #socialstream #social #news #funded #hollaz #facebook #techcrunch #MGSiegler #parislemon #SXSW #AustinTX #Austin #web3 #retweet #rtmon (retweet monday foolz) More tags coming soon when I get a break from fixing our VoIP system. Techie please!

Getting eyeglasses cheap – Locally!

-- Sharp guy with cool glasses

My glasses are pitiful. Deep gashes in the lenses, discolored anti-reflective coating (from rubbing them with my shirt – hey they can’t get any worse, right?), flaking “titanium” paint, and one of the bridge thingies that sits on your nose fell off a few months ago. I was in need of new glasses.  I had read a bit about getting cheap eyeglasses online at http://glassyeyes.com/ and was set to do it.

I went to my local optometry clinic, got my exam ($5 copay), looked at all the way overpriced frames (anywhere from $102 to $300), and asked for my prescription, asking them to include my Pupillary Distance (needed for ordering glasses online). Just because I was curious, I asked if they had any bargain-basement frames that would be anywhere close to my covered insurance level. The nice lady said they had received a whole box of new Nike frames in bulk recently and that she would price them at my covered amount (all the other Nike frames on the shelf were $160 and up)

The frames I chose were priced at or below $85  (that’s my insurance coverage, so I had no out of pocket). Lenses will be $5 copay, and I *did* get the polycarbonate material ($50, but less than the ultra-high index at $115), but I figured the anti-reflective is a waste (108$) as it rubs off after a year, looks “green” and even though it has a 2 year warranty, who seriously warranties out their lenses? So for $55 I got glasses that I used to pay 2-300 dollars for just by asking for a deal. I’m still going to get a pair online for a backup, but thought it might be good information to share.

Not staying the course (with Vista)

Here I sit, 10:00PM on a weeknight formatting my harddrive after backing up all my important data. Looking over my shoulder I see the yellow progress bar of the XP setup telling me that the quick format is almost done. Soon setup will be copying the old files I know so well back to the system32 folder and I will begin my day tomorrow with the fresh feeling that reloading XP from scratch brings. Maybe I just wasn’t ready for the future. Maybe the future is broken. Maybe XP is just better.

I’ve been running Vista on my Dell Latitude D620 laptop for the past 4 months, enjoying some of the great new features such as the integrated search (Start, type “SMS” and you’ve got the SMS admin console at your fingertips), the live thumbnails (wow, I can see what my browsers and media players have open when I ALT+TAB!), and, um, hmm… what else was there? Well, I can’t remember right now. The only thing I can remember is that routinely I’ve had to suspend my computer just to get my wireless working, had to wait for what seemed like hours to copy files across a gigabit network, watched packet captures that showed my computer (while looking idle to an end user) pouring out megabyte after megabyte of unnecessary traffic onto the wire, and have had to hold down the power button to shut down my laptop more times than I can count just to try and get some work done. There is no way that Aero is worth this. A pretty UI does NOT make up for (so far) the most unstable and counterproductive OS I’ve come across (next to Windows ME).

I’ve seen XP / Vista performance comparisons that show that even if I wait for Vista SP1, I’m still going to have a crippled system. Granted, Vista SP1 is supposed to have some stability and reliability fixes in it (and boy does it need them), but just look at the charts, man. In another post, the team doing the comparison goes over what the test entailed. I can buy it. I’ve had enough of wasted time, lost data (powering off your PC after notepad freezes when it can’t connect wirelessly to your home folder to show a save dialog REALLY sucks), and all the frustrations that have gone along with my Vista experience. I didn’t really mind the UAC prompts so much, and I liked the look of the UI, but GIVE ME MY XP BACK NOW!

I’ll post an update once I have everything back to normal to verify that the issues were XP and not hardware.

Axim X50v unofficial Windows Mobile 6

Dell may have abandoned its Axim handhelds, but hackers haven’t. A new unofficial ROM puts Windows Mobile 6 on the Axim X50v and it runs far smoother than Windows Mobile 5.0.

read more | digg story

WYSE terminal causing DHCP issues

One of our sites has had numerous intermittent DHCP issues. The symptoms were varied and unpredictable. Normal Windows clients would sometimes fail to lease an IP address successfully, preventing users from logging onto the domain. Usually, the client would eventually get an address, but during times of high utilization this could sometimes take many minutes (if it worked at all). The issue came to a head when doing our summer deployments. Our imaging process consists of booting from a CD/floppy or PXE and joining a Ghost multicast session. Not getting an IP address from DHCP was comlpetely halting work Our technicians had to manually assign each client an IP address and remember which ones were already used. Needless to say tensions and blood pressure were high.

Previously I had tried troubleshooting the problem by updating to the latest firmware on our switches, checking their configs and trying to rule out any problems on the DHCP server. These were all dead ends. I couldn’t see anything strange in the packet traces, and was runnning out of ideas. One of our technicians however noticed that if he booted a system to Windows and let it get an IP address first, the BootCD would then grab the same address and everything would work. I decided to latch onto this and dig deeper. I compared traces from “cold” booted machines, and “warm” (boot to Windows first) booted machines. At first I couldn’t find anything. but that was because I was only looking at BootP messages.

To try and cut down on the amount of traffic I was capturing, I set my capture filter to only grab UDP. In doing this, I also saw ARP requests coming from the DHCP clients. The machines that booted fine followed a process like this:

1. (client) Discover
2. (server) Offer
3. (client) Request
4. (server) ACK
5. (client) ARP for offered IP
6. (client) ARP for offered IP
7. (client) No response to ARP – claim IP

They had no trouble getting an IP because Windows had already done all the hard work of collision detection. I unfortunately did not capture traffic from a Windows client in this environment. It would have been nice to see how windows handles this. The failing (cold booted) machines would proceed like this

1. (client) Discover
2. (server) Offer
3. (client) Request
4. (server) ACK
5. (client) ARP for offered IP
6. (other client) ARP Reply
7. (client) Broadcast ARP Reply
8. Repeat 1-7
9. (client) Blank DHCP Request
10. (server) NAK
11. Repeat 9-10 until client gives up (long time)

The difference occurs at step 6. In this case, a WYSE terminal (1200LE) replied to the gratuitous ARP request from the client. In seeing another device on the network, the client then rebroadcast the ARP reply so others would see it, and then proceeds to request another IP address. The server tries to assign the same address to the client, seeing that it already has leased it to that client. The client then tries to request again and is sent a NAK each time. This process repeats until the client gives up.

So - why would a DHCP server try to hand out an address still in use? Because the lease time was up and the device did not renew during the lease time. Normal server operation is to delete a lease when it expires. Why would a client not renew it’s lease? I’m not sure. I’ve contacted WYSE to find out why the device doesn’t just renew it’s address instead of requiring a restart when the lease is up. No response yet. There’s even an option in the WYSE config files to choose whether to restart or shut down the device when the lease expires. The restarting isn’t really the issue though. When the devices are left on, they seem to go into a standby mode until woken up by mouse, keyboard or pressing the power button. When the device wakes, it presents the prompt “The dhcp lease has expired. You must restart.” Unfortunately, when in the sleep state, the devices respond to ARP but not pings. Windows DHCP servers use ping to test for collisions.

So who is at fault here? I’m not sure. I am going to read the DHCP spec and try to figure it out. Mainly because I want to know who to blame. If you have any ideas please share.

 Joel

Late Night YTMND

Apologies up front, but I can’t leave this alone:

• http://dredlawwww.ytmnd.com/
• http://itssweetyoulllikeit.ytmnd.com/
• http://berriezandcream.ytmnd.com/

Just click. You’ll thank me later. And no, none of them will get you Rick Roll’.

My Del.icio.us Tagroll