Saving wifi! Fixing Bufferbloat! Fighting the vendors and the FCC!

  ¶ 7:43 AM 0 comments links to this post

  Virgin Media - Fixing the epidemic of bufferbloat with a little more truth

  ¶ 2:29 PM 0 comments links to this post

  49... and trying to find my navel

  ¶ 6:34 PM 1 comments links to this post

  Wheels down on mars!

After all these years I can hardly stand to sit through the last stages of a landing - everything has already happened, all you can wait for is news of success... or await the silence and questions that followed for months on all the unsuccessful missions, like the mars polar lander, and Beagle2...

This time tho, the cheers rang out, in mission control, with the 600+ people in the theaters here tonight, and no doubt on web streams and television rooms throughout the globe, after the mission controller cried out:

"We are safe on mars!

... We continue to recieve telemetry from mars. We're safe."

I've had this song running through my head all night.

and...

"We've got wheels down on mars!"

  ¶ 10:44 PM 0 comments links to this post

  Tracking the landing of Curiosity, from Seattle

Shortly after arriving in Seattle, I discovered Ray Ramadorai had mailed me a link to the live mars curiosity landing event being put on at the museum of flight tonight...

My cousin Aaron and I took one look at each other, unloaded the dogs, and boogied over there.

The theater was full when we arrived. The overflow theater had overflowed, too. And the air conditioning had failed. The audience sat rapt as the engineers and scientists explained each subsystem, each design, each instrument...

As I write, the main entrance to the theater is now full, and they've just added a second monitor for the rest of the crowd that keeps flowing in, and in, and in.

I remember in the time from one presentation to the next, Curiosity moved 8,000 km closer to Mars, 28k away, about 20 minutes ago, as best as I can remember... maybe it was during the 7 minutes of terror?

The time continues to fly by. Landing in less than 70 minutes! There's people here with their babies, guys with their girlfriends, girls with their guyfriends, NASA employees, members of alt.space, Aerojet, Planetary Resources, MPL, name it, and they all sit with eyes aglow in anticipation and hope...

##curiosity is an irc chat channel on freenode dedicated to the event.

And they are about to switch over to the NASA tv feed, on the big screen, in high def.

Chris Lewicki is up now, describing the comms systems.

  ¶ 9:14 PM 1 comments links to this post

  spotting NEOs from around venus's orbit

I gotta say that seeing all the excitement around NEO exploration *finally* is rather wonderful. The B612 foundation is planning on Charting the NEOs from near Venus's orbit, which is a good place for energy, and maybe a few interesting rocks on reachable orbits...  ¶ 11:02 PM 0 comments links to this post

  Asteroids as lunar orbit resources

For nearly 9 years, I've had one of my early attempts at a coherent argument for leveraging near earth asteroids in the space program sitting on the right hand side of this blog. I've been meaning to revise that piece for years too (nearby rocks are moving fast and are useful to hit, but not land on), but haven't got around to it.

A few weeks back I learned of a NASA study that costed out how to do a major sample and return mission to lunar orbit. It's cheaper than a moon landing. By a lot.

And today I learned about Planetary Resources, which with some billionaire backers is aiming to take on asteroid mining on some scale, according to the WSJ.


Wow.

For the last 4 years I've been watching the successes of Spacex, Armadillo Aerospace, Bigalow, Xcor, Scaled Composites and others, thinking that they needed payloads, and places worth going to.

Of late, I've been too busy helping fix the internet to pay much attention. I was in Florida for the (scrubbed) 3rd launch of the Falcon 9, and was hoping to be there for the upcoming one, but that's about it.

I'd still, despite admiring those successes, basically given up.

Years ago I'd suggested that Google throw 30m into an Xprize for asteroid exploration. I didn't expect something as big as what Planetary Resources might turn out be. I look forward to more detail Tuesday.

...

Back in the early 90s I tried writing a SF book about the economics and human interactions of what realistic near earth asteroid living would be like, the spectre and the glory of travelling between Jupiter and Venus every four years. I'd hosted one civilisation on the asteroid "Toutatis", in particular. I had fun building the place out with lap pool around the waist, installing plumbing (if you know anything about Toutatis's rotation, this is actually an interesting problem) and observatory, and the stuff that would keep settlers alive and happy for long periods of time, and how cultures would exchange people (much like the American Indians) and knowledge (Usenet!)

I projected 20-30 years of robotic exploration and then a severely modified humanity, to make it out - legs amputated at birth like the dangerous appendages they are - etc. I never finished writing the book, though. Vinge beat me to it with the Usenet hook, dang it.

But: along the way, I proved (to myself at least) that using the NEOs as waystones was the lowest cost/highest payoff starting point for expanding out into the solar system, which has led to countless debates with Mars Society members. I'd steer people at Lewis's "Mining the sky", at Buzz Aldrin's Cycler concept, and at the mass requirements for O'Neil colonies.

I've kept sort of a low level interest in the subject ever since. I always use examples of spacecraft failures in my talks about reliable embedded software design. It's always a good hook with grad students. I keep up on launches, subscribe to several enthusiast lists, read what little literature emerges and sometimes even make it to a conference.

I was a little non-plussed by the recent NASA study for landing people on an asteroid. First up, in my mind, was always robotic exploration and extraction. Still: the costs on that looked pretty good, and I knew mars was more expensive...

If bugged about space stuff, I try to explain that the solar system looks nothing like what Asimov depicted in "Marooned off Vesta". Exploring the rest of it is far more akin to the age of sailing ships that took years to get anywhere, rather than 1g spacecraft that bop from place to place in days, as all the early SF writers mislead us. The economics of sail apply. Except that space is *bent*, in weird ways. The archipelagos move, and the shortest time between two points is never a straight line, and there are keyholes and other interesting gravitational phenomena, and you don't need rockets to get around an asteroid (springs and tethers suffice), etc, etc.

Another big problem has been that most people's impressions of the solar system were formed by the Apollo program and those aforementioned writers - in stories that predate the discovery of thousands of Near Earth Objects in interesting orbits.

Over the years I tried dozens of ways to get the points across, trying to educate people about resonance orbits with Jupiter, proposing a grand tour of the NEOs, and talking about replicating Deep Impact on a grand scale, and various other serious bits like that.

Sometimes I seriously tried to be unserious, in trying to find bridges from everyday experience, to space. I went looking just now for the serious stuff I'd written about Deep Impact and instead stumbled across me channeling Howard Cosell during Deep Impact here, here, and here. Ah, I'd wrapped it around "The Thrilla in the Chilla", indeed. In retrospect I'm pretty sure that I missed both core audiences with those pieces...

I threw a party every four years on Feb 29th, called Asteroid Appreciation Day. Asteroid appreciation day, this year, I moved to 12/12/12 or so (still haven't planned it out)...

I wrote songs about the Challenger and the KT event. The KT event one, "Living in the 00ze", well, I have a much better version of it than this, but haven't bothered to lay it down.

As for the other one... I've played "Rhysling and me - Falling Free and Flying High" more times and in more venues than I can remember - most recently twice a week for over 3 years, live, in front of unknowing tourists, while I was in Nica. I've recorded it with flautista, and in the studio, and a couple other ways, and even wrote a script for the video.

That song got me invited to the Heinlein centennial! Spider Robinson didn't sit in, but what evan and I did went over well - recorded at the Heinlein Centennial. (Evan's song Home went over well too, and I just discovered I still have this enormous archive of all the filksongs played that friday night that I should edit down. Justin Kare, Margaret Middleton...). I also got a chance to meet a bunch of people that I'd only corresponded with, and a few I'd admired from afar.

Not a lot of people know the story of the first sax in space that I reference in the song.

I haven't had much urge to play that song since the last shuttle landed. I'd like to write something happier.

I even tried writing a children's book once, in verse, figuring that having lost this generation, I'd try the next. "Son, your mommy and daddy are both unemployed, and you want to live on an asteroid?" (I still think it was a wonderful story - the kid flits off to dactl and has to be pulled back via tether by his dad - I got a chance to try and explain tethers with a Tom Lehrer/G&S reference - try finding rhymes for centrifugal! )

Every couple of years, I publish links to Lance Benner's work on NEO asteroid rendezvous and intercepts pages. It's my way of keeping up with the rate of discovery.

I was critical of the Dawn Mission on two different occasions because they were focused on big bodies, not small, and of New Horizons, for the length of the mission and the data rate.

While I look on with admiration at Dawn's early mission photos... not having spacecraft, in the space program, with a reasonable cycle time from design to construction to launch to data, from redesign to *mass construction* to launch to more and more data, to enough raw materials for a sustainable ecosystem, frustrates me.

I worry a bit about Apohele NEOs, they are both nearby, frequently, and hard to see. Since discovery, 10 have been confirmed. Apoheles also strike me as an opportunity. I don't know if anyone's calculated intercept or rendezvous orbits for them yet.

My main machine is named cruithne. Backup laptop, ida. My Kindle? dactl.

So like, I dig asteroids. And rockets. And space.

Please note if the above seems excessive - I've just written up what little I'd managed to accomplish in 30+ years of trying, not hard enough, in my spare time. I have no idea if anything I've ever said, did, wrote or sung about space ever succeeded in convincing anybody about anything.

But I'm really happy with the direction the space program is finally going.

All I ever needed was a rocket ship, and the stars to steer her by.

Labels: alt.space, asteroids, space

  ¶ 1:23 AM 0 comments links to this post

  SOPA is bad news

I haven't been paying a whole lot of attention to American politics of late. At least one variant of SOPA, as proposed, will mess with DNSSEC.

I believe - wholeheartedly - that DNSSEC has the ability to improve the security of the Internet for everyone. It's a critical component of my cerowrt project, comcast just rolled it out nationwide, many other providers are also doing so, and that has taken tens of thousands of people, over 10 years of effort to accomplish.

I have also been working for years to make DNSSEC just work, only to look up now, to see clueless politicians in the pay of a a few lobbyists playing with technologies they don't understand.

I've been busy this year, on another front - trying - and succeeding!! - making the Internet, actually better, for everyone.

It really bugs me to see all the time, energy, and money, that can go into screwing up the internet, especially vs all the time, energy, and money that goes into making it better.

I don't have a comprehensive grip on what sopa does, but here's one reference that seems good. Doc Searls also ranted well...

Seeing wikipedia go dark today, was a terrible, terrible thing. I'd like to keep the light of knowledge, burning bright, throughout the world, for everyone, all the time.

Labels: sopa

  ¶ 9:44 AM 0 comments links to this post

  Departing France for England, then 'home'.

So, I looked up, and realized that it was time to leave Paris.

Tomorrow.

I'm sad to be packing up and leaving, and mad at myself on a few particulars, but on the whole, it's been a wonderful experience to have spent time in one of the still beating hearts of civilization.

As a tourist I got to see the Eiffel tower, and tour the entire Louvre - (it wasn't the Mona Lisa that grabbed me, it was the painting on the opposite side of the room that really blew my mind - and the statuary - oh, my!)

I ate dinner at nearly every restaurant between my apartment and my graciously donated office at LINCs lab, and a few other places. I got turned onto some wonderful places with friends that I will never find again. The wine, was wonderful... I had a couple dates, too, but romance escaped me - and I was working my brain dry... I spent 99.9 % of my time obsessively trying to beat the bloat.

I met some people I'd only corresponded with for years, helped a couple students, gave a couple talks, did a ton of research, and my life revolved around work, primarily. Darn it. Again.

There was so much more I could have done - in particular I'm sad I missed seeing both the catacombs and Monet's gardens, but I hope to be back one day, and those will be tops on my list. I wish also - although I fell in love with Duc Lombards and the two other big jazz clubs in Halles - that I'd actually got to *all* the jazz clubs, and seen some plays and concerts, but it was very hard to navigate alone...

I didn't manage to learn that much French. I will have to study the language far more fully before I choose to return. I did learn lua, though.

I'm off to London now for a week (I'm giving a talk at UKnof21), and I hope to spend time there, seeing things and people that I've always wanted to see, in that country. Then... Back to Florida to see my folks, and after that...

I have NO idea. Usually I write up a summary of the last year, and make some new years resolutions, and review the old, and I guess I'll start doing some of that, while trying to ingest as much of Europe as I can, in my time remaining here.

I wish I'd got to Amsterdam, Spain, and Italy on this trip, too.

Ahh, well. The work was good.

Labels: paris

  ¶ 8:53 AM 0 comments links to this post

  An incredible stream of co-incidences passing me by

For the past couple days, while waiting for code to compile, I've been playing an old song of mine, "Living in the ooze", revising the lyrics, coming up with a bridge (finally! 6 years without a bridge!) that I think might work. This morning I even went out to get batteries for my recorder. I didn't get them because I don't know how to pronounce 'batteries' in French, and staggered back, defeated, to my apartment in Paris.

I've told the story behind the song in multiple posts over the past 6 years, and we're coming up on asteroid appreciation day feb 29th, next year, and I don't have a plan or place for the party... and in part those stories are always about impossible co-incidences and weird stuff happening at all the same time...

I have NOT been paying attention to space stuff AT ALL for the last 9 months, being deeply immersed in bufferbloat. I even stopped reading the Arocket mailing list. I spent the weekend completely immersed in a set of algorithms that finally, finally might reduce latency over wireless to sane values once again, only to end up with one bug per line of code and multiple kernels that wouldn't boot. Never left the house all weekend. Still haven't made it to the louvre. Or anywhere else in paris but the LINCs lab. I'm insanely frustrated with this project.This morning at 12:40 AM I was ready to chuck it all and get on the next plane to anywhere that didn't have computers... but was stuck on thinking where on earth that might be, so I did a few other things about my stuck-ed-ness..

I haven't been reading the news, either, as news.google.com insists on coming up in French no matter what I do. I'm still stuck on the six different ways to pronounce 'e', in my French-english dictionary.

So, later this morning, in an impossible co-incidence, I meet someone who just spent the summer in Nicaragua, who points me at her huffpost article, and I send a link back to one of my adventures in australia which happened to be the last thing I'd written about this particular song... and happens to be about some weird coincidences in spite of my agnostic-ism...

And then... I notice... at the top of HER post...

Quarter mile wide asteroid to miss earth Tuesday.



Nerts. It's going to miss Texas. It almost hits the moon though! That would have been spectacular! And the article has quotes from Lance Benner who I've exchanged multiple emails with over the years, and I still point actively at the data he collects on plausible rendezvous trajectories...

Perhaps the universe is telling me something. About what, I don't know. Perhaps some sort of sacrifice to it is needed. Perhaps it - or something in it - needs attention.

Either I get back to recording this song or go back to work, or go for a rendezvous.

OK, I've googled for how to pronounce batteries. And I wonder what can rhyme with UV55?
PS: Boycott the Hôtel Ibis Sainte Catherine, Bruxelles  ¶ 5:46 AM 0 comments links to this post

  12 suggestions for startups

I'd written this months ago and meant to post it then. Having been through 14 startups now (of which 3 I started, 1 (none that I started) that worked out ok), I figure I have sufficient background to give advice... especially to myself. I tend to forget one or another of these points much to my own detriment, in the heat of creating a startup.

I keep forgetting to apply points 10 & 11 below to my own work. At present, I'm merely applying startup-like methods, some of which are working, some not, to a major R&D project. The number of points missing on the strategy are sometimes glaring - I merely want to fix a problem (bufferbloat) so well that it never bothers anybody again. As for profit motive, I have none - merely cutting my own annoyance level regarding the behavior of wireless networks back to "mellow", almost suffices.

12 suggestions for startups

1) Retain control. Most successful startups go through three rounds of
funding.

If done wrong, by the last round the founder(s) generally reduced to a tiny minority shareholding and have lost control of the company... and are usually forced out by the 4th round or IPO.

If done right, the founders retain 80% or more of the stock in the 3rd round.

I note that 9/10th of startups are not successful, and that a large percentage of the ones that 'make it' did so without benefiting the principals or employees to any real extent, only the VCs.

2) Paul graham is good. I'm a big fan. The ycombinator model appears to be working, but as to how much it meets goal 1, above, I currently have no insight.

3) Have a pitch, and a plan, pitch it often, *listen* to the responses and objections, and revise after every meeting. Don't be deterred by failure. Make sure your goals are shared with the people working with you, as well.

4) Selling something is great.

Selling something you can actually make and sell at a profit is vastly to be preferred.

5) Get to plausible promise before seeking any money at all. If you can't do a startup's initial development on what you already have, you can't afford to take risks of this size, and should stay gainfully employed at something else.

Once you start chasing money, chase it hard, chase it continously, get it in the bank, and spend it appropriately. While there is such a thing as 'too much money' while in growth mode, it's a nice problem to have.

6) Decide on your exit strategy early. "go public, "stay private", "make money for 5 years", "lose money for 5 years".

Cynically I note that you don't have to share your exit strategy with your employees or your investors.

It does help in planning and in motivating your people if you know what your exit strategy is - there are very different motivators for 'GOING IPO IN 2 YEARS' vs 'Make money, reliably, starting in 2 years', or 'provide a valuable service', or 'solve a global problem', and it changes the kind of people you get.

7) Incorporate early. Twice. The first company loses money, the second is in reserve. Both are handy to have around. Shell companies in particular establish credit that exists independent of your own income, over time.

It is best to have had a company around for 3+ years before starting to really use it.

If it were legal, given todays legal and tax environment, I'd incorporate kids at birth. As it is, at the moment I'd recommend incorporating 'em as soon as legal in Delaware rather than overseas. It really complicates you life unnecessarily. (that said, doing business with other businesses outside of the US is mildly easier, and shipping your kids overseas will give them a bigger picture than they can get in the US of world needs)

8) Get a good lawyer. Also, get a good accountant. Take the advice of both particularly as regards to points 5 and 6, not me! Two lawyers and two accountants are an even better idea. Bad lawyers and bad accountants have sunk more than a few startups.

Have a clear goal, corporate rules, etc, laid out, in a mission statement, etc.

I have no opinions regarding sub-s or LLCs. A few years back LLCs were all the rage. Talk to two lawyers and one accountant about it.

Regardless, you're going to lose money for a while. It's good to make that tax-deductable any way you can. Cash-flow will always be a problem, whether it's early days of no income or while facing unexpected growth.

9) Contract to hire. Never hire until you have to. The costs both of hiring someone and of firing them far outweigh the extra costs of paying contractors of various worthiness. Getting business insurance is a problem, getting health insurance is also a problem.

Limit your fixed commitments rigorously.

Contract yourself if you want. I use MBO partners for this - they get me a pair of 401ks, 1m in business insurance and take care of a ton of details that I don't want to, in exchange for 5% off the top of my billings (in the US) - where most agencies take 40%.

10) Recognize your own faults. If you are a detail person, get someone that can stay focused on the big picture. If you are a big picture person, get people that do details. If you can do both, do both, but on separate days. REMEMBER to do both, regularly.

(Being a detail person myself, I have to reset with long weekends periodically to review the mis-fired plans and replan. I've had to do a lot more of that this year, than I'd like)

In all cases, having a good and well enabled AA/secretary working for the CEO primarily is tremendously useful. Get stuff WRITTEN DOWN. (I use transcribr.com whenever possible)

Also having a good shared scheduling system is helpful. I could go on for pages here...

11) Set goals, and plan, rigorously, and both conservatively and optimistically. Revise your plan monthly. Software developers are notorious for over-estimating what can be done in a month, and underestimating what can be done in 2 years. Marketing guys are notorious for missing trends until they've already happened, and selling visions of things that can't be built by any software developer on the planet and promising them 'tomorrow'.

12) If you are risk-adverse, don't do a startup. By all means, DO! form a company to fund your own interests, and lose money with it profligately, it makes the IRS mad, and that's worth it in itself.

The seed capital to accomplish the starting procedures above is about $15k. Yearly, maintaining the corporations depends on your locality, but runs less than $1k each, accountant < $1k, lawyer less than $3k, and if you can't find a way to lose 3x that much money pre-tax on a regular basis to make up for it, see multiple points above.

Labels: startups

  ¶ 5:17 AM 0 comments links to this post

  Steve Jobs, RIP.

I am going to miss Steve Jobs. I'd admired/hated/loved/wanted to be like/unlike him for much of my adult life, and now that he's gone, I can't think of anyone else in the industry that could have stirred up such emotions and thoughts in me. He spoke well, here at Stanford. RIP, Steve. I intend to stay both hungry and foolish, for as long as I can, thanks to your inspiration.  ¶ 3:39 PM 2 comments links to this post

  Welcoming my father to the blogosphere

Ron Taht, my dad, is finally, blogging! My mom and I have been encouraging him for years to stop wasting his time with letters to the editor, which would - whenever published - always get published, truncated, with salient points removed.

A few months back I showed him how to use blogspot and finally - after a few false starts, he started generating some good stuff, at lengths more suitable to what he has to say.

I'd delighted to see him finally getting his full say - and while my dad and I don't see eye to eye on many things, he instilled in me a great love of debate, that I didn't exactly appreciate when I was younger, while I seeking moral guidance rather than debate. Back then, he'd always pick the opposite side of whatever I was thinking about, no matter what he actually thought, just to sharpen my wits (and leave me confused about, well, just about everything)

Now, after finishing his career as a prosecutor, lawyer, and judge, he speaks with conviction, about what he really thinks, and *I'm* the one that automatically picks the opposing side when discussing anything with him!

I'm very glad he's finally writing his stuff down, and speaking his true thoughts from his heart and mind.

He has tons of entertaining stories that I'd like him to blog about, too - fishing tales, golf stories, multiple episodes in court both tragic and funny, hysterical funny scenes from housing sales, and scary stuff from his prosecutor days...

But, as he's not much of a typist (as yet), he's primary writing about the issues that concern him now, deeply, about the current problems and future of America... and I find myself agreeing with him far more often than I'd like (or am willing to admit, while debating with him)

I hope for a big welcome from the blogosphere it's newest 75 year old member at ronsravings.blogspot.com, my dad, Ron Täht. Comments and criticism of what he's writing about will be deeply appreciated.  ¶ 10:22 AM 0 comments links to this post

  Any other musicians at linuxcon in Vancouver?

I was wondering if the old band could get together for the 20th anniversary... I've got the management behind borrowing the hyatt bar and the piano....

Labels: linuxcon vancouver

  ¶ 9:31 AM 0 comments links to this post

  Me, 46, Cerowrt - RC5

Usually I write a long, contemplative blog post on my b-day, but I simply haven't had time to gaze into my navel all that much.

I AM however, hoping that the RC5 candidate for CeroWrt proves out to be a good one, and begins to address the mistakes I made a decade ago, and mistakes everyone has been making since the 802.11n deployment.

I spent the last week with the incredibly helpful folk at ISC getting a lab put together to test this release of CeroWrt, and it's looking really good... but I did make the go/no go decision on a RC at 2:38 AM this morning, which worries me - but hopefully there will be few problems.

I'm hoping actually, for a dramatic difference in wireless network behavior for those giving the RC5 a test - certainly in testing I saw some of the cleanest TCP/IP streams I've seen all year.

I also look forward to people exploring all the new ideas inside of Cerowrt - DNSSEC, mesh networking, a local web server, etc, etc.

There are still a ton of bugs left to fix, but no priority ones, and that... is good enough to take a day off on, and enjoy wandering a park or two in california, and play some guitar. I'm off to Vancouver for a pair of conferences next - and I hope to stage a reunion from the band we played in, in Nicaragua - with the flautista, Angel, as well. Got a few new songs in my stack now, notably 'Please come to boston'.

So while blogging has been light, if I'm lucky, things will slow down enough for me to talk about what we've been up to for the last 8 months, and where we're going. Where I'm going next, after Canada... is Paris!

The greatest gift I've ever got for a birthday! - was the help of hundreds of people, all helping to fix, and finish the internet.

I'm in awe and delight. Thank you all, above!

Despite all that sentiment... First up this morning, on my list, is laundry.

Labels: cerowrt

  ¶ 10:33 AM 2 comments links to this post

  Battling the bloat

So, I've been so busy for the past few months as to have let multiple things slide. I've been all over the US - florida, Georgia, Pennsylvania, NJ, Boston, California, Georgia, and now, I am back in NJ, living out of a suitcase with increasingly irrelevant clothing for the weather.

I hope to jaunt to Europe next, after my billings catch up with me. Then back to Nicaragua.

On the minus side:

My main email went down for a month. I hardly noticed.
I completely forgot an important anniversary... for over a week. The lady involved is not speaking to me.

My laptop crashed 5 days after I'd used the USB stick I use for backups for something else.
I haven't checked my voicemail in a month, at least.

I got no exercise.

On the plus side:

Eric Raymond now has IPv6. So does Evan Hunt. Two down, several billion to go.

The latest and greatest bind9 - with dnssec support - is now available for openwrt in the Cerowrt git repository, for testing, as part of ISCWRT. DNSSEC is a mere 3 configuration commands away!

I gave a fairly well received talk at asilomar about the problems with the Net outside the USA.

I attended jim gettys' recent google tech talk, and breathed the same air as Vint Cerf.

The debloat-testing kernel now has most of the features we were trying to test (SFB,etc) , and it has been updated to 2.6.39.

There is now an extensively debloated (but not perfectly by a long shot) version of
openwrt - which also contains the critical stuff from debloat-testing, so we can test
end to end connectivity in all sorts of ways.

That release is entering the final stages of testing, and is codenamed, Capetown. It works on the netgear wndr3700v2, which is a wonderful piece of hardware.

And some routers using that are now up and running, capetown, South Africa, as part of the Bismark project, which I'll be helping out at through mid-august.

The last 4 weeks of my life I worked at a level I have not worked at since my late 20s.

I'm beat. And I'm taking the weekend off. Hopefully someone else will enjoy this stuff and put it to good use in their research into battling the bufferbloat problem.

Labels: bufferbloat

  ¶ 7:50 AM 0 comments links to this post

  Operator overload & nuclear troubles at Fukushima 1

I'd actually written my last blog post on monday, thinking it was tuesday, and wednesday in Japan. Shows how much sleep I've been getting.

The quality of media coverage has improved significantly, but the scope of the cascading failures at the Fukushima nuclear plants has grown - with secondary damage from the explosions and over 500 aftershocks in Japan complicating matters.

The original links I pointed to in my first post have been updated and revised, with MIT's department of nuclear energy doing a better job of filtering through the events involved than any other media organization.

MIT has not addressed the questions raised about the Mark I containment facility that concern me the greatest. I daren't speculate.

Several other aspects of the news coverage and analysis bother me:

A) lack of understanding of the effects of all those aftershocks, and for that matter, coverage seems to be limited to talking about the first quake, even on wikipedia. It's obvious that these had effects, in part, making post-quake inspection difficult.

B) lack of understanding that these were some of the oldest and most obsolete plants in the world, not just Japan. I keep seeing calls for increased safety, or damning nuclear plants for their lack thereof, when these were generation II plants, kept running long beyond their initial design life, due, in part, to the difficulty in getting new plants built.

Generation III+ plants such as the AP1000 have, for example, entirely passive cooling systems, and have safety ratings 1000+ times better than the Gen II plants did. Furthermore they use their fuel more efficently with less waste. Nuclear energy is much more well understood now, nearly 50 years after these plants were designed.

C) The on-going storage of the fuel rods - due to being unable to find another place to store them - is likely the largest danger now, as it appears as though at least one storage pool was damaged in one of the quakes and explosions.

Everybody - on all sides of the nuclear debate - agrees that continued storage of the fuel rods at the facilities was dangerous - and most facilities were not designed with long term storage in mind.

Now, that after the fact, the "out of sight, out of mind" nature of the ongoing storage of fuel rods in ad-hoc facilities in the presence of such debate has been exposed - perhaps some rational decisions about what to do with the spent fuel will emerge.

I doubt the US will become rational on this point anytime soon. I suspect Japan will become so. I also think we'll see a surge of interest in solar power worldwide.

D) with less than 50 operators on duty, that translates out to 12 or less operators managing the 4 reactors in trouble - I imagine that some of the additional failures since monday were in part caused by exaustion and overload, and the inattention to other pre-emergencies due to these factors. It's unclear how many people are monitoring plants 5 & 6.

One of the big causes of both prior major nuclear accidents was operator overload. Too many things beeping, and buzzing and alarms going off, and too much complexity in the control systems.

Future plants - if they are ever built - should have a good offsite management and monitoring facility inconceivable to those in the pre-computer design era.

For all that, I do wonder that the world-wide reaction is overblown. I can't imagine, were I in charge - not sending in every available qualified volunteer and resource available.

For all the worry reported in the press, this (via wikipedia as of about 10AM Wed MDT), remains true:

“To date, the radiation leaks beyond the plant's boundaries have not reached a level high enough to constitute any significant detriment to public health. However, there is still significant risk that a leak at levels high enough to affect public health may occur.”

E) There have been more than a few hair brained schemes floated to cool the reactor pools - for example, cooling the overheating fuel pools by dumping water via helicopter. Dumping water by helicopter cannot be done gently, and would release a great deal of radiation to the crews dropping it.

There is a huge amount of mis-information on the web regarding the deaths of the pilots that flew over Chernobyl, I've been unable to determine the truth of matters.

I find myself tearing up at the dedication of those working to stop an even worse nightmare not just at Fukushima, but throughout the country and the world. I wish I could help.

I also remain in awe and admiration at an high-tech engineering culture and country that could go through a disaster this size, and have under 20,000 dead.

Labels: Fukushima, japan quake, nuclear energy

  ¶ 9:39 AM 0 comments links to this post

  bad wednesday for nukes in japan...

after my last blog entry, lots of bad, scary information came to light, 2 more reactors ended up with more problems than the first two.

Notably, the presence of spent fuel rods onsite and old flaws in the mark 1 containment system led to my greater concern, particularly after more hydrogen explosions damaged the surrounding area.

According to wikipedia, after a fire at reactor 4, hourly radiation reached 100 000 μSv. That's a big number. A scary, bad, number. But not a (rapidly) deadly number. Reactor unit 3 reached 400,000 μSv. Why people are reporting micro (10^-6)rather than milla (10^-3) bothers me,

Years ago, I wrote about the dangers of running nuclear plants past their design life. Now, with accident cascading into accident, the operators are tiring and making mistakes, and all seems grim in Japan to constrain meltdowns in several plants.

My heart goes out to those attempting repairs. Things may turn for the worse as it gets tougher to spend time at the site, safely.

Update: Wednesday 8AM MDT

I'd written this blog entry on monday, actually, thinking it was tuesday in the US and wednesday in Japan. Shows how much sleep I've been getting.

Labels: japan quake, nuclear energy

  ¶ 11:19 PM 0 comments links to this post

  Heroic civil engineering and disaster planning in Japan

Shortly after I'd written a piece on my other blog about the amazing successes of civil engineering in Japan to withstand the 8.9 earthquake vs previous disasters, I got wind of another piece that was written by someone IN japan that goes into more and better detail overall, and a third, that talks clearly to the nuclear issues.

Universally - wikipedia and the bloggers have beat the conventional press hands down for accuracy on this nightmare. It's too bad that everything that hit print thus far is so off.

Labels: earthquake

  ¶ 11:52 AM 0 comments links to this post

  Beating Our bloat

At my brother's wedding last week (way to go Steve!), one of his friends noted I had stopped blogging... instead, I've been making serious progress on this New Years resolution:

Collaborate more. Make a stronger effort to find people worth collaborating with. Use email more. Use usenet again. Push into the mainstream more patches - but logout at the end of the day - create some music.

Back in mid-November I'd first caught wind of bufferbloat from Jim Gettys' blog.

At first I was merely intrigued...

I had seen the kinds of TCP traces jg was getting while I was in Nicaragua (working on the wisp6 greenfield wireless mesh network), and in several cybercafes and hotels. I'd assumed then it was merely the tin cans and string connecting Nica to the rest of the world. I'd seen traces like this, in particular, a lot:



I didn't understand the effects on TCP of bufferbloat until I saw his traces and then his more detailed analysis with different tools.

This is a normal TCP trace:



This is a bufferbloated TCP trace:



It looks like an EKG on crack! (That's what Steve Lord called it, anyway). I envision this picture on a milk carton, with the caption:

"Have you seen this trace?

Login to bufferbloat.net to learn how to fix it..."

I got interested.

So, I re-ran his experiments, against the wisp6 router testbed. The results, under bad conditions (heavy rain), were horrifying.... 10s of seconds of delay in the routers (!@#@!#!)... and explained why NTP, DNS, ND, DHCP, and most other traffic had stopped working under those conditions.

Still, even at this point (late December), I thought it was a local, device specific, problem. I did a little patch to those routers and fixed it, (but good!) and went on my merry way, trying to cope with my other wisp6 problems of autoconfiguration, ipv4 in 6 encapsulation, ipsec, mtu size...

Then I saw the netanylzr data and watched and listened to jim's presentation about 3 times...



The diagonal lines are showing latencies - across paths that should be taking under 100ms to do anything - all over the world - measured in SECONDS.

I realized, finally, it wasn't just me and my devices and my little network in Nicaragua.

Bufferbloat was a global internet-wide problem, one probably growing worse, rapidly.

I got alarmed. If NTP, DNS, DHCP, ND, etc., start breaking we're in a world of hurt, but if TCP/IP starts breaking worse really bad things will happen...

I emailed Jim Gettys on January 10th about the mis-understandings thus far in the press that I'd been trying to correct, and volunteered to donate a pair of servers that I had lying around, and maybe write an article about traffic shaping... he told me I was exactly correct in my own analysis...

I'd met him a couple of times, we'd worked on the same stuff, like handhelds.org, X11, and OLPC...

...and so I found myself instead hacking ruby and redmine, getting multiple servers running, using my rock and roll promotion skills to get people all over the world in disparate disciplines involved, hacking kernels, fiddling with AQMs and new algorithms, reading 70+ theoretical papers, writing multiple pieces and wiki pages, making deals, swapping services, picking up dropped balls, making a ton of phone calls and exhausting my personal email address book to get bufferbloat.net to be a real, functioning entity, with developers, theorists and users from all over the world, and not a talk shop.

And the rest, is history in the making.

I still haven't got around to writing the piece about traffic shaping.

Basically, Bufferbloat (see FAQ) is a new name for an old problem (RFC 970) that has gradually been re-introduced over the last 10 years. It's especially bad in cable modems, 802.11n gear, FIOS, but also can be seen in just about anything that has a wide dynamic range (GigE switches hat do 100Mbit). It's bad, it's ugly, it's screwing up the Net, big time, and it's just a mistake that we've (as engineers and network designers) have all been making for a long time...

Head. Desk. Head. Desk. Head. Desk.

The Bufferbloat problem is almost as bad as Y2k... And more solvable. It's just that the Internet is so much bigger now than in 1999 that is intimidating. More cell phones are being added to the Internet every quarter than we had total users in 1999. There's also a persistent fear that it will get much worse, before it gets better.

So we've been lining up people to fix it ever since.

While doing all that, along the way, I came up with a good idea for a cosmic background bufferbloat detector that was extensively discussed on usenet, and the bufferbloat mailing list. Nobody found any holes in the concept which means (darn it) I'm going to have to code it up - or convince someone else to do so.

Good stuff keeps happening... there are nearly 200 members of the bloat mailing list now, John Linville just released a debloat-testing kernel containing not only a new algorithm (eBDP) for wireless, but two new AQMs and some driver patches. Doc Searls graciously loaned me his column for an editorial in Linux Journal's upcoming June issue... Vint Cerf loaned Jim Gettys his column for IEEE computer (due out in a few days), multiple other writers have chipped in... Theorists, coders, cats and dogs, all talking to one another on the mailing lists...

About the only flaw in all this activity of mine is that I've been so buried by it all as to stop blogging!! The effort required to write something for a more general audience is so much greater than carrying out conversations with the people I'm collaborating with presently on email and irc that I've stopped journaling entirely. I'm trying to fix that today, a little.

I've learnt that while journaling/blogging is important, even necessary, to the writer and his/her creative process, writing the history down behind the writing matters to no-one else. (I'm journaling today so that I can remember the timelines here)

Also, cutting the history from the finished work helps a lot. I just learned this trick from esr, who has also taken time out on irc to teach me more about writing in the last 2 months than I've learned in 10 years of blogging. (It also took 5 other polished writers - Evan Hunt, Bill Weinberg, Richard Pitt, & Jim Gettys, to tell me in no uncertain terms that I was doing some things wrong - for it to register. I've undergone a writerly "intervention". It was painful, but I'll survive)

I wish now, that I'd opened up my writing to a writers cabal 25 years ago, or earlier. I might have got a few books done by now.

Tomorrow (wednesday) I'm in open-to-all VOIP conference call about bufferbloat, with the freeswitch folk. Please join the call to hear more. Or check out bufferbloat.net.

After I gave up on SIP based VOIP (after working on it for 6 years), and gave my last presentation on it, in 2006, at Astricon, I'd had no idea then that a goodly portion of the problems I'd had with SIP were tied to bufferbloat. No idea what-so-ever.

Solutions seem feasible, across the Internet, for a whole new level of interactive applications after we get bufferbloat fixed. SIP phones now do IPv6, which solves a lot of problems, too. I'm seriously encouraged.

Sometimes it takes giving up on something, utterly, in order to make progress. It's been a zen 2011 that way. And also resolving to actually resolve your new years resolutions - works too.

All this said, I'm going to take a break from all this soon and write a bit about listening to, and making great music, and about an old, cherished concept of mine (and jeff stram's) called the jam-o-phone.

Labels: bufferbloat

  ¶ 7:07 AM 0 comments links to this post