Twitter, we’re looking at you

On a number of sites I (along with ‘jrgp’ and ‘furai’ and others) are responsible for operating, there is a pattern that has been present for quite some time: one particular large Internet content provider has been responsible for the only IPv4-only elements in a majority of our pages…

twitter-v6

 

See it?  Twitter.  This is true on much of the content on https://forum.kag2d.com and I have seen it on other pages of ours and other folks’  — where the page, google/yahoo/youtube/facebook objects and others are all v6-enabled — but Twitter stands the lone cheese without IPv6 support.  Heck, even LinkedIn has IPv6 support on their main content and CDN infrastructure now!

New Home For Albany Community Television

Unfortunately the albanycommunitytelevision.com domain expired, but the Albany Community Television content is still around! Albany Community Television can be found at https://act-archive.u13.net , which has all of the historical posts.

New content is not being posted, now that Albany has “real” public access television, which was the underlying goal of the Albany Community Television project.

For anyone looking for ACT archive videos for download, feel free to contact me at “ryan at u13.net”, I would be more than happy to provide all of the video content for download. Alternatively, all of the ACT content is available at https://act-archive.u13.net and https://act-archive.u13.net/v.php for download/access, but bulk access can be provided by contacting me.

Thanks for your support over the years, and continue to encourage transparent access to your local, state, and federal governments!

Oil Change Details: Crush Washer

We own a 2000 Honda Civic that we have been limping along for miscellaneous errands and trips while living in Chicago.  The last time I changed the oil (the first time changing it after we took ownership of the car), it had a slow drip from the oil pan/drain plug.  I tightened it down a bit harder than I would have liked, the drip seemed to go away, and I called it a day.

~4500 miles later, we are due for another oil change and I recalled the drip.  When the oil was changed last time, the “crush washer” was not changed.  When picking up the new filter from Autozone, I asked at the counter if they had such washers for sale and the person behind the counter had never heard of a “crush washer”, the washer on the drain plug, being changed as a routine part of an oil change.  This time, however, I was prepared – I had purchased a small bag of aluminum washers of the correct inner and outer diameters for this application a few weeks ago.

Said “crush washer” is really just an aluminum washer, likely just the normal 6063 machining/hardware aluminum alloy.  They may not need to be replaced with every change, but if you do not and then notice a leak… you’ll end up changing your oil again just to replace a $0.05 washer.  After each future change of mine, I will probably inspect the washer carefully, but err on the side of replacing it.  I took a few pictures of the old washer from our Civic alongside the new washer, which clearly shows the damage/abuse leading to its replacement.

 

Here are two side-by-side pictures of the new/old washers, the first showing one side, the second showing the other.  The first picture shows a lip on the inside of the old washer (on the right), likely from overtightening.  This lip alone would cause a slow leak unless the washer and lip sit just the right way on the plug and oil pan between changes, as well as sitting “just right” with any other debris that may have been on the plug, washer, and pan in the past (causing the other damage/impressions in the aluminum).  This is why replacing this, and reinstalling the oil plug with only a reasonable amount of torque, solves the drip.  Overtightening on the old washer just to stop a drip risks damage to the plug and oil pan threads – so as mentioned above, replacing the $0.05 washer is a much safer bet. IMG_20140705_163545
IMG_20140705_163534

Mobile device IPv6 on kag2d.com and others

When I came across this Reddit post about IPv6 traffic sources broken down by mobile OS, I was a bit leery of the results.  While I never explicitly looked in the past, I didn’t recall seeing nearly that much Blackberry and Windows OS phones on kag2d.com, soldat.pl, thd.vg and their subdomains/related sites.

 

I did a quick grep of our logs for the past ~5 days, very simply for the strings iPhone, Android, ‘Windows Phone’, and BlackBerry – and came up with very different results than the article showed.

Here is a Google Drive spreadsheet I’ve put together with the raw data from the past 5 and 70 days.  Note that this is a basic measurement, I don’t pay attention to versions of each OS.  Also, the article above may have a “long tail” of other versions as a result, I don’t – but that should not be significant to the measurements.

Important differences from the above article’s data:

  • We have not seen a single Blackberry IPv6 hit in the last 70 days, probably longer.  On the other hand, BlackBerry OS 10 alone is given almost 6% in the article’s number.  It just doesn’t make sense…
  • The article shows Windows Phone OS 8 being responsible for 12% of v6 from mobile IPv6.  My data, aggregating all versions of Windows Phone together, is approximately 1.3%.  The article does not cite the sources for their data, but if their numbers are accurate, they must be pulling data from a site which is very heavy in Windows user and/or developer traffic

As a result, Android and iPhone dwarf the statistics – whereas the article shows much lower levels of each (especially iPhone).  This can possibly be attributed due to their numbers having more buckets/a longer tail, so the iPhone/Android is likely chopped off… but something still doesn’t “smell” right.  Their numbers of the more recent versions of each should be significantly higher.

Much of this can be chalked up to differences in content and viewership, but the fact that we have not seen any v6 hits from Windows Phone or Blackberry in at least 70 days (did not look back further) suggests to me that something else is causing a discrepancy between our numbers.

I will be contacting the editors of GCN to get them to provide sources for their raw data, because their numbers don’t sound right to me and some colleagues who have done similar analyses.

IPv6: Name and Shame

After a discussion on IRC tonight, it didn’t take long to come up with a short list of key technologically-involved companies who have a complete or near complete lack of public IPv6 deployment progress.  This list is mostly focused on content providers, or those who others rely on to provide their content (short of ISPs):

  • Canonical (Ubuntu’s corporate overlords):no IPv6 peers up, much less prefixes announced.  As such, no IPv6 available for the default repositories, sites, wikis, bug trackers and many other systems involved in supporting Ubuntu.  Here’s a bug report against Ubuntu and a RT ticket aimed at the infrastructure folk.  Neither has received a proper reply in years.
  • Sourceforge: (and github et al) While a couple download mirrors have IPv6 enabled, the site itself has nothing – despite hosting thousands of forward-looking FOSS projects of all sizes.  A fair share of those are tools specifically for IPv6 undoubtedly, but hosted somewhere off the IPv6 Internet
  • Slashdot: (and others controlled by its corporate overlords) Similar story here – Slashdot has had more than its fair share of IPv6-related stories over the years, and stories of IPv4 runout.  However Slashdot does not have any visible IPv6-related deployment progress despite years of people asking in comments to IPv6-related stories “Why doesn’t /. have IPv6 yet?”
  • kernel.org: along with the OSU OSL, not sure what else needs to be said here.  They’re probably one of the few major, major software mirrors out there who don’t support IPv6
  • Microsoft: Be it microsoft.com (which had IPv6 for W6L but removed it shortly thereafter), Microsoft Azure, or the lack of IPv6 on Windows-supporting infrastructure such as Windows Update content servers; MS has a surprisingly small amount of IPv6 success on their own infrastructure, despite an operating system which supports IPv6 extremely well.  Particularly MS Windows Azure, their cloud platform, it is amazing that such a massive cloud infrastructure could be deployed without IPv6 from day one
  • Amazon: Same goes for Amazon AWS – no public information, roadmap, or statements about the intent to really make progress on deploying IPv6.  Additionally, I’m sure a nontrivial number of compute instances used by organizations such as IPv6-connected colleges could be run IPv6-only and help conserve IPv4 number resources that I’m sure Amazon is chewing through (both public addresses and RFC1918 space which I suspect they are using many times over).  Yes, the ELB offers limited IPv6 support for services which can be published through it, but that was unveiled quite some time ago with zero progress (that I’m aware of – feel free to correct me) since then.  This includes newer services like Route 53 and Glacier which have come out since Amazon had support on the ELB (indicating that someone is working on pushing an IPv6 agenda).  These brand new services launched without IPv6.  Lastly, there are companies such as ours whose entire usage of AWS would be over IPv6 (largely off-site backups and some monitoring) if it was available for services such as S3.
  • CPanel:  A bit different than the others listed – but CPanel is a building foundation (love it or hate it) for probably millions of sites worldwide.  The hosts of these sites cannot offer IPv6 support to their customers until CPanel itself offers IPv6 support.  Think about what the IPv6-ready content world would look like if CPanel wasn’t such a blocking factor for so many hosts.
  • Tumblr, WordPress.com, et al: These companies host an impressively large number of sites for  individuals of all popularities and organizations of all shapes and sizes.  Though they serve a different purpose than CPanel, they have a similarity in that they advertise simplicity of management, ease of publishing, and are a blocking factor for thousands upon thousands of sites.  (Disclaimer: WordPress’s software works flawlessly with IPv6 when hosted elsewhere; hat tip: blogspot is IPv6-enabled since Google enabled it for W6L)
  • Network Solutions: Network Solutions has been behind the ball on v6 for years.  In their DNS management interface you cannot view or edit any AAAA records for hosted zones, let alone try and register DNS glue or inquire about IPv6 for any of their other solutions.  These requests take an e-mail to their support address and can take days or weeks to see progress on.  It probably goes without saying that none of their infrastructure or services (such as their DNS servers) have IPv6 deployed (contrast with GoDaddy who seems to be making decent progress).  A quick search of the NANOG archives turns up some other anecdotal evidence.  Also see that they have no IPv6 prefixes announced or peers up in BGP.
  • Sendgrid et al: Sendgrid has confirmed for me that they have no IPv6 deployment and do not plan on deploying it until “more receivers move to IPv6″.  This includes a lack of access to their APIs, as they too have no IPv6 peering up and no IPv6 prefixes from ARIN (based on a whois search).  This is significant because companies such as Sendgrid strive to be a replacement for conventional datacenter infrastructure but fail to be the lowest common denominator between what customers may want or need.  That is, if someone has an IPv6-enabled mail infrastructure in the datacenter, moving to a hosted 3rd party solution like this requires a step backward.
More generally:
  • Many Others who publish articles/stories (not as much for syndicated stories, but original content) regarding the urgency for IPv6 deployment yet run their own sites without IPv6 accessibility.  They should be practicing what they preach and getting IPv6 deployed on their infrastructure.  In the case that they are hosted/managed by a company which does not support IPv6, they should vote with their wallets and lead by example for others.
  • Game Developers: I wrote a devlog post for King Arthur’s Gold earlier this year regarding IPv6.  The associated forum post has some additional follow-ups with more of my thoughts on the matter. Games stand to feel major hurt as CGN and IPv4 instability/unreliability increase and general quality decreases over the coming years.  In addition to problems with the general IP-literal-centric nature of gaming, I also discuss some issues with the commonplace behavior of IP address bans in many games, and how with CGN this is going to have unexpected consequences.

 

Have suggestions for others?  Leave them in the comments and they’ll gladly be added to the list above.

Have some free time?  Call/write/tweet the companies above early and often, asking for IPv6.  These companies (and plenty of others) need to realize that it is in demand explicitly from many customers, countless more who are passively and quietly waiting, and many many more who implicitly would benefit from its deployment.  Call your ISPs, e-mail support for your hosts, tweet Sprint, AT&T et al today!

These companies are ultimately no different from countless others who do not have IPv6 deployed and no public plans to – except these companies for the most part provide critical services and resources relied upon by countless individuals and organizations worldwide.  As such, you would think they should be amongst the first to have an extremely mature IPv6 deployment and be leading the pack.

 

As a follow-up to some comments on Reddit, I am adding some extra explanation here as to why it is time to name, shame, and generally pressure companies who are behind on IPv6 deployment.  Carrier Grade Nat (CGN) is increasing in popularity amongst mobile providers, ISPs of various sizes (though not some of the major ISPs in the US yet) and campus networks such as universities.  The more traffic remains on or is lit up on IPv4, the more traffic must flow through these bottlenecks.  Forcing traffic through a stateful chokepoint like this is inherently hard to scale properly.  The same goes for operation of an IPv6-only network like T-Mobile’s which uses NAT64 for reaching v4-only content.  NAT64 and CGN are transition mechanisms which need to see a decrease in IPv4 traffic over time in order to help bridge the gap.  Network operators have incentive to push content providers to get v6 deployed, as native IPv6 will forever be cheaper to maintain than operating/scaling CGN devices as the IPv4 content flow through them continues to increase.  We saw this a few months ago with one university’s open letter to Pandora asking them to please get IPv6 deployed, presumably since that is a non-trivial amount of IPv4 traffic flowing through their NAT boxes – additionally it maintains at least one long-running HTTP connection/state to Pandora’s content servers.  Both of these are expensive resources to offer to students when operating a mandatory, stateful chokepoint.

A number of people ask “why should content be made available on IPv6 when there are not really any users on IPv6-only yet?”.  The answer is that when content is available through CGN and NAT64 instead of native IPv6, this bottleneck inherently introduces an extra point of potential failure, congestion and complication.  Additionally traffic may need to go out of its way to reach the CGN outbound and more importantly inbound through a network to reach the user, adding path latency that would not be there if the traffic was able to utilize the networks’ native peering on IPv6 (or IPv4 in the case of a user who did not sit behind the NAT).  Users share IPv4 addresses in these scenarios, which will cause significant problems with bans from sites prone to comment spam (one zombie computer behind a CGN and suddenly the IPV4 /32 or /24 of NAT pool space is suddenly on a RBL and blocked), sites like mega/whatever-upload.com which have daily IPv4 “free download limits”, game bans, IRC/proxy/etc bans, and so on.  These problems are real and are being seen by users behind CGN today.

One last example is systems like the Playstation 3, which does not work properly behind NAT which alters the source port of UDP flows.  By definition of CGN you have many, many users behind a certain IP address.  There is no way you can ensure that each PS3 owner’s traffic will not have its source port changed in translation.  While they may be able to share a source port as long as they are connecting to different game hosts/servers, you and your neighbor may no longer be able to call each other up and join the same game.  This problem is two-fold, as the Playstation 3 has no IPv6 support (not since its launch nor in any subsequent updates).  This means that game developers on the platform haven’t even had the opportunity to start approaching the problem which is late/rushed IPv6 adoption.

The fact is that offering native IPv6 alongside CGN/NAT64 (granted native IPv6 is implied in the latter case) avoids *all* of these problems, along with others involved in IPv4 address depletion and IPv4 routing instability and unreachability as the IPv4 full table continues to grow.  These problems (shared IPv4 address reputation and accounting [in the case of free services], IPv4 quality issues, application breakage through NAT) all boil down to user experience issues.  These users are users of the content/site as well as the ISP, so this arguably puts twice the pressure on the content providers.  Users will complain to their ISP when IPv4-only content (regardless of whether the ISP offers IPv6 or not) is broken, bans them, etc.  The users will also turn to the content provider to voice their displeasure.  Then the ISP will also directly or indirectly have incentive to add pressure on the content providers.

Lastly, and this one is kind of a common sense answer to “why should pressure be exerted on those who have not deployed IPv6 on content yet? It’s not like there are any appreciable IPv6-only userbases?” is this: IPv6-only userbases  *can not* exist until sufficient content is on IPv6.  It’s as simple as that.  There used to be a chicken-and-egg issue where there wasn’t enough demand for IPv6 from content providers, and not enough IPv6-accessible content so that eyeball networks didn’t have demand/reason as well.  This is no longer the case, as 5 of the Alexa top 10 sites and countless others have enabled IPv6 in the past year.  Networks with CGN or soon to have CGN as life support for their IPv4 userbase have all of the reasons above to push for content on IPv6.  This puts the vast majority of responsibility for IPv6 deployment on content providers because they are now the blocking factor.

A few footnotes:

  • This post is almost exclusively talking about the reasoning and pressure for IPv6 deployment on content/content providers.  This post intentionally overlooks/ignores reasoning for deployment on user/eyeball networks though does allude to this not being a “blocking” matter any more.
  • There are plenty of companies who have been pushing IPv6 on the content side in addition to deploying IPv6 themselves.  In no particular order: Google, Hurricane Electric, Netflix, Facebook, Yahoo!, and plenty of others of all sizes.

Explanation of 2006, ‘MySQL server has gone away’ with Python + uwsgi

Earlier today I was working on the King Arthur’s Gold public developer API when I decided to finally get the server side using persistent database connections.  I had approached this before but had issues (which I had forgotten the nature of), so I left it alone in the interest of getting more work done on the API features itself.  This meant that with every single request to the API, I was making a new connection from the Python code to the MySQL database.  This is an expensive operation time-wise when the goal is to make API requests as fast as possible, especially when the Python is being executed on a separate physical machine from that which hosts the databases necessary.

When used with uwsgi, your Python code is more or less executed as a hook of sorts – you have an application() function in the top level of your module which uwsgi calls for each request.  This way it can keep the module compiled into bytecode and in memory persistently instead of forking for every single request.  My code, as written before today, generated the MySQL connection a few modules deeper, in 2 or 3 areas that actually interacted with the user and server databases.   As a result, that code was called for every request as necessary.  In an attempt to make the API more scalable (especially since anything touching the database is or will be my points of contention when scaling horizontally), I wanted to alleviate this.

After giving it a bit of thought, I realized the easiest way to work persistent database connections into this code is to have the database connection and cursor declared at the highest scope as a global variable(s).  So I set ahead and did this – I created the connection and cursor as globals, and the code inside my application() function passed these into the constructors of other methods as needed.  Initially this appeared to work fine, but when I went to do some performance benchmarking (to compare this to the previous revision) with apachebench, I noticed an issue.  During concurrent requests (with -c > 1 passed to ab), the code started throwing 502s shortly after starting.  In the logs I saw “2006, ‘MySQL server has gone away’ with Python + uwsgi'” which I recall from past work in C/++ to mean that either the connection to the database sucks (it didn’t) or I was doing something that was corrupting the database handle.

I figured that the concurrency created by uwsgi was causing multiple worker processes to use the same database handle, and as a result corrupt the state of the connection thus forcing the server to close it.  I confirmed this by doing multiple tests with “ab -c 1 …” and turning uwsgi’s configuration down to a single worker process.  With any number of workers, I noticed that starting uwsgi only resulted in a single connection being opened to the MySQL server (as shown by netstat -n | fgrep 3306).

To further narrow down how it came that separate worker processes were sharing the same value (instance) of my MySQL connection, I declared a global int which application() incremented before proceeding on to handle the request.  I initialized it to 0.

I hit a resource in the API 10 times manually from a browser and saw the int increment to 10 in the logs.  Then I ran ab -c 10 -n 25 and saw the integer continue to increment – but certain numbers were repeated — and never lower than 10.  This indicated to me that the worker processes weren’t sharing memory/instances as much as they were likely being forked from one another in some manner.  Some creative Googling led me to this page on the uwsgi wiki which states:

uWSGI tries to abuse  fork() copy on write whenever possible. By default it will fork after having loaded your applications. If you do not want that behaviour use the –lazy option. Enabling it, will instruct uWSGI to load the applications after each worker’s fork()

This is well and good, but it means that it will clobber the non-thread-safe MySQL handle that is declared at the global scope.  This explains why only one MySQL connection is established when uwsgi starts up, but I see multiple workers incrementing that variable independently.  This is explained by the use of fork() which will share the stack between each child created from the parent in question.

As explained in the quote above, a way to disable this behavior is to start uwsgi with the ‘lazy’ option specified (–lazy if starting from the command line).  Sure enough, as soon as I added this option I could run with my usual 32 worker processes and not experience any issue at all.  Additionally I received a nice performance boost from using persistent database connections instead of churning through thousands of new MySQL connections every minute like the code used to.

Lastly, if you wanted a way to do this without modifying your uwsgi config, you can likely use the @postfork decorator to properly create a new database connection immediately after a worker process is forked.  You can read about that here.

While working on solving this issue, I came across this question posted on serverfault.com, and I have posted roughly this same answer there as well.  With any luck it will help prevent folks like the OP in that thread from having to take to drastic measures — such as switching to pgsql from MySQL (though that may very well work out better for him in the long run).

Bad Customer Service

There was an AskReddit thread about wtf bad customer service experiences, and I felt like typing up two of mine that have stood out (one recent).  Here’s the post as it was on Reddit, I may brush up the formatting at some point:

 

I have two: (hindsight edit MASSIVE WALL OF TEXT, tl;drs at end)

1) My high school graduation present was to be dual 17″ flat panel displays for a computer I had rebuilt (this was ~2004).  I had shopped around and found the model I wanted (some Sony, with VGA input) and a place with a very good price on them, nt-micro.com (thank fuck this place isn’t around any more, but their reviews on reseller ratings are still up apparently).  My dad and I ordered two of them, and about a week later two appropriately-sized boxes showed up at home, win!

I opened them up to find that they were both actually a significantly more expensive model that had better panel quality and DVI+VGA inputs.  As a high school geek I was very happy with my luck, figuring that they must have been out of stock on the ones we ordered and bumped up to the next model.

Upon setting them up, I noticed one had a bright red stuck pixel.  I suspected they wouldn’t be able to do anything about it, but called anyway.  They said (paraphrasing) “sure! here’s your RMA number, just send it back and we’ll replace it, no problem!”.  I said that was awesome (which it was), but clarified that I really wanted the same model in exchange so that the panels were matched.  I said if they couldn’t guarantee that be the case, I’d rather send both back and get the lower model for both.  They said it would be no problem whatsoever.

Fast forward 2-3 weeks after we shipped the stuck-pixeled one back at our own expense, a box shows up.  What do you know, it’s the lower end model.  I called back and explained the situation and said I’d gladly ship either one back, as long as we got a matched pair of working panels.  They apologized for the mix-up and said “let me go check what we have in stock in the warehouse”.  He comes back to the phone and says “yep, we have one of the <better models> sitting right here.  Just ship the wrong one back and we’ll get this out to you”

Fast forward another 2-3 weeks, shipping the lower model back to them at our own expense again.  Another package shows up at home, the box indicates it is the right model this time!

I hooked it up to find that it was the one that was sent originally, with the bright red stuck pixel and the same serial number (which I had written down somewhere just to be diligent).  What do you know the one in the warehouse was there because I HAD SHIPPED IT BACK 1.5 MONTHS PRIOR

So I call back, frustrated at this point.  I explain the whole situation, very patiently.

“Sorry sir, our dead pixel policy is 5 or more stuck pixels or 7+ dead ones, we can’t accept an RMA for that item”.  I spent the next 15 (?) minutes explaining that they already accepted this RMA on more than one occasion in the process of trying to make this correct.  They apologized for that mistake (the mistake of trying to help in the first place when it should have been declined) and essentially stopped talking to me until I hung up.

I was heartbroken.  I didn’t give a shit that it had a little red pixel.  If they had declined the RMA the very first time I wouldn’t have cared in the slightest.  But just that I had put so much time and energy into this just trying to let them make it right as they wanted to.  I felt like so much time had been wasted working with them, and I was continually let down until finally insulted by being treated like crap on that last phone call.  Also ignoring the fact that (iirc) this was a $850 order, and I had spent a bunch of my own part-time-job money shipping these back and forth to process RMAs

~~~ FIN (ish) 1 ~~~

2) This was very recently when I was ordering Comcast at our new apartment online.  I had gone online in very early June to order service starting on July 1st.  I got through the entire order process until the point when you select an installation date, and I saw that they only let you order service 2 weeks in advance.  “Ok, that’s fine – I’ll come back in 2 weeks or so”

I went back around June 18th and went through the same process.  First the special deal “for new customers!” or something very similar in phrasing – $19.99/mo for the first 6mo of <some tier> of service.  Great!  I went through that process (including one or two steps that confirmed I was a new customer with no pre-existing Comcast service) until it said “We are now finding a chat representative for you so you can finalize your order.” Hmm, I didn’t go through that last time.

The rep tells me (clearly not a native English speaker and working off a script of some sort) that the package I selected was for existing Comcast customers.  I said it most certainly was not presented that way and I specifically was asked if I was a new customer while selecting it.  He explained that it was a package for existing Comcast customers who are new Internet service customers.  total wtf because he failed at explaining that the first few times he tried, and the site had enough information that it should not have led me down that path.  Whatever, so now it’s $29.99.  Even if something I misunderstood technically said it right, I was slightly bitter because it felt like a bait and switch (or more accurately, it felt intentionally misleading)

So now I go and start the process over, choose the right package, etc.  At the screen where I had previously (in early June when I first tried) been shown the calendar to select my installation date, I instead was brought to the same screen “Please wait for a chat representative to join you and finish your order.”  Fuck, I hope I didn’t screw up again…

He came online and explained that he was there to help with the order.  Upon inquiry he said this was the normal process and nothing was wrong (they must have changed their process).

Now, on the previous screen right before being forced to chat with a non-native English speaker, you are prompted for your birth date and SSN for a credit check.  Fine, that’s normal enough for a service where you are billed after service is delivered (and since they likely report back to the credit bureaus, which is good for someone younger like myself).

The first thing this guy does is ask for my SSN and birthdate.  I explained that I just entered that into a previous screen on Comcast’s site and didn’t understand the need to provide it to a chat representative.  He explained that it was for the credit check, I said “I know, but I already gave it for that.”  He said “Yes, but we are not allowed to access the information from that previous form, For Your Security  (TM)” (sarcastic trademark mine…).  I facepalmed so hard he probably heard it in India and explained that I was uncomfortable giving this again because what he’s saying makes no sense to anyone who cares about information security or protecting their personal information.  He insisted it was for my security that he didn’t have access to my information and I had to give it again.  I again refused.  I asked to speak to his supervisor because this was just baffling to me.

He has the supervisor call me, and we talk, and I explain how braindead this policy is.  They both continually insist that this is For Your Security ™ and therefore it makes perfect sense that they aren’t allowed to access this exact same information that was supplied to Comcast.  I continued to explain that I didn’t need to chat with anyone in the first place, I knew exactly what I wanted and thought the change in the order process was silly (and clearly flawed).

Anyway, I asked if there is any way for them to continue without my SSN/birthdate, they said no.  “Ok I’ll give it to you but very begrudgingly so and I want to say for the record that this is ridiculous”.  So I gave it to him on the phone, he gave it to the chat guy, and they finalized my order (and I stressed the same talking points to the chat agent).

So the talking points here:

1) Comcast’s marketing material on their site was very misleading

2) They changed their checkout process to REQUIRE that everyone chats with a non-native English speaker, even if zero assistance is actually needed

3) The process requires the agent asking for data a 2nd time, citing “security protections” which ironically say to me YOU DON’T HAVE ACCESS TO THAT INFORMATION FOR A REASON

footnote: Comcast actually really has their shit together technically – deploying DNSSEC-enabled resolvers for customers and having IPv6 available to a huge swath of their service area.  It’s too bad that their customer service side is apparently ass  (not technical support, haven’t dealt with that yet; no comment) and their politics are ass as well (playing games with their upstream ISPs, see: netflix/comcast/level3 debacle last year)

tl;dr 1: don’t buy hardware from small outlets, and if you do for the love of holyshit check resellerratings/etc first

tl;dr 2: Comcast has most of their non-technical heads up their giant company giant asshole;  “Comcast won’t give us this information [that you already gave them] for security reasons, so you need to give it to me in this chat window”

Recent project: King Arthur’s Gold

So as many of you may know, MM has teamed up with a few other developers such as Max “Geti” Cahill to create King Arthur’s Gold, a new 2D side-scrolling game in what seems to almost be a genre of its own – “Build ‘n’ Kill.”  I can’t really explain the gameplay, so take a look at these videos:

WTF is: King Arthur’s Gold

Let’s Try: King Arthur’s Gold

King Arthur’s Gold First Look

 

There are a good number of new players in the videos and some newb builders, but it should give a general idea of the gameplay.  I started hosting the sites for KAG but didn’t play the game for a while, I didn’t think it appealed to me or that I’d enjoy it.  Finally a few months ago I tried and was shown just how wrong I was.  This game is awesome.

I bought it within 2 or 3 days of first playing.

Since then, I have gotten more involved in the community and started working on the game with MM and Geti, including flying out to San Francisco a few months ago and meeting them after they finished attending GDC.  I have taken on a project of replacing all of the authentication and server list/browser/registration/status code since it is buggy as hell and not (and shouldn’t be) MM/Geti’s main focus/concern.

I’ll hopefully write more about this later, but I ended up settling on the solution of a RESTful API which the game client and server use to accomplish these tasks.  As of this writing, the API has the auth code implemented and an update was released for the client and server to use this new system this past week.  Next up is the server registration/browser functionality, then other fun features like buddy lists/player status and eventually some player statistics.  Time permitting, I’ll hopefully write some more about this functionality as I work on it, including discussions of some of the design decisions I made along the way.  The community is encouraged to use the API to build feature-rich community tools and sites (we all know how much the community development contributed to the success and appeal of Soldat).

API functionality is being documented on the KAG Wiki, there is an overview page accompanied by a category dedicated to API documentation.  I will post more explaining how the API is structured and how to use it – including open-sourcing some of the code used in the game client and server once it is more mature/feature complete.

Follow-up to Verizon Wireless hijacking Google DNS; switched to T-Mobile

Last year I posted about Verizon Wireless hijacking Google DNS‘ resolvers 8.8.4.4 and 8.8.8.8 (breaking one of them).  As far as I know this is still happening, but I am no longer a VZW customer.  I have switched to T-Mobile primarily to vote with my wallet and no longer be a customer of Verizon Wireless.  Part of this is due to differences in opinion I have with them technically and politically, and part of it is because I have had piss-poor support experiences with Verizon as a FiOS customer.

A few times over the past year and a half, I have chatted with Verizon FiOS tech support to ask about the status of their deployment of IPv6.  The results have been comical but insulting to my intelligence at the same time.  I’ll abbreviate the stories, but what I relate here is not embellished in any way.

The first time I contacted them about this, they clearly had little or no idea what I was asking about.  Instead of saying this, or that they hadn’t been trained on this or offering to escalate my case to get a proper answer they made up an answer.  I have posted about this before, they said “Please be assured the issue will be resolved in 2-3 days” which was a flagrant lie.

The second time I contacted them, they told me that Verizon will not be deploying IPv6 until IPv4 no longer works, and that no ISPs are deploying IPv6 until then.  Both parts of this are a lie, as Verizon Business and FiOS appear to be working on adding IPV6 support to their various services — and plenty of ISPs are working on if not nearly done deploying IPv6.

So that all said, I switched to T-Mobile.  I look forward to leaving FiOS when my contract is up in a few months.  I chose T-Mobile because they have a 3G/4G IPv6 beta program open to “friendly” customers, and I wanted a carrier that could do IPv6 on 4G/LTE (like Verizon can).  Their prepaid plans also worked out well for me, they had a “web special” for a $30/mo plan which has unlimited data, unlimited text, and 100 minutes/mo of voice (and more available for $0.10/min).  I went out and bought a Samsung Galaxy Nexus and have not looked back.  T-Mobile’s IPv6 deployment seems to work 100% fine with NAT64 (the phones are not dual-stack connected), but some applications which rely on v4 literals will not work.  No surprise there, and it doesn’t impact me day to day.  It sounds like IPv6 is live on the default T-Mobile APN now, however phones are not provisioned to use it by default [yet].

GoDaddy CashParking – ::1 for all!

Update 06/28/2011 : GoDaddy has fixed this issue – the AAAA record appears to be removed from the wildcard answer provided by the CashParking nameservers.  Thanks to those who read what I had to say about this and tweeted/called accordingly.

At work the other day I noticed that all of our domains we have parked with GoDaddy are getting an AAAA of ::1 returned for all parked domains.  We have quite a few at work, and coincidentally yesterday someone started to point this out on the ipv6-ops mailing list.

nova-dhcp-host111:~ ryan$ host sitereport.org
sitereport.org has address 68.178.232.143
sitereport.org has IPv6 address ::1
Host sitereport.org not found: 3(NXDOMAIN)
nova-dhcp-host111:~ ryan$

This appears to be the case with any domain parked with GoDaddy’s CashParking service.

Upon calling and asking, they insisted that this is intentional and correct behavior, and that if we wish to have the AAAA for ::1 removed from our domains that we will need to be escalated to the development team as there is no way for them to modify the DNS records returned on a per-organization(account) or per-domain level.

Personally I don’t care – it wasn’t my decision to park these domains nor is anything of value being lost.  But look at the larger picture: surveys done of what domains are publishing for AAAA records have been used by some people to gauge the state of IPv6 maturity, and having tens of thousands or more domains with a ::1 greatly inflates these statistics in the “broken” direction.  Additionally, GoDaddy is one of the largest names in domain registration and hosting of web services.  Their ignorance on this matter and unwillingness to correct it does not leave the best impression of their technical competence or ability to serve their customers.

Lastly, for those who do park domains (for whatever reason), they have users who will either:

– Hit a locally-running httpd listening on ::1 even if they do not have outside IPv6 connectivity

– Get connection refused, and the browser may or may not fall back to v4.