Forgot your password?
typodupeerror
Sony The Almighty Buck Games

Sony Forgets To Pay For Domain, Hilarity Ensues 276

Posted by samzenpus
from the turning-the-lights-back-on dept.
First time accepted submitter Dragoness Eclectic writes Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them. Oddly, the forums where company reps might have posted some explanation weren't reachable, either. A bit of journalistic investigation by EQ2Wire came across the explanation: SOE forgot to renew the domain registration on SonyOnline.net, the hidden domain that holds all their nameservers. After 7 weeks of non-payment post-expiration, NetworkSolutions reclaimed the domain, sending all access to Sony's games into an internet black hole. Sony has since paid up. SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.

Sony Forgets To Pay For Domain, Hilarity Ensues

Comments Filter:
  • Black hole? (Score:4, Interesting)

    by djupedal (584558) on Wednesday July 16, 2014 @07:58PM (#47471525)
    Hole in someone's head, maybe - after all, a simple spreadsheet to track something this basic or a reminder in a calendar with alerts with someone assigned to keep an eye on things would take care of things like this. They're lucky it wasn't held hostage...
    • Re:Black hole? (Score:4, Insightful)

      by Anonymous Coward on Wednesday July 16, 2014 @08:06PM (#47471555)

      You want to assign someone to keep an eye on things that can be fully automated? Is your hair pointy?

    • Re:Black hole? (Score:5, Insightful)

      by lgw (121541) on Wednesday July 16, 2014 @08:08PM (#47471565) Journal

      Same thing happened to Turbine a couple years back: DDO, LotR, etc all down for exactly the same reason. You wouldn't think this would be that hard to get right, but chances are no one in dev at either company survived from the early days to when the problem happened, so the tribal knowledge was lost.

      • by X0563511 (793323)

        Domains and SSL certificates. It seems nobody can handle these things expiring until OMG IT'S HAPPENING TOMORROW!!!

    • Black hole? (Score:5, Insightful)

      by Bovius (1243040) on Wednesday July 16, 2014 @08:11PM (#47471577)

      This sort of lapse has happened in every company I've worked in, big and small, when the person formerly responsible for this kind of thing leaves the company and someone else has to pick up their responsibilities. Sloppy, unorganized? You betcha. Also what I've come to expect.

      • by Qzukk (229616) on Wednesday July 16, 2014 @09:30PM (#47471951) Journal

        This is apparently my president's nightmare because he will call me at midnight and ask me when our domains and SSL certs expire.

      • Also human (Score:5, Insightful)

        by Sycraft-fu (314770) on Wednesday July 16, 2014 @10:21PM (#47472179)

        Anyone on Slashdot who gets smugly superior about this and how "stupid companies are" is just being a hypocrite. We have ALL forgotten things in our lives. We've all forgotten an event we were supposed to be at, a bill we were supposed to pay, something we were supposed to bring with us. It happens.

        What's more, everyone has been in a situation where something didn't happen because they, and everyone else, assumed someone else was going to deal with it. You don't go and check on everything that ever happens around you or involving you, you mentally categorize things you are and are not responsible for and ignore the latter.

        So ya, companies, which are made up of people, can fuck up too. It's amusing, but perfectly normal.

        • Re:Also human (Score:4, Insightful)

          by Anonymous Coward on Wednesday July 16, 2014 @11:49PM (#47472475)

          Thats why you build in redundancies.

          Companies are STUPID because they've gotten hooked on the idea of "employee efficiency" to the point that employee efficiency is being negatively impacted. In the past, when a mistake was made, you could easily nail multiple employees simply because they were supposed to be watching/covering one another. If one (or more) screwed up, it meant the others weren't doing their job so they all got punished. It cost a lot more in payroll, but it made sure the job got done, on time, correctly (as far as procedures were concerned). Nowadays, GM can't even find ANYONE to pin the blame on for the ignition switch recalls.

          So yeah, companies can fuck up too. But when you can't even find someone within the company you can point to say "that person is the one who fucked up", what does that say about the company?

          • Re:Also human (Score:4, Insightful)

            by Opportunist (166417) on Thursday July 17, 2014 @01:39AM (#47472761)

            This. A billion times this.

            Corporations are stupid for simply assuming that people are automatons. You come to work and do it flawlessly, always following the ISO 9001 standard. Yeah. Sure. And monkeys fly out of my butt.

            People are people and people are making mistakes. Always. Every single day. Anyone in security learns that VERY quickly. And he also learns quickly that you cannot trust humans to be flawless. Not because people are stupid but because people are NOT automatons and make mistakes. Yes, even (actually, especially) if doing the same job for ages. Show me a person who makes no mistakes and I show you a person who does no work!

            Security is FINALLY starting to get wise and build systems that are tolerant of human error. Let's see how long it takes 'til the rest of the system catches on.

        • Re: (Score:2, Troll)

          by Oligonicella (659917)

          Dude/tte, I have five domains. All are paid up through 2020. In 2015 I shall extend them until 2030. It's not hard to do and it's not hard to remember, FFS. For a company that large to neglect is inexcusable. Just buy them for then next 20 years, it doesn't cost all that much.

        • Re:Also human (Score:4, Insightful)

          by Sockatume (732728) on Thursday July 17, 2014 @04:43AM (#47473219)

          The whole point of having a corporation (or any other sort of team for that matter) is that you find ways to be less failure-prone than you are as individuals. You have to do this to offset the fact that a failure of the group affects every member - the cost is multiplied.

      • by Darinbob (1142669)

        It's pretty easy to forget when the renewal date is so far off. Plus if they do everything online and via mail it's easy to lose the reminders and email addresses change all the time in corporations, but snail mail often gets to the right department at least. Even if it's on someone's calendar, that person gets laid off or quits or transfers. IT groups especially have high turnover from top to bottom. If someone pays for the upcoming 5 years, that person will almost certainly be gone from that job when

    • by peragrin (659227)

      the problem is you fire someone and those alerts and reminders and tracking spreadsheets are lost in the change over.

      • Re: (Score:3, Insightful)

        by Anonymous Coward

        When I fire someone, I redirect their email to their supervisor. It's right there in their employment contracts that their work email address and any correspondence are the property of the company (as if that wasn't obvious, but CYA applies). For things like this we have title addresses like dnsadmin@example.com, noc@example.com etc. which are broadcast to several staff responsible for the management of such affairs.

        Also payments such as these are lodged in our recurring expenses ledger and paid by account

    • Re:Black hole? (Score:5, Insightful)

      by geekoid (135745) <dadinportland.yahoo@com> on Wednesday July 16, 2014 @08:25PM (#47471649) Homepage Journal

      " simple spreadsheet to track something"
      that is the bane of corporations. Important info sitting in a spreadsheet, somewhere.

  • 7 weeks? (Score:5, Insightful)

    by MrLogic17 (233498) on Wednesday July 16, 2014 @08:06PM (#47471557) Journal

    Wow, giving the company 7 weeks before Network Solutions took the site down? That's going way above & beyond. The average luser like me would be taken down the day of expiration.
     

    • by geekoid (135745)

      I've gone weeks. For the same reason Sony did, oddly enough.

    • That's the problem with too much leeway.

      The procrastinator in us invariably assumes there is seemingly an infinite amount of time to take care of this... by the time they send the We really fucking mean it this time! notice, well hell, they've been crying "wolf" so long it doesn't mean anything.

      And to be fair, didn't the nerdtastic Mecca website herself forget to renew a certificate recently?

    • by ghn (2469034)

      Managing hundreds of domains here, from various registrars. Expiration date is expiration date with all of them, it just stops working at midnight. Period.

      The only intriguing thing about this article to me is how they got that special treatment of 7 weeks leeway..

  • I went to a training session for our new $50k accounting system. They had forgotten to renew their own license for the training classroom. Took an extra hour to get their tech in there to get it fixed. Yup, should have got up and went home at that point.

    sigh

    We bought it cause it was industry specific (well focused at least) and by a small company that only did this for 20 years. Next year they are bought by a national company and instead of being 1 of 200 customers now we were 1 of 20000 on a minor product.

  • This is why you don't directly use employee email addresses for certain business activities. These activities get their own emails which forward to whoever the responsible person or persons are. Ex. domain_registration@sony.com. Note "forward to", these would not be standalone email addresses that someone has to log in to.
    • by msauve (701917)
      So, forward domain_registration@sony.com to former_employee@sony.com. Let us know how that works out for you.
      • by perpenso (1613749)

        So, forward domain_registration@sony.com to former_employee@sony.com. Let us know how that works out for you.

        That's why I wrote person or persons. Plus when someone is told they are now responsible for or involved in domain registration they go update the recipient list for the email address. There is no need to update some outsider's records. There is no need to get into the former employee's email. It really is an improvement over using employee emails directly.

      • by nobuddy (952985)

        Group mailboxes are a thing, and very useful for such as this.

        • by msauve (701917)
          It still requires tracking and making changes. It's easier to change the local email system than a registrar's database, but in either case, updates must be made to be effective. With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries. If company X acquires company Y, dns@y.com is apt to be forgotten, too.

          You're suggesting a tactical solution to a process issue. Better to have the
          • You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.

            I only hold a couple of dozen domains, but this is exactly what I do. I get notifications from the registrar directly to a specific e-mail address I've set up for that purpose, but I also automatically generate an email to my personal account on the first of each month reminding m
          • With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries.

            Stop thinking in terms of employees, that's the point of this exercise, the email addresses on the distribution list can include functional roles. company_web_site_manager@sony.com, senior_web_admins@sony.com, etc. Basically the slots in the corporate org chart come with an email address based on the function so you don't necessarily have to know who the person in that role is nowadays.

            You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.

            So your calendar server has a list of people rather than your email server, that's not much of a difference.

      • by Solandri (704621)

        So, forward domain_registration@sony.com to former_employee@sony.com. Let us know how that works out for you.

        It works a lot better because if domain registration emails are being sent directly to former_employee@sony.com, then only he knows that domain registrations are being sent to him. There is no record at Sony saying that he was the one getting those emails.

        If you instead have it sent to domain_registration@sony.com with a forwarder, when former_employee is fired, the sysadmin can look at the enti

        • by msauve (701917)
          So, your plan is that former_sysadmin@sony.com makes the change. OK, but how is keeping track of what emails need redirection when an employee arbitrarily changes more reliable than keeping track of when registrations expire, which is known well in advance? Is it somehow easier to remember to grep email accounts for "dns@example.com" than to query a database for "domain_expiration<90days?"

          See the difference? One places responsibility for a mission critical function on an external party, and the other d
  • I long for the good ole days when they actually send out paper invoices in envelopes! ;^)

    And from the archives:

    "In December 1999, Microsoft forgot to renew the domain name Passport.com,
    and so rendered its Hotmail service partially crippled. A Linux
    programmer, Michael Chaney, paid the $35 fee and promptly handed over
    ownership to Microsoft."

    It happened again in 2003:

    http://www.theregister.co.uk/2... [theregister.co.uk]

    Will they ever learn? ;^)

    • by zephvark (1812804)

      I long for the good ole days when they actually send out paper invoices in envelopes! ;^)

      You actually still look at your paper mail? I tend to assume it's all just spam. Then again, I tend to assume that of my email, too. What was the last year we had a communications system that had more signal than noise? It seems to have been a while.

    • by gangien (151940)

      A guy who posted on /. renewed one of the MS domains. Or atleast, that's what I recall.

  • by Rone (46994) on Wednesday July 16, 2014 @08:46PM (#47471737)

    If the address was unread now, it must have been monitored originally.

    What are the chances that the original recipients were RIFed at some point to goose the quarterly numbers?

    • by pla (258480)
      If the address was unread now, it must have been monitored originally.

      Not necessarily - I have a domain. It has a "real" administrative contact email (a throwaway GMail account). I haven't checked it since I had to confirm it as valid (the registration just autorenews - Pssst, SCEA, you live off subscription models, ever thought of using the same damned idea to keep your domains/certs/etc active?).

      Administrative contacts for a domain amount to nothing more than a pre-confirmed spam address. Why the
    • by scsirob (246572) on Thursday July 17, 2014 @06:48AM (#47473481)

      Must have been "support@sony.com"

  • "Hilarity Ensues" (Score:5, Insightful)

    by steelfood (895457) on Wednesday July 16, 2014 @09:08PM (#47471849)

    Hilarity Ensues

    You keep using that word. I do not think it means what you think it means.

    Now, if some domain squatter had taken over the name the moment the domain expired, that would be funny. Giving them 7 weeks is just ... well, sad.

  • Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them.

    Could the users have used another server to connect with each other? Or is this a case of DRM ("Digital Restrictions Management", when properly viewed from the perspective of its effect on the users) [defectivebydesign.org] and, more generally, nonfree software restricting users from running the games with other people?

    • Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them.

      Could the users have used another server to connect with each other?

      Not much of a gamer, I take it? Most, if not all, of the games affected are not peer-to-peer style multiplayer games; they're MMOs. There's no matchmaking servers involved here.

  • by Charliemopps (1157495) on Wednesday July 16, 2014 @11:03PM (#47472333)

    come on guys.. There's lots of reasons to hate on SOE. Hell, I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right. The title of the freek'n thread to announce the disappointment was "CharlieMopps was right, not a troll, there are no frogloks!!!" (paraphrased, the threads been deleted for some time now) If you don't know what thats about you've no reason to hate on SOE. Ok ok, I'm just tryning to point out I have no love for them...

    Anyways... Managing a domain is a pain in the ass. I've worked in a few places with large website, I'm sure a few of you have. Maintaining that domain registration is deceptively difficult. Think about it as if you were the one in charge of it.

    You tell your staff "Register out domain!"
    They go off and come back "well, it appears we can register it for anywhere from 1yr to 5yrs, which you would like?"
    You say "5yrs of course!"
    They tell you "how would you like it billed? We can pay it one time now... or put it on the company credit card?"
    You say "The company card of course! It will renew!"
    ***5yrs later your site goes down***
    How could this happen?!?! An in-depth review shows that the entire team you assigned to take care of that task has either moved on or transfered elsewhere in the company. Doh! Even worse, credit cards only last for 5yrs before they are canceled and reissued, you were doomed from the start. All the phone numbers you gave them were moved, the people gone, and those that answered barely knew what a domain was in the first place. You're biggest fault was apparently setting the renewal so far out. If you'd set it for 1yr at least you could have a repeating process for people to get use to as newhires rolled in and out.

    But wait! There's a "contracts" department that should have cought this!
    Well "contracts" kind of sorts things in order of importance by cost and that domain registration cost what? $20? So that out it between free Twinkie Friday and the new coffee pot... not really on their radar.

    As many times as I've seen this happens it still baffles me to this day why there isn't a service that went something like "$10k per year and you'll never have to worry about any of your domains... ever... pay us, we take care of it"

    anyways, whatever... point is, it's not as simple as it appears on the surface.

    • by ledow (319597)

      IT department.
      List of all domains.
      Expiry date of those domains, culled from WHOIS.

      How hard is it? Ten minute job. And you KNOW what domains you have to use - you've been including them in game titles, software on the systems you put out, and keeping those domains running somewhere.

      This is NOT a huge task. Even for a multi-million dollar company with 10,000 domains. Hell, it's barely an IT task... more an office admin kind of thing (did they have to "renew" their subscription to the newspapers and tech j

    • I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right.

      What do you mean froglok's don't exist, I've seen them. There was a HUGE underground fortress/city FULL of them down south near the Ogre/Troll town in EQOA. High level Froglok paladins and such.

      They were NPC's in the original EQ release too. Ykesha made them playable.

  • Did is what they get for outsourcing IT.

    Even if they still had some stuff still in house things can get lost in the shuffle

  • From: Kazuo Hirai

    To: John Smedly

    Re: SOE Rego

    @#$$%^&*()!) (*$%@#$$%^&*()!)(*$%@#$$@ #$$%^&*()!)(*$%% @#$$%^&*()!)(*$%

  • Another big company that I worked for, curiously also starting with S, had exactly the same problem. With an internal server, so nobody buy the people working there noticed it. Why? I have no idea.

    But once you spend a few years working, you notice that the term "professional" only means "doing it for money". Not "doing it professionally". So please, don't think corporations are in any way more efficient than you are, or that they would or could do a better job. And how should they? It's just people doing fo

  • by HangingChad (677530) on Thursday July 17, 2014 @07:17AM (#47473541) Homepage

    SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.

    The same one used for customer service inquiries.

  • by RoloDMonkey (605266) on Thursday July 17, 2014 @08:45AM (#47473839) Homepage Journal

    I have been doing web work for a decade, and I can tell you this happens all the time. In fact, older employees in marketing have told me horror stories about 800 numbers and mailing addresses that were never set up, misprinted, or never updated.

    I always tell clients that they should set up emails that describe the job/function, like marketing@example.com and webmaster@example.com, and make sure that those emails go to a distribution list that goes to at least two people.

    You wouldn't believe how often critical accounts and webforms are only accessible with the email addresses of Sally the Secretary or William the Webmaster. When they leave, no one knows there is a problem, until it is a big problem.

You are an insult to my intelligence! I demand that you log off immediately.

Working...