Erlang Deep Bugs (At least one of which has been fixed recently...)

-
Eons ago, in the paleolithic era (2009) when R13 was fresh and new and neat and The Greatest Thing Ever™ we ran into a little tiny problem.  Ok, it wasn't a little tiny problem, it was a great big honking problem, viz., Shit Stopped Working.

(The cry of "Elaborate Please"rings through the streets of New York)

Fine, fine.
Y'see we run a VoIP service in the Cloud (yeah, yeah. don't snicker) - a fairly standard thing nowadays with phones, calling, users, GUIs, etc.
The thing about phone services, is that people expect to be able to make phone calls.
And most calls stopped going through.  Whats worse, most people couldn't log in to complain that their calls weren't going through, and most people couldn't call us to tell us that they couldn't call us, and so on and so forth.

Notice the use of "most"? Its intentional.  Some of the calls - apparently random ones - worked, but by and large, it would take up to 3 minutes for your call to go through (we automatically hung up on you after three minutes of FIAL anyhow, and that part worked just fine.)

This, of course, at 10:00AM on wednesday. Panic in the streets, fire-alarms going off, the works.

By the way, literally fire-alarms.  The building was evacuated, while I sat in the conference room frantically trying to keep things up on one hand, while debugging like mad on the other.  My CEO, Doug Strahan, in an epically brave and kind gesture, hung out w/ me in the conference room to - as he puts it - "rescue me from myself if things got dire"

Whats the point behind all this?
I'm glad you asked.
It turned out that we had a Process Problem.
Y'see, every  call, every user , every login, (basically everything) was a process. And, given that our system was very distributed (remember that?) we registered these processes globally. And when you wanted to do something (like make a call), you'd basically end up running

gen_server:call({global, SomeProcess}, {make_a_call, With, These, Parameters}).

So?
Nothing wrong with that, eh?
Ah, I'm glad you asked.  It turns out that when you wade through the code for gen_server.erl, you end up with the following
  •  gen_server:call ends up in gen:call
  • gen:call  actually use where/1 to identify the node that the process is on
  • where/1 is actually a shout out to global:safe_whereis_name/1
  • global:safe_whereis_name/1 is actually...
Wait, what?  WTF is global:safe_whereis_name/1?  Clearly not in the docs, is it?

Turns out its a hidden function that runs the following piece of charming code

gen_server:call(global_name_server, {whereis, Name}, infinity)

This was roughly the point at which I started cursing softly but oh so profoundly.  (In case you didn't get it, all requests to gen_server:call({global, SomeProcess}, ...) are serialized through one gen_server!!!!!!!)

Which may be fine for small systems, but when you have a bajillion phone calls and a kajillion web-requests hitting your Highly Distributred And Theoretically Scalable Cluster™, well, this is about as useful as a lockpick made from shoelaces, and is just about as painful as a kick in the nuts. Maybe more.

The solution was really simple - we basically replaced 

gen_server:call({global, SomeProcess}, {make_a_call, With, These, Parameters}).

with

Pid = global:whereis_name(SomeProcess),
gen_server:call(Pid, {make_a_call, With, These, Parameters}).

uploaded stuff, and Boom, everything started working like a charm.
 
I know, I know, what if something happens between the two statements, right? Don't worry about it - the above is a simplified version of what we did, but it should get the point across.  And you get no points for being a wise-ass

 Moral of the story?  I'm really not quite sure.  It was a righteous PITA to track down, but once we (Juan Jose - Erlang-er extraordinaire - and myself) had a rough idea of what was going on, we found it right quick through the simple expedient of wading through the Erlang distro source code.  Which, BTW, is highly readable, and fairly easy to grasp.

 Coda 1:
   Geoff Cant (@archaelus) ran into the same thing back in 3/2011. After some back and forth, it turned out that there was a "deep bug" that required things to be the way they were.

 Coda 2:
   As of R15B01, the bug is gone.  In gen.erl, where/1 is now a shoutout to global:whereis_name/1, and global:safe_whereis_name/1 has ceased to be...  

From the release notes
Calls to global:whereis_name/1 have been substituted for calls to global:safe_whereis_name/1 since the latter is not safe at all.
The reason for not doing this earlier is that setting a global lock masked out a bug concerning the restart of supervised children. The bug has now been fixed by a modification of global:whereis_name/1. (Thanks to Ulf Wiger for code contribution.)
The undocumented function global:safe_whereis_name/1 has been removed. 

Life is definitely looking up. I just wish it had looked up earlier... 

Comments

Post a Comment

Popular posts from this blog

Erlang, Binaries, and Garbage Collection (Sigh)

-
Image
The nice thing about garbage-collection is that it is makes memory management absolutely transparent to you Of course, the above is true right up to the point when it stops being transparent, at which point it does a phase change from nice to clusterf**k of biblical proportions .  There are entire posts, books, and suicide notes that have been written about this, and I have no great desire to add to them, save with the minor comment that paranoia about garbage-collection is still justified , albeit only in the edge-cases for most people. That said, I'll give you that Erlang GC is in many ways much, much better positioned in this regard than, oh, Java, but it still rears up and bites you in the butt when you least expect it. ( More information about Erlang's GC vis-a-vis Java here , and Jesper's truly excellent description of Erlang's GC here ).  A story from the past should serve to illustrate this. Once upon a time, in a magical land far far away - well, Her
Read more

Visualizing Prime Numbers

-
Image
From Jason Davies , we have El Patron de Los Primos , probably one of the coolest prime number visualizer that I've seen.  For every Natural number, he draws a periodic curve that intersects at that number and each of its multiples.  Needless to say, prime numbers are touched by only two curves (itself and 1 ). You can scroll to the right, and move up the prime food change. (The image above is a screen grab.  Go here to see the actual thing)
Read more

Its time to call Bullshit on "Technical Debt"

-
Image
Seriously. You know what " Technical Debt " is, right? Ok, heres the (official?) Wikipedia definition   Technical debt   (also known as   design debt   or   code debt ) is a   neologistic   metaphor referring to the eventual consequences of poor or evolving   software architecture   and   software development   within a   codebase . The debt can be thought of as work that needs to be done before a particular job can be considered complete. As a change is started on a codebase, there is often the need to make other coordinated changes at the same time in other parts of the codebase or documentation. The other required, but uncompleted changes, are considered debt that must be paid at some point in the future. And in this - somewhat purist - sense, " Technical Debt " is absolutely valid. The problem arises in the way it is almost invariably used, viz., as an excuse to not do something. Herewith a simple exercise - the next time you hear someone cite &quo
Read more