Nathaniel Eliot

I did an Ignite talk at DevOpsDays Austin on the culture that makes Infochimps work, and they asked me to expand into a blog post.

Infochimps uses many cutting edge tools (Chef, Amazon Web Services, Hadoop, Hbase, ElasticSearch, Flume, MongoDB, Phantom.js, etc. ad nauseum), and we’ve written a number of custom tools to help corral these sometimes wild horses into a working team. Ironfan, our Chef specialization for big-data in the cloud, coordinates the installation and configuration of the many necessary components. Wukong is our Ruby library for Hadoop, combining the flexibility of JRuby with the raw power of MapReduce. Wonderdog is our Hadoop interface to ElasticSearch, allowing us to deliver large amounts of data quickly into a stable and searchable NoSQL data stores. Swineherd, the workflow engine for Hadoop jobs, helps tie all of this together into a coherent framework for running multi-stage data ingestions.

To crib a DevOps aphorism, however, it’s not the technology that makes Infochimps work: it’s the culture. Specifically, it’s about culture that keeps the challenges from all that novel technology manageable.

Our hiring process is a big part of building and maintaining that culture. We have multiple interview passes, to efficiently separate the few who will fit from the large mass of potential hires. The first pass is with our office manager Holly, a sweet lady who weeds out obvious mismatches in personality, interest, or resume. Next is a phone interview with Adam, our technical team lead, to help weed out those with obviously insufficient skill-sets. After that comes a team interview at the office, to do a finer test on the cultural fit, and start sniffing out where the candidate’s skills and interests are. The last hurdle before an offer is a short initial contract job (a week or two long, paid on completion); nothing demonstrates work ethic and development style clearer than actual development work. Although there are two technical passes, in all cases the focus is less on existing experience, and more on attitude and potential: even the most experienced candidate will lack experience with most of our tools, so adaptability and initiative are important traits in a successful hire.

Our management style relies heavily on what we have won in the hiring process: a work environment full of capable, intelligent, and self-motivated people. Management structure is very flat; I regularly consult with the C-level folks, and everyone else, as a particular task requires. Well-defined (but flexible) roles help keep communication open, as it’s usually obvious who should be included in a discussion. The technical leadership is focused on setting and tying together goals, leaving most choices about the implementation to those doing the work, but always available to clarify what choices align best with the bigger picture. Shared language for common pains and frustrations (e.g. spending currency as an analogy for causing developer frustration, or our various terms for types of technical debt) help encourage empathy, and a shared focus on troubleshooting over blame-assignment. Above all, management strives to avoid mandatory overhead for development (i.e. regular status reports and meetings), instead relying on each employee’s good judgment and occasional casual check-ins to decide how they communicate status and needs.

Beyond core management style, Infochimps goes to great lengths to support their employees. Developers aren’t always the best at remembering to eat in the middle of deep code delve, so lunches (and gentle reminders from Holly) are supplied, in addition to a fully stocked kitchen. There’s an employee joy fund, which employees propose and vote on uses: past choices have ranged from “a new coffee maker that doesn’t suck” to bimonthly yoga classes. There are company outings, both formal and impromptu, and some fun and games around the office too (including the occasional Magic: the Gathering free-for-all).

Employee career development is another big key to employee joy. An employee’s focus is largely self-directed, with interest trumping experience in all but the most time- or stability-sensitive projects. To paraphrase Flip, one of our founders, our aim is to make employees awesomely valuable to the open job market, and totally disinterested in it.

Our development culture is heavily agile, embracing elements from Scrum, Kanban, and DevOps without slavish adherence to any of them. Though core technology choices often come from the C-level, good ideas can and do come from anywhere, from the newest hire to the office manager. In a similar way, although operations are my core responsibility, they are not mine alone: we are closer to the ideal of DevOps (or perhaps NoOps+1 or AllOps), in that ultimately everyone shares the goal (and some of the load) of keeping everything operational. Repeatability is key to many of our core products, but we balance with an understanding that automation is best done to address boredom or terror, not just inefficiency; a task must be either be too predictable to be interesting, or too complex to be feasible, to be a good reason to add further infrastructure. We are also consciously risk-taking, preferring failure from audacity to failure from inaction, and failing forward instead of rolling back wherever possible.

Our infrastructure choices are made with similar goals in mind: developer experience and ergonomics are important criteria for tool choice. Resources are open by default (in cultural assumption, where security concerns prevent it in actuality), so that developers may get to what they need to easily. Components should ideally be small, decoupled, and late-binding wherever possible; reducing the interdependence the system improves both how manageable it is, and how flexible your architecture can be in the face of changing business needs. Making infrastructure repeatable (by making it from code, via Ironfan) means that building anew is an attractive option, which can free you from some of the worst of legacy code upgrade cycles. Archiving unused code and data from production systems, as opposed to supporting everything without question, makes the resultant systems easier to understand and trust.

So now that we’ve got this great workplace, what’s next? We foresee (and are even starting to experience) some growing pains as we shift into our enterprise focused work. How do we handle the impedance mismatch between our model and our clients’ models? What do we do as the company grows beyond the size of the monkeysphere? How should we tackle user segmentation and security as we build our Platform out?

Ultimately, the answers boil down to the same thing we have been doing: find the best teammates we can, then tear down any barriers between them and being awesome.

This post was originally written for Infochimps, as a response to a GigaOm editorial titled "Is It Too Late For OpenStack?"

Prognostication seems to be all the rage again this week. Allow me to polish the crystal ball, and take a look into our cloudy future.

The competition isn't between OpenStack vs. Eucalyptus or CloudStack. It's between Amazon's closed AWS API ecosystem and OpenStack's market-driven API process. On the one side is the grandfather of IaaS and several notable open-source descendants, all fulfilling the API contract that Amazon owns wholesale. On the other side are a coalition of large hardware manufacturers, hosting providers, and space agencies, working to define an API (and a core reference implementation) that provides a broader market for innovation among cloud providers. The ecosystem shift of CloudStack, from OpenStack to AWS, is certainly notable as a signal of the AWS API market's viability. Like Kia's choice not to enter the luxury car market, however, it says almost nothing about the market it opted out of.

Amazon isn't going to abandon AWS, but its development will be slower than the collective development on OpenStack. Because they're built as monolithic blocks with different internal interfaces, its open-source imitators will find it harder to share code, and Amazon will probably never open source more than a small percentage of their core code. The AWS API contract will remain largely immutable to outsiders, although Amazon is open to outside input, as the Eucalyptus agreement shows. The hosting ecosystem there will be strongly commodified, which is great for SMB projects, and less exciting for large enterprises and institutions.

OpenStack isn't going away, either. CloudStack's dramatically-timed exit from the standards process is a sign of that process heating up, not cooling down. There are still many big players with their irons in the fire, because they see the longer-term benefits of a market that allows differentiation. OpenStack's focus on API standards has given those players a common set of interfaces, each of which can be fulfilled by separate components; this lets companies focus their development on those parts they do best (be it computers, file stores, or networks), knowing their work will be compatible with others. Large enterprises and institutions are well placed to use the performance advantages this provides, and large hardware manufacturers and service providers will prefer the bigger profit margins they can achieve by selling specifically those advantages. The secondary tool ecosystem here will be a bit smaller, but where AWS-compatible API shims aren't available, and what it lacks in broad size it will make up in more enterprise-level features.

And in the end, tools like Fog (and Ironfan, built on Fog and Chef) will make the differences between the two markets one of taste, and reduce the lock-in to specific providers yet further. Users will be able to freely move between the AWS and OpenStack markets as their needs change, maybe even on a minute-to-minute basis.

So who wins? In a super-market full of good options, ultimately, everyone does.

So my boss recently recommended that I watch Bryan Cantrill's talk on Dtrace. Unfortunately it wasn't until the end of video, after I'd been thoroughly seduced by dreams of widely usable production-safe software testing, that the other shoe dropped: this toy is for Solaris and BSD, and with a license (CDDL) that appears designed to make it difficult to include in the GPL ecosystem.

Thankfully, I'm not the only one to say "fuck that with a largish cactus". Since the instructions to getting it going were somewhat baroque and hard to find, I'm endeavoring to collect a little HOWTO here. Thanks to lazyswamp for the instructions I'm cribbing from, Paul Fox for the Linux port, and Cantrell, Shapiro, and Leventhal for the original code.

# Set up the build environment
sudo aptitude install bison flex zlib1g-dev libelf-dev libc6-dev-i386 binutils-dev libdwarf-dev
git clone https://github.com/dtrace4linux/linux.git dtrace-linux

# Install (or reinstall)
cd dtrace-linux
git pull
make clean
make all
make test # Optional, but good practice and interesting to watch
sudo make unload
sudo make install
sudo make load

Next step is probably getting this into a Chef cookbook, so I can easily deploy it on Infochimp's servers. I might also be convinced to undertake a .deb packaging, if enough people mention an interest. I've not cut one before, but it's a skill worth knowing.

(Edit: No, next step is "learn D". Hrmm . . . )

While going through her effects after her death, I found this in Reesa's email:

Nathan's mom might contact you guys by email just to get a second perspective on his behavior, per the conversation we all had up here the other day about something in Nathan being broken and him unwilling to fix it. She seems to think an intervention would work; I think that he doesn't trust anyone enough for an intervention to have any effect from any of us. After getting her perspective, it sounds like he's had this bullying behavior for quite a long time and just managed to hide it fairly well from most people.

For the sake of a peaceful memorial, I kept my mouth shut about this. It cost me a solid month of lost sleep and shouting at empty air during commutes. The memorial has past, and with it my interest in silence.

Holly Eliot:

I cannot trust you in my life anymore. You slandered me as a bully to my dying wife, an accusation which would be laugable, save that you waited until we were arguing, and that she (sick, heavily drugged, and worried) believed you. I will not rehash those arguments: a bully would have easily won them all, because he could have easily pushed around a woman in a hospital bed, strong-willed or no. That you transmuted my treatment of my dying wife as an equal (even in arguments) into this slander is vile.

I also cannot trust this to private conversation, because you were always better at whispering nasty doubts in quiet corners. This is the third major time you have attempted to pull the rug from under me, and always you have excused such behavior as well-intentioned. This time I leave no room for doubt or excuse. If your intentions are as good as you claim, they are so ill-executed as to be indistinguisable from raw malice. Far more believable is that you are a sad control-freak, who bent her own nose out of shape running into ever firmer boundaries of mine, and who would rather think her son evil than herself at fault.

I will not exclude you from the lives of your grandchildren, but I also feel no urge to foster those relationships. Your shitty behavior leading up to the memorial impressed nobody, and left Dylan uninterested (of his own accord) in visiting with you further. Iliana is too young to know what you've done here, but it will be your responsibility to work through the layers of distrust you have engendered, in arranging any further visits.

I am utterly disinterested in any other interaction with you.

I am sure you will, sometime in the future, try to apologize for this. Know that I have heard your apologies many times, and their sincerity has only been rivaled by their inability to prevent this (or other intervening wrongs). There may come a time when I trust you once more, but I cannot see how you could get there from here. You have backed yourself into a corner, and I've got no sympathy left to help you get out again.


Everyone Else:

Holly Eliot doesn't know me: she has simply played the part, based on a years-old acquaintance. She has slandered my good name repeatedly, causing others and myself many times to doubt that I am the strong, capable, hard-working, honorable, and kind man that years of proof have shown me to be. Reesa spoke passionately against this mistreatment in its earlier instances, and while I'm sad that she finally got tricked by it, I am not convinced.

If you consider yourself my friend, know fully and completely that Holly Eliot isn't, and act accordingly when she next attempts to "intervene" in my life.

In Sadness and Anger,



Nathaniel Eliot

The d20 Monster campaign grew out of several frustrations. While I've been able to get some gaming in during all of this, the adult campaign I'm in is intermittent and not really appropriate for a 13-year-old. I'd missed a whole lot of play time with my son while his mother was fighting me, and all the events of this last year have taken their toll on our previously scheduled game time (a Friday night card-game night). And I'm currently the only one available to run a game regularly at our house, but my last campaign fizzled when the plot got more complex than my ability to manage it.

So the simultaneous goals are:

• Run a game enjoyable to a thirteen-year-old.
• Run a game enjoyable to my adult friends.
• Spend as little energy on prep as possible.

What emerged is a burlesque noir sandbox d20 blog mashup. It will be epic. It will be gritty. It will not be overly serious. It will have serious enemies, who will often try to kill the PCs for their presumption.

It's set in the center of the Sinking Cities, a huge metropolis cursed by She-Who-Lives-Beyond-The-Waves to a slow watery death for a failure to pay tribute. Largely abandoned by the Tawy, the Tengu/Yuan-Ti/Gnoll empire that built it on the backs of human slaves, the metropolis has become a haven for all kinds of monsters (including the human kind) and other miscreants unlucky enough to be dropped there.

Which the PCs (monsters of the non-human sort) have been, by the Overseer of Tawy, a ghoulish vizier to her majesty Hssspatia. In their first two days, they have tangled with lizard men, carnivorous vines, avaricious kobolds, and dangerous humans. They've made enemies of powerful guilds, and friends of fools and the downtrodden. They've been transported, held for ransom, hired for a posh party, assaulted, and tricked into a home invasion. None of them have died yet, but its only a matter of time . . .

(ObDad: Dylan understood the concept immediately, and is picking up the rules quickly. The first session was slow, but also competing for attention with Minecraft; second session he hit his stride. Watching him buff his flail with a bit of liquor and his ifrit's innate fire abilities, and then improvise passing a guard at the party a drink when collared, was great. He also ate a bee, got sat on by a fat lady, and smashed a window to help get party-goers out when the house they were in got attacked. Whoot!)

What will happen next? I have no idea. That's the point of a sandbox: the PCs have been given some starting motivations, and a setting that encourages single-session adventuring from a common point (thus allowing players flexibility in attendance without disrupting the game).

Now it's their job to provide the plot, while I make the setting react try to kill them.

So the overwhelming response seems to be "keep it together", with a couple of "well, what do you intend?"s (and a technical "and my axe!" from cavorite, vis the underlying CMMess problem). The question as to what I want to do with this is fair, and since it was asked twice and I'm lazy: I don't know, exactly.

At first glance, it seems to be mostly to get writing again. I liked writing publicly, but the way I did it required more mental effort than I can spare at the moment. Finding ways to make it fit better into the new routines requires regular practice. Hat-tip to my wonderful wife, for leading by impressive example here.

There's an element of wanting to build the stream of technical braggadocio which seems the calling card of the 21st century alpha-geek. Reading those blogs can be heady pipelines of technical information, but at the expense of general applicability: I've stopped reading some tech blogs simply due to an overwhelming stream of talk about tools I don't use.

So for now, I'm going to stick to one stream and tagging. That's what I'm familiar with, and changing things when I'm not sure how I want them to change seems foolish. It may prove to be "wrong", but (at least in computers) a consistent wrong choice is often much easier to fix than intermittent or poorly applied cleverness. (Speaking of which: hello, Facebook and Twitter!)

Up next: inter-generational D&D for the exhausted.

::tap:: ::tap:: ::tap::
::cough::
"Is this thing on?"

So . . . yeah. I've been pretty quiet for a while.

Lots of reasons for that, and thankfully enough of them were good reasons that it has balanced the bad reasons. Iliana Wednesday Eliot Brown was born March 30th, having decided that another month wait was just not going to do. Dylan continues to be an awesome teenager, alternating between astounding maturity and intentionally childish absurdity, sometimes in the span of a sentence or less. The job with National Instruments failed to go full-time just after Christmas, but I've launched into a new position with InfoChimps, which is far closer aligned to my interests: full of heady challenges on bleeding edge technologies.

But Reesa's back in the hospital, because during the unscannable months of pregnancy cancer took root in her spine and hip, and the chemo and radiation treatments didn't catch it quick enough to avoid a massive cascade of pain and mobility loss. She's recovering, but its slow and frustrating going for her and everyone around her. Between that, last year, and all the other surprise drains on my energy, I hit a wall of exhaustion. In true Eliot fashion, I've been head-butting through it, but at the expense of some of my image (hah!) as a polite and gentle soul.

On the gripping hand¹: there's a good team here, in several respects. It's all mashed up of smaller teams, and the coordination problems and hurt reactions can make it hard to see it working, sometimes. But it's there, and part of what keeps being promised to me is space and time to recuperate.

And since writing is one of my outlets, I should probably get it going again. However, I'm at a bit of an impasse here. The trend in blogging has been toward blogs with a single topic, and that seems to the most sensible response to the state of the technology and the art; there just aren't enough readers and platforms that allow per-topic subscription to allow for multi-topic blogs. But the alternative is that I make three or more blogs, all of which get 1/3 of my already meager writing time.

So what do you think? I can see a few alternatives:

• Buck the trend, be a rebel, and pour it all out in one place. Write about bleeding edge computer technology, tabletop role-playing games for teenagers, and family cancer-survival all in one stream. Tag for those weaklings who must see only one side of it.
• Write 3+ blogs, and aggregate to another for those who want the firehose.
• As above, but each links to the others as part of a weekly links post.
• Some other middle ground I've not thought of, which will hopefully be suggested to me by one of the three of you still reading.

1.  The Gripping Hand, this month's five-pages-a-night reading material. Good stuff.

Okay, lazyweb: Who can recommend a bad-ass financial adviser in the Austin area? I've got a largish mess that I need some to help cleaning up, now that I've got the stable income to do so. Personal recommendations are my best hope for finding one worth their salt.

I still think "negative for metastases" is pretty damn awesome, but "completely normal baby" is currently wrestling it for the Happiest Phrase Evar crown. Wednesday Addams Eliot Brown (provisional name only), ETA: late April.

Watch out, world . . .

The choice is not for or against leaks, in any meaningful sense, though both sides are trying to make it so. Information leaks will happen, and they will happen more as technology advances. People do not understand the baroque and constantly shifting security implications of their actions, and they will be duped. National cybersecurity professionals are not immune.

The choice is between openness and poorly kept secrets. The choice is between everyone being able to reuse the data, or only the authorized and criminals being able to. The choice is between allowing average citizens to know what those working in their name are doing, or keeping them intentionally ignorant in the midst of ongoing information warfare between government, business, and criminal elites.

I prefer openness because its the only way I see that the little guy might not get chewed up and spat out without any recourse. Trusting without the capacity to verify is not governance, but the abdication of that fundamental responsibility. I prefer openness because I believe it's the best way to ensure that government of the people, by the people, for the people, shall not perish from the earth.