devops – dan's tech blog

Devopsdays Berlin 2018; recap

The sixth edition of DevOpsDays Berlin was held over the 12th and 13th of September 2018, and despite the fact that it’s one of the most consistent, longest-running Devopsdays events on the calendar, this year was my first ever opportunity to attend. The tl;dr is that this was a very fine event, and the local organising team clearly has their act together (though they are actively recruiting for new volunteers).

The talks were not recorded, so for the benefit of those who couldn’t make it, I’ve refined my notes from the talks on Day 2 (unfortunately I had to miss Day 1).

Dirk, on trust

The day started with a presentation from Dirk Lehmann of SAP entitled “Trust as Foundation of DevOps”. Dirk has been a DevOps practitioner and booster for years, and I’ve had the pleasure of seeing him speak a handful of times before – as usual, he didn’t disappoint. The central thesis of this talk is right there in the title: that trust, that fickle, difficult, human emotional thing, is the fundamental element of DevOps. He proposed a few ways to think about trust beyond a philosophical concept, and introduced a way of measuring the level of trust within an organisation: by measuring speed. Fast decision making, execution, and turnaround are hallmarks of a trust-based organisation. In terms of fostering a trusting environment, he proposed three key elements: managing team size, cultivating diversity, and embracing “positive failure culture”.

Regarding team size, he looked to the seminal Mythical Man Month, as well as the work of evolutionary psychologist Robin Dunbar (see: Dunbar’s Number). Dirk’s conclusion? 15 people is the largest a team can get before it it breaks – and often it should be smaller. Concerning diversity, the take-away was that teams should organise around projects, products, or goals, instead of the traditional “business unit” (or work-type silos) that have been commonplace for much of the post-industrial era. Finally, citing Google’s research that investing in psychological safety – more than anything else – was critical to making a team work, Dirk argued that embracing failure as a learning experience is paramount.

Yuwei, on mentoring

The next talk was by Yuwei Ren, a Production Engineer at Facebook, entitled “Mentorship: From the receiving end”. Despite the title, most of her talk was about mentorship from the perspective of the mentor, and it was chock full of good tips for anybody engaging in a mentoring relationship.

For example, as a mentor, how do you get off to a good start with your mentee? Start by establishing career goals, and then ask “How can I help get you there?”. Ensure that the expectations are set, and that outcomes are clearly defined by both parties. This is done by establishing a plan to achieve those goals, and setting up milestones along the way. During the process, schedule regular check-ins, and stick to them! Set the bar high, and along with those lofty expectations, make sure that both parties are accountable to each other. The mentor/mentee relationship is a big responsibility, and it must be treated with respect and commitment.

Both parties must acknowledge that disagreements and misunderstanding are inevitable, so empathy is key. Assuming good intentions is important, but that’s only the first part of the equation – both parties need to actively listen and respect each other both professionally and as human beings. Finally, the mentee needs to take responsibility for their career, actively seeking and providing feedback, and acting on the advice of the mentor.

Ken, on DevOps transformations

Next up was Ken Mugrage of Thoughtworks (and fellow member of the DevOpsDays global core) with a talk entitled “Want to successfully adopt DevOps? Change Everything”. His talk was very dense, and to be honest I didn’t quite capture all of it. His four key points to successful DevOps adoption were: redefine words, change your organisation, change your architecture, and use CD to safely deploy more often.

Within a team specifically, and an organisation in general, agreeing on specific definitions is critical. In terms of defining the word DevOps, Ken argued that DevOps should not define a toolset, nor a role, nor a team. Instead, DevOps is verbs: developING and operatING. If you need a noun, why not CAMS or CALMS (upon which much has been written before). Ken then offered his interesting description of DevOps:

A culture where people, regardless of title or background, work together to imagine, develop, deploy, and operate a system.

Words are important, but action is critical, and the best way to implement CA(L)MS is to change how your organisation is, well, organised. Ken invoked Conway’s Law and by way of illustration, explained how just renaming the Ops Team to DevOps team isn’t going to help (nor is any other action that just leaves silos in place). Teams need to be organised differently, which means that the whole organisation might need to be organised differently as well. Furthermore, the success of those teams must be measured by how they deliver business value (not just whether their software functions or not).

Ken then moved into continuous delivery versus continuous deployment, and this is where things got fuzzy for me.

Sponsors!

Three sponsors had an opportunity to say a few words on stage: Chef, Google Cloud, and Quest. I mention this because it’s important to acknowledge the important role that sponsors play in the DevOpsDays series – without them, the events simply wouldn’t be able to happen. Thanks, sponsors!

Tiago, on Microsoft

The final full presentation of the day was from Tiago Pascoal from the Azure DevOps Customer Advisory Team, speaking about how a unit within the Microsoft empire experienced its own DevOps transformation. This one started slow, but once Tiago got going, it ended up being very interesting – so I’ll skip right to the good stuff.

Within the Azure DevOps unit, teams are assembled into 10 to 12 person groups, the composition of which is heterogeneous and organised around products (see: Dirk’s argument above). These teams are self-forming, so everybody decides which team and/or manager they want to work with. Apparently, while 100% of the group gets a choice, typically less than 20% choose to change from their existing situation. (To me, this is wild, and I’d be very interested to try something like this out at some point in my career.)

Tiago invoked Daniel Pink’s popular book Drive, noting that need teams 3 things: autonomy, mastery, and purpose. He added another layer, which he referred to as “Alignment vs Autonomy”. Autonomy is everything related to planning and execution, whereas alignment is the organisation, the team structure, and most importantly, the cadence of work. For example, they work in 3-week sprints, and at the end of the sprint, whatever is in the master branch gets deployed – so be ready.

At an organisational level they track metrics; but NOT the classics like burndown, velocity, original estimate, completed hours, team capacity, or number of bugs found. Individual teams can track those if they wish, but at a high level there’s only one metric that matters: impact. Are they delivering value? Speaking of measuring, in terms of code quality, Tiago explained the “Bug Cap”: n * 5 = x where n is # of engineers, and x is maximum bug limit. If the bug count exceeds the bug cap at any time, the team stops working on new features and moves immediately to stabilise. Neat!

Ignites

After lunch, it was time for everybody’s favourite DevOpsDays tradition: the ignites! I was preparing for my own ignite so I missed the first one (sorry).

Daiany, on relaxing at speed

This was an excellently-delivered lightning that told the (true?) story of two teams and their relative stress levels heading into the GDPR cut-off date. It was fun, light-hearted, and carried a strong message: when you’re doing the DevOps at high velocities, even massive EU legislation can’t stress you out! Well done, Daiany.

Daniel, on serverless

I did a lightning talk on serverless. It was fun. 🙂

Thanks!

First off, thanks to the local DevOpsDays Berlin volunteer group. Also, thanks (again) to the sponsors. Finally, thank-you to my employer, Datadog, for sending me out there to speak. There are so many moving parts to make even a small conference happen.

I hope to make it out next year!

Genesis: Terraforming a New Home for Firefox Crash Reporter

Last year, my esteemed colleague JP Schneider and I were invited to keynote at a couple of conferences last year. We gave two variations on the same talk, entitled “Genesis: Terraforming a New Home for Firefox Crash Reporter”, at each of Hashiconf 2015 and Velocity EU 2015 respectively.

The blurb for these talks is as follows:

Everyone loves talking about different stacks used in migrating applications to DevOps methodologies, but the most important and valuable change is not a technology stack; instead, it is the human stack.

In this keynote address, we will cover the intersection of technology and humans, and walk the audience through a real life example of the move of Firefox crash reporter. The three engineers tasked with this had to build the plane while it was in the air, all without landing or crashing.

As with many projects, hard deadlines and requirements made the team work through a lot of tough decisions and compromises while simultaneously training devs, product managers, managers, and other engineers in the new world of DevOps and Continuous Delivery.

The talks were a lot of fun and were well received by both audiences. We keep things light (including images and quotes from Shakespeare to Mobb Deep) and we keep them honest (both our successes and failures).

The Velocity talk, which at 25 minutes in length is the shorter of the two, is aimed at a more general audience; the Hashiconf talk is longer, and includes a lot more detail about the Hashicorp tools that we used to reach our goals. I hope you enjoy either or both of them. 🙂

Devopsdays Belgium 2014; recap

Devopsdays Belgium 2014 was held over 27 and 28 October in the charming town of Ghent (or Gent, or Gand), Belgium. This event marked the five-year anniversary of the Devopsdays series of conferences – and with such an important milestone in play, expectations were running high. I’m happy to report that those expectations were met.

First off, the basic details:

The Devopsdays format is half-day presentations and half-day open spaces (the programme is available here).
All of the presentations were recorded by BMC and are available on Ustream for your viewing pleasure.
Perusing #devopsdays on Twitter is probably the fastest way to tap into the over-mind of the conference attendees, both in real-time and in retrospect.
The popular Hangops show/podcast did a recap of the event featuring speakers, attendees, and even Patrick Debois himself (if briefly).

Looking at the programme, it’s interesting to note that there were no strictly-technical talks (the same was true, more or less, for the ignites as well). Broadly speaking, the topics ranged from management skills, to brain chemistry, and as far as a post-modern analysis of the Devops movement itself. Indeed, in terms of content, this was one of the most human-centric conferences I’ve ever attended. Five years ago this wouldn’t have been the case and it is an indicator of just how far we’ve come as a community – in other words, huzzah, we’ve finally managed to stop talking about tools constantly.

Concerning the presentations specifically, much has already been written, and I don’t feel like I can add much more to the existing discourse. While I would recommend watching the videos themselves, for those looking for an executive summary of the two mornings, you could do worse than Carlos Sanchez’s posts at InfoQ here, here, and here.

The best part of almost any conference, for me, is the so-called hallway track: those serendipitous encounters near the coffee machine or spontaneous introductions around a lunch table. In many ways, the open space format is a formalisation of the hallway track, and one that works for everybody (not just people who are comfortable approaching strangers). In either case, when combined with the highly-local nature of Devopsdays, it made for a great opportunity to meet and speak with people whom I’d normally never encounter – the perfect environment for challenging discussions and generating fresh ideas.

A great example of this environment came courtesy of Bridget Kromhout, who (bravely) challenged the conference at large to use more inclusive language, especially when referring to groups within our industry (i.e. say “people” instead of “guys”). The subsequent conversation was civil, reasoned, thought-provoking, and – notably – itself inclusive, not only in terms of gender, but culture as well. This was interesting since it reminded everybody that, hey, different languages have different ways of dealing with this, and that while the problem exists everywhere, specific characteristics can vary wildly from region to region.

I also had the pleasure of meeting Dave Mangot who turned me on to something called the “1-1-1 model” (as pioneered by Salesforce). Basically the idea is to donate 1% of profits, product, and time (respectively) to charitable ends. This got us thinking about how we could apply the model to Devopsdays, and by extension, tech conferences in general. Put another way, is it possible to leverage our industry’s many social gatherings as a force for social good? I would submit that the answer is a resounding yes. For example, following the model, a given conference could:

Donate 1% (or more, there’s no upper limit!) of the profits to charity.
Set aside a pool of tickets for people who otherwise wouldn’t be able to afford to attend – or better yet, put together a grant programme to help for the cost of transport, accommodation, etc.
Facilitate an open space where attendees can volunteer to assemble care packages, sort clothing or food donations, or other types of manual activities along those lines.

Personally, I’d love to see Devopsdays adopt a model such as this one. Given the local, grassroots nature of the conference series, it seems like a perfect fit. Perhaps even the upcoming Devopsdays Paris, scheduled for April of 2015? Something to think about.

As always, if you want to chat, feel free to leave a comment, or hit me up on the twitters. Salut!

DevOpsDays Paris 2013, the event that was

I recently helped organise the inaugural Paris edition of the popular DevOpsDays series of conferences. It was a great event, and I’d like to share some of my thoughts and observations here.

The DevOpsDays series of events is, as the name implies, centred around the “devops” movement, and is intended as a way to introduce people to this style of IT workflow, project, and people management. Since this is not related to a programming language, this conference falls outside of the normal sorts of events that Mozilla generally finds itself involved in. I believe this to be a very good thing, and I am proud to have represented Mozilla both as a sponsor of the event, and as a European devops community member.

The audience was mostly French in composition with a not-insignificant number of attendees from both Francophone and non-Francophone European countries. Said audience was a healthy mix of developers, IT operations, and managers across a wide spectrum of company sizes, types, and even industries; frankly, I was impressed at how broad the composition was, and it was refreshing to see interest in the devops movement from such a wide group.

Though the event was held in Paris, all of the talks (with the exception of some of the ignites) were done in English. This was by design – a internally contentious decision that, in my opinion, ultimately proved itself to be the correct one. The open spaces during the afternoon were in a mix of English and French in order to ensure that everybody could participate equally. Concerning the open spaces, we weren’t sure if the format would work here in France, but they were a smash success! Everybody seemed to really enjoy the format as a platform for discussion, debate, and idea-generation. I’d wager that for many of the attendees, it was the first time they’d ever been exposed to such a thing, and my hope is that they can bring the format to others in the future.

Since devops is so new to France, the majority of the presentations themselves were entry-level, and thus not particularly interesting to me directly. That said, there were two presentations that really stood out (and would have held their own even at a more “advanced” event): “CustomerOps” by Alexis Lê-Quôc, and “Map & Territory” by Pierre-Yves Ritschard.

Alexis’ presentation on “CustomerOps” centred around the concept of providing customer support using engineering principles – and, indeed, delivered by engineers themselves. This really hit home for me because in Mozilla IT/Ops, we’re not only the people who build and provide technical infrastructure, but are also the people who provide direct support to the consumers of that infrastructure – a situation that is absolutely not a given in many other companies (i.e. the admins and the customer reps are not the same people). Alexis illustrated the importance of communication, and how to measure success (read: customer satisfaction) in meaningful ways.

Pierre-Yves’ presentation was based on a very interesting philosophical conjecture: that our mental model of the world is not the same as the reality said model attempts to describe. Put another way, a map isn’t actually land, it’s a representation of the territory it describes (hence the title). Therefore, the most valuable models are the ones that can describe reality in useful ways, and it’s in defining “usefulness” that the real effort must be made. In a more applicable sense his thesis was simple: identify your “key metrics” – the numbers which literally describe the success or failure of your business – and make sure you are collecting, analysing, and modelling them above all. Every other metric is either secondary or potentially uninteresting in the first place.

Personally, I spent a lot of time mingling with the attendees, talking about Mozilla, our projects, and our mission. Generally speaking, the first question was, “Can I have one of those Firefox stickers?”, but the second question was, “When can I get my hands on a FirefoxOS phone?” As usual, everybody wanted to see one, and (unfortunately), as usual, I didn’t have one to show them. The more events that I attend on behalf of Mozilla, the more I realise that continues to be a wasted opportunity to promote our most exciting new project. I’ll have to work on this going forward.

Of course, since this was a devops-related event, people were also very curious about if and how Mozilla is implementing devops internally. The overarching theme of devops is communication, so this event was an excellent opportunity to talk about IT at Mozilla, and to promote not only our successes, but dig into our failures as well. This sort of interaction is vital in order to avoid stagnation.

In summary, it was a fine showing for our first Parisian event, and I am looking forward to the next edition. Hopefully I’ll see you there!