Agile, Rails and the Cloud - Pivotal Labs

Agile, Rails, and the Cloud Why your customer should care about Agile, Rails and the Cloud

Ian McFarland, VP Technology

[email protected]

Or... Why companies can’t afford to ignore the efficiencies of modern development approaches

Ian McFarland, VP Technology

[email protected]

You

You • Most or all of you are Rails Developers, Agile Developers, Craftsman Developers, and run a rails shop

• Most or all of you have current deployments • On hardware not owned by your company • ...managed by someone else like Engine Yard? • On EC2? On other ‘Clouds’?

What were you doing before? • • • • •

Java Developer C/C++ Developer PHP Developer COBOL Developer Nothing: Rails/Ruby is my first development platform

• Goat Farmer? Stock Broker? Shoe Salesman?

Pivotal Labs • We make Pivotal Tracker • We’ve been doing agile for 10 years, first in SmallTalk, then in Java, now Rails and JavaScript

• We’ve grown from 20 to 70 since starting the Rails Practice 4 years ago

• We do enablement, but mostly implementation

Why do we build software • Because it’s cool • Because it’s fun • Because it’s exciting • Because we love the challenge • Because it beats the crap out of coal mining • Because employment is a nice thing to have

Why do Companies build software

• To make money • To save money • To manage risk • To satisfy business and government requirements

Building Business Value The revolution in Software is about one thing: Building business value as cheaply and efficiently as possible

So what’s expensive about software • Developers • Defects • Deployment/Operations • Code Maintenance • Change

Three Trends • Agile • Rails • The Cloud

What is it?

What’s in it for Us

What’s in it for Them

Agile

Cleaner, more flexible code; more fun coding; code is more maintainable

Fewer defects, more predictable delivery, more transparency, business determines what’s built

Rails

Cloud

More powerful, more Less effort = lower cost expressive, less grunt Less effort = shorter work, more gets done time to market with less effort, more Less effort = lower TCO fun. Lower TCO, No initial Easy Scaling, capacity investment, lower planning, setup, no operating costs, shorter pagers deployment cycle, no sunk cost

It’s the Economy, Stupid

• Budgets are smaller • The stakes are higher • Departments have to do more with less money • Failure, though always an option, is more catastrophic

So what’s this Agile stuff?

Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Title

Individuals and interactions over processes and tools Goes Here documentation • Bulleted Working software Text over comprehensive Customer collaboration over Here contract negotiation Text Goes • Bulleted Responding to change over following a plan

• Bulleted Text Goes Here That is, while there is value in the items on Bulleted Text Goes • right, the we value the items on Here the left more. • Bulleted Text Goes Here

Kent Beck

James Grenning

Robert C. Martin

Mike Beedle

Jim Highsmith

Steve Mellor

Arie van Bennekum

Andrew Hunt

Ken Schwaber

Alistair Cockburn

Ron Jeffries

Jeff Sutherland

Ward Cunningham

Jon Kern

Dave Thomas

Martin Fowler

Brian Marick © 2001, the above authors this declaration may be freely copied in any form, but only in its entirety through this notice.

That’s nice... How do we do that? • Business Driven: Requirements come from business stakeholders • Iterative Development, with Short Iterations • Test/Behavior Driven Development • Continuous Integration, Continuous Releasability • Pair Programming • Productive Work Environment

Business Driven • Requirements come from business stakeholders • One designated Customer is empowered to make decisions • Priorities are set by that Customer • The Customer can change priorities on anything unstarted • The Customer accepts the work in fine-grained increments • The Customer is intimately aware of progress, and projected completion dates

• Closing the feedback loop is critical

Accept

Reject

TDD/BDD • Good tests tell us when we’ve met the customer requirements • They tell us when we’ve broken behavior that used to work • They tell us when we haven’t, so we can refactor with impunity • Writing tests first keeps us from overdesigning/doing things we don’t need to do

• Writing tests first forces cleaner API design, because we have to call into our own code in order to write it

• It leads to looser coupling and encourages higher cohesion • Good developer testing keeps the cost of change constant

Iterative Development • Because the customer is seeing the work on a daily basis, the feedback cycle is short

• This keeps the cost of change low, by preventing unnecessary work

• It allows for new insights to be gained from the work we’ve already completed, and for those insights to be incorporated into our new code

• Iterations are as short as we can make them

Continuous Integration, Continuous Releasability • Knowing when things break is critical to reducing the cost of fixing defects.

• Keep the build status visible, so you can fix it quickly • A broken build is a ‘stop the line’ event • Continuous releasability does not mean you release every day. • It just means you can. • Releases can be distracting, so weigh the cost of a release against the value it adds to the business.

Pair Programming • Do we really have to pair? • Isn’t Pairing Slower? • I don’t like pairing. • My teammates smell funny. • I don’t want to look stupid.

Do we really have to pair?

•Yes, you do. • ...but only if you want to be efficient • This is one of the least-used practices, and one of the most important.

• And stop whining! You do it already when you get stuck on something.

What do developers really do all day? • Coding • Reading web pages about coding • Stuck on some problem, unsure of: • The right approach • What the API for that object was • How SQL indexes are selected • How bind(this) works in JavaScript • Checking email • Checking news, stock price, staring blankly into space

How does pairing help? • 80/20 rule: You don’t get stuck, so you spend your time on the most interesting part of the code.

• As you eliminate the grunt work (thanks Rails) more of the work requires real thinking, and design

• You talk through design, and refine before you code. • You learn from your pair, everything from design and testing techniques to SQL, CSS, and JavaScript tips.

• Focus matters: Your pair keeps you paying attention, and can smooth over disruptions

How does pairing help? • More developers in a smaller space • How many truly independent fronts are there in your codebase on which you can make progress?

• New team members: You’re really productive the first hour, not marginally productive starting two weeks in

• They have a local sherpa to tell them how the code they’re working on actually works.

• Knowledge Silos: Your bus number approaches ∞

Productive Workspace • Open Workspace • Colocated Developers and Customer • Consistent Pairing Stations • One big screen, 2 keyboards (we use 24” iMacs) • No laptops on the floor • Visible build monitors • Everyone can see the backlog in Tracker • Breakfast, snacks and beverages on hand • Space for interruptions away from the workspace

Title • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here








Why Sustainability Matters • Predictable delivery is at a premium • Tired developers introduce bugs • Developer retention is still important • Good developers are never easy to come by • Ramp-up is still expensive • Team changes expose companies to risk

Why Developer Happiness is Important to the Business • Leading Indicator: Developer Happiness strongly correlated to Developer Productivity Grunt Work = Money Wasted

• Retention Matters • Happy workers are more focused

Rails: Why should businesses care? • Convention over Configuration • Fewer lines of code • More developer comprehensibility • Much greater developer productivity • More extensibility (DSLs & metaprogramming made easy)

• Agile baked into the libraries and the culture ...oh and by the way, it’s Web based!

Rails compared to Java: From an anecdotal perspective • When we started doing Rails, we couldn’t hire Rails developers, because there weren’t any

• So we converted Java developers into Rails developers • The same developers were about 2x as productive in business terms after one month

• Once they actually got good at Rails, they were about 4x as productive

Rails compared to Java: From an anecdotal perspective • Some people report as much as 5-10x • I suspect that’s because for them “Rails” includes “Agile”

• Large client: 10 Pivots, 10 client, vs. 50 offshore • Pivotal Labs had already been doing Agile for 10 years, so we already got the productivity benefit.

The Cloud • The Cloud means you don’t have to buy hardware anymore... ever. And scale is available on demand... if you architect (or refactor) for it.

• Lets you stay focused on where you provide differentiating value. (Unless you’re in the Ops business or have a sick fascination with pagers, you shouldn’t do your own ops.) [See also: Rails, Agile]

• But... enterprise business rules don’t always allow for this, at least for some systems, at least not yet.

How the Cloud changes the Game

• Obvious Quantitative Wins: • Almost zero provisioning time • Scalable on demand • No risk, no capex, low opex • specialization breeds efficiency

How the Cloud changes the Game

• Less Obvious Qualitative Wins: • Spin up test environments that match production • Realistic load testing environments much simpler • Clone production data when debugging • Snapshot production to isolate production issues

So how did we do? • Developers • Defects • Deployment/Ops • Code Maintenance • Change

The ARC Tool Chain • • • • • • • • • • •

Pivotal Tracker RubyMine or TextMate Engine Yard Cloud RSpec/TestUnit, Selenium Cruise.rb GitHub New Relic RPM Scout HopToad Zendesk Lighthouse

Differentiating Value

Differentiating Value: Questions to ask yourself • Does the work I’m doing materially relate to my company’s core line of business?

• Does it provide competitive advantage? • Does it make users happier or more productive? • Does it reduce operating cost, or improve efficiency? • Does it eliminate drudge work, so people can concentrate on higher-value work?

• Could it be done better by an existing library or product? • If I didn’t do this work, would it matter?

Some things that don’t pass that test • Writing accessors (unless you need to change their behavior) • Writing CRUD • Writing Inner Classes • Writing features customers don’t really need anymore • Writing provisioning, monitoring and backup scripts • Writing your own libraries, when existing ones will do • ...unless you’re in that line of business.

Interesting Problems ARC is about getting to where developers can work on the interesting, differentiating problems, at each layer

If giants are around, by all means, stand on their shoulders!

“Classic economic theory, based as it is on an inadequate theory of human motivation, could be revolutionized by accepting the reality of higher human needs, including the impulse to self actualization and the love for the highest values.”

Abraham Maslow

Risk

…and how to manage it

• Transparency exposes risk earlier • Agile planning is reality-based. Tools like Tracker keep you honest. (With your customer and yourself.)

• Cloud-based projects get you live faster, without sunk cost, and respond to growth with linear cost.

The Dirty Secrets • Ruby and Rails still are big resource hogs • The Rails developer community (more than the software) isn’t quite Enterprise Ready

• The Rails Ecosystem has a lot of niches that need filling

• The Enterprise isn’t really ready for the Cloud

Rails Performance • Not as good as Java or C++ • For the most part, good enough. …especially for the enterprise

• Slow is usually because of design and architecture, not execution speed

• And this will change with market pressure

Rails... Title • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here

Ready for the Enterprise?

This is more what they had in mind.... Title • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here • Bulleted Text Goes Here

Is the Enterprise Cloud-Ready

• In general no...

• Lots of CIOs really want everything running on the hardware they spent all this money on.

• There are (sometimes) legitimate concerns about security • There are sometimes legislative restrictions

• But vertical providers are showing the way • People run CRM in the cloud with tools like SalesForce • People run HR in the cloud with tools like Workday • People run financials in the cloud with tools like Intacct

Is the Cloud Enterprise-Ready • It’s getting there • Many entire businesses are running on the cloud today

• The obstacles are being overcome • Tools are improving around managing, provisioning, security, monitoring, etc.

• As the business need is articulated, solutions are built

• The economics will push it the rest of the way

Déjà-Vu all over again In case you don’t remember...

These are exactly the same reasons they said Java would never work in the Enterprise.

Economics always wins • The benefits already outweigh the costs • One proof is that many Fortune 500 companies are seeing benefits in Agile, Rails, and Cloud deployments

• The problems are shallow • We’ve been down this road before

Enterprise is Easy • Internet Scale is the Hard thing • Enterprise is more fussy than hard • Data tends to be deeper instead of wider • Concurrency is in the 10s, not millions • You’re integrating with lots of fussy endpoints

• Enterprise is more about orchestrating services than about building massive throughput

• Hmm... This calls for a scripting language

Less Baby, More Bathwater! • Rewrites are scary... Don’t do them lightly. • Keep what’s already working well, and iterate from there. • Replace systems from the orchestration layer down • Test legacy systems from the edges in • Replace verticals when integrating with them is more expensive

• Refactor your way to happiness, one system at a time.

It’s not about the technology • Businesses don’t care about technology • Technology is a means to an end

Think like a Business • Think about what’s motivating your company’s technology decisions. It’s probably not about ‘doing something innovative’, but more about ‘making sure things get done cheaply, quickly, and well’.

Which leaves us to ask...

• How can companies afford not to be doing Agile, Rails and Cloud

Q&A More Resources http://pivotallabs.com/talks Contact Info @imf or [email protected]

Photo and Text Credits • Agile Manifesto: http://agilemanifesto.org/ • Flickr Photos used under Creative Commons: Cloud by akakumo, Risk Factory by kyz, Laundry by T.M.O.F., Tank engine by Corvair Owner