Brian's Blog


As highlighted in my previous blogs on Big Data and Advanced Analytics, especially in “When Starting Big Data You Don’t Have to Start Big”, getting started in Big Data neither has to be, nor should be, a “big bang” rollout.

The traditional old-school development approaches tend to be a big bang or waterfall development approach. This approach highlights gathering an exhaustive list of requirements for a potential application then, in a structured way, conducting an application-wide design, development, integration, testing, and deployment. This approach served the world well from the 1950s through the early 1990s but the time to gain a return from this approach if prohibitively long in today’s world.

In the late 1980s and 1990s a less structured lightweight approaches that highlighted a “spiral” development methodology began to take hold. But during that time, there still were minimal industry universal standards that took hold to share best practices, standards, and training capabilities -- that is until February 2001.

In the following link, “Agile for Dummies”, the authors present a thorough discussion as to the fundamentals of Agile and how to get started. To help baseline the purpose of Agile and that Agile is meeting its objectives, the founders of Agile developed the 12 principles that drive the Agile Manifesto. Although these areas are detailed in the book, I summarize them below:

1. The highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

7. Working software is the primary measure of progress.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10. Simplicity — the art of maximizing the amount of work not done — is essential.

11. The best architectures, requirements, and designs emerge from self-organizing teams.

12. At regular intervals, the team reflects on how to become more effective and then tunes and adjusts its behavior accordingly.

Relative to Big Data, we have studies dozens of government Big Data and Advanced Analytics deployments. In thinking through some of the primary best practices in implementing Big Data, it becomes clear as to how a traditional old-school development is not well suited for its development and how Agile is the most effective approach.

In a series of studies over the past year, we at IBM and our partners have identified a series of Big Data and Advanced Analytics best practices. Below, I map these best practices to the Agile philosophy:

  • Big Data is transformational: Like most high-impact application the results of getting started are transformational. But it does not stand to reason that it has to be a big bank implementation. In fact, it is helpful to think about the lower cost, higher impact iterations, or projects, as being the best place to start. If done thoughtfully, the latter iterations of development can actually be self-funded by the earlier iterations. 

  • As the data gets bigger, our innate ability to understand it diminishes. And since this is the case, it makes the most sense to develop Big Data into bite-sized chunks of development to realize incremental return.

  • From studying those organizations that have started, we now know what success looks like. Agile tends to be very outcome-focused and not process or documentation-focused. As a result, in using an Agile approach to developing and rolling out Big Data initiatives, the team has targets of which they can study to target in the light-weight documentation standards of Agile.

  • The journey to success is evolutionary – not revolutionary.  The best practices that we have documented also help to determine the entry point of this evolutionary approach. 

  • Imagine the “Art of the Possible”. Agile welcomes a changing evolution of requirements and harness those changes for the organization’s competitive advantage. This open requirements environment welcomes business users’ to imagine the art of the possible without ruffling the feathers of the development team – in fact, well-trained Agile development teams tend to like the challenge.

All of these factors above are well suited for open communications in a partnership-based relationship between the business users and the Agile development team and encourage business users to use their  imaginations as a project triangulates towards the big returns of big data.



Actions: E-mail | Permalink |

Latest Tweets

Search Blogs