Blog IconAccommodating a business’ storage needs all too often creates a snowball effect of infrastructure costs. Particularly for businesses mining Big Data for B.I. purposes – but also in certain industries such as healthcare, media production, or utilities – data growth rates frequently more than double the amount of storage required annually. The end result of this massive reoccurring expenditure is a shifting of funds away from projects that actually matter.

Every terabyte of unnecessary media, particularly for high performing applications, can cost your organization thousands of dollars to acquire, and 3-10 times that much over its life cycle. Unfortunately, most businesses are going about the storage expansion process all wrong, and this is a costly mistake. The crux of this issue is accurate capacity planning and effective capacity management.

When asked why a 70% storage expansion was planned for this year, one infrastructure manager replied: “because that’s what we did last year”.  So why is this method used so frequently? Ultimately, it is easy and safe. Essentially, the size of the ‘bucket’ of available storage this year is used as the basis for next year’s ‘bucket’.  All too often, storage expansion becomes a self-fulfilling prophecy that isn’t actually connected with what matters – the individual drivers of new data. By focusing on data growth, rather than storage growth, organizations can trim the fat off of their infrastructure budget.

This approach sounds great in theory, and does work in practice, but there is a risk here. Under-provisioning of storage could occur if growth estimates are too conservative, and can interrupt your ability to conduct ongoing business. Hence the critical role of accurate, and granular, capacity planning. The current options available for capacity planning are either limited in view or unreasonably expensive.  This is something that we have attempted to address with a new planning tool in our recent blueprint “Tackle Explosive Data Growth on a Tight Storage Budget” (click on the infographic below to go to the project blueprint).


As a side benefit, understanding the sources and requirements of individual data creators doesn’t only ease the strain on your IT budget. It also lets you make smarter long-term investment decisions impacting performance and other end-user requirements, since for the first time you truly understand the inner workings of your ‘buckets’.

A change in mindset is the important first step; move from reacting to data growth to actually mitigating it and shaping how it will occur.

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter
big data word cloud
Avoid a big data mess by learning and applying an architectural approach with Info-Tech’s blueprint, Create a Customized Big Data Architecture and Implementation Plan

Big data is moving beyond hype into solutions that are providing real insight and business value. It is no longer the elephant in the room that nobody wants to talk about: it is a growing ecosystem of data, technology, and resources that everyone wants to understand more about, and do something with.

So why is this ecosystem so complex? Well, big data itself is complex: in fact there still isn’t consensus on what it actually is; there is only a set of attributes that try to describe what it is, and even those aren’t consistent across the industry. Our definition of Big Data is “rapidly increasing amounts of data, generated by multiple sources, in many formats; analyzed for new insights.” Essentially a paraphrase of the traditional 3 V’s: Volume, Variety, Velocity – with added aspects of Veracity and Value.

In contrast, traditional data comes from known sources, at controlled volumes, with understandable content. Therefore, it’s no surprise that big data architecture is different from traditional data architecture. Today’s data architects are trying to understand the ecosystem, and deal with the paradigm shift that big data is causing in their knowledge and capabilities in data architecture.

The big differences in big data architecture include:

  • Big data architecture starts with the data itself, taking a bottom-up approach. Decisions about data influence decisions about components that use data.
  • Big data introduces new data sources such as social media content and streaming data.
  • The enterprise data warehouse (EDW) becomes a source for big data, rather than a destination for transactional data.
  • The variety of big and unstructured data requires new types of persistence.
  • Data persistence is horizontal, not vertical.
  • NoSQL is very different from SQL.

Architecture is much more about making decisions than creating specifications. Big data architecture requires decisions in four primary layers:

  1. Data: what kind of data is part of the organization’s big data value chain?
  2. Data Integration: how is the data captured and integrated for analytics?
  3. Data Persistence: how and where does the data need to be stored for analytics?
  4. Data Analytics: what types of analytics does the organization need to perform?

Understanding how the organization wants to leverage big data through selection of a business pattern will help with decisions about data. Data sources, types, and volumes will influence decisions about data integration and persistence technology. How the data is organized and persisted will influence decisions about what types of analysis technology is required.

Setting principles and guidelines about the use of Open Source Software (OSS) vs. vendor solutions can also influence architecture decisions. Given that most of the big data solutions originated out of OSS, the decision to use or not use OSS is a little more difficult than traditional approaches to the problem.

Without a structured approach to big data architecture, organizations could find themselves in a Big Data Mess: they risk their existing data architecture being unable to handle big data, eventually resulting in a failure that could compromise the entire data environment. Also, they risk solutions being picked in an ad hoc manner, which could cause incompatibility issues down the road.

With the rapid change of big data and associated technologies, governance is critical to maintain structure and organization in the big data environment. Big data architecture will help establish the governance structure and boundaries, and anticipate change. An Architectural Review Board and Change Management processes will be very helpful to ensuring the big data architecture continues to work smoothly and effectively into the future.

Avoid a big data mess by learning and applying an architectural approach to big data with Info-Tech’s blueprint, Create a Customized Big Data Architecture and Implementation Plan.

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter

LiveCollaborationInfo-Tech is launching our first Live Collaboration on Choose the Right Development Tools for Big Data. Live Collaborations are a chance to share best practices with your peers and our analysts through an interactive video conference. Join us on December 11 at 2:00PM EST for a Live Collaboration and gain valuable insight on your next big IT project. REGISTER HERE

Choose the Right Development Tools for Big Data is a new Guided Implementation Blueprint aimed at helping application development managers use the right tools for handling big data. Organizations are increasingly examining big data as a means of analyzing vast amounts of data rapidly. This applies not only to BI initiatives, but also to real time commerce related activities that are based on real time consumer patterns and behavior.

Much of the literature around big data focuses on architectural benefits such as divide and conquer or entity relationships. Little focus is given to the actual tools other than the use of generic programming languages like Java. This limits an application development manager’s ability to provide development tools to maximize productivity. This is the first complexity vector in Big Data tool selection – developer productivity — that can easily lead to increased maintenance costs and future derailment of an important business initiative. Now is the time to consider the right tools or tool chain to help ease development and maintenance burden on IT.

A second complexity vector in Big Data tool selection is integration. Legacy applications were not built to handle Big Data design. From a development perspective, tool bridging now becomes part of the roadmap into Big Data projects. However, this introduces additional complexity around legacy test automation and harnessing. That, in turn, introduces complexity with deployment and release caused by dependencies amongst various bits.

The final complexity vector in Big Data tool selection is a meta project issue around communication. Big Data can disrupt existing architectures. Communication and impact analysis is imperative. But how do we go about discussing these concepts? Classic data flows aren’t enough. We now need to talk about metadata and master data and strive for effective multi domain communication.

Big Data represents some interesting possibilities. Jumping into it without thinking through complexity vectors can result in significant pain later on. Better to plan this out now and improve development velocity and quality over time as more learning happens.

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter

security blogIn this day and age, anyone and everyone is vulnerable to a security breach. Our upcoming Implement an Information Security Management System blueprint will allow you to implement a systematic approach to information security.

Our past approach to securing assets is long outdated; it’s time to secure your most valuable assets like the President, not like Fort Knox. Our upcoming blueprint Build an Optimal Asset Security Services Plan will change your view on securing data in motion.

More often than not, your end users are your weakest link. Our upcoming Implement an Identity Security Services Plan blueprint will help you secure your vulnerable end users.

These projects are currently in progress, and we invite you to participate in all three. Help us make these projects that will really make a difference in security management. Continue reading

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter

The cloud is causing the most significant disruption to the IT organization as we know it. The trend to cloud adoption is picking up speed, and as confidence in cloud services and service providers increases, the adoption curve will accelerate. All parts of the in-house IT organization will be affected.

Historically, IT has controlled the hardware and software in the data center, as well as the delivery channels for connecting end users to data center resources. With cloud, IT is increasingly losing control of its role in application selection and development as the business contracts directly with cloud providers for software solutions, using their own OPEX budget.

The ease of procurement, flexibility, scalability, and more predictive time-to-deploy all make cloud services attractive to the business. If IT cannot move faster and show the business what is possible, it will become increasingly marginalized, and will likely be absorbed into the cloud and the business.

The cloud is marking the end of classic IT and the plan-build-run model, as businesses increasingly move away from the traditional “own and operate” approach to IT. IT needs to adopt a new model, enable-integrate-manage, scale down its operations, and re-tool itself with new capabilities to take on new accountabilities.


Chart graphic

The key capabilities that will bolster IT’s strategic position within the organization are:

Business Strategy:

  • Building a technology-integrated business strategy rather than a technology strategy based on the business strategy.
  • Enabling business agility and growth.
  • Facilitating the innovation mandate.

Technology Leadership:

  • Understanding how your customers exploit technology in their day-to-day lives.
  • Mastering the capabilities and uncertainties that come with rapidly evolving technologies.
  • Seizing opportunities presented by technology innovation.

Service Management:

  • Developing and managing an efficient and flexible infrastructure.
  • Managing your portfolio of investments in business changes involving IT.
  • Optimizing the value, cost, and risk of IT sourcing arrangements.

Big Data Analytics:

  • Processing, discovering, and analyzing massive data sets for deeper insights and more effective decision making.
  • Tying together multiple big data sources including social graph, intent graph, consumption graph, interest graph, and mobile graph.
  • Enabling real-time self-service business intelligence.

The IT leader needs to lead the IT organization through a transition that will change IT’s focus from mainly projects and operations to facilitating the creation of advanced business capabilities. There will be fewer roles and people in IT, and a more strategic focus. It will not be an overnight transition, but it will happen, and those IT leaders who do not take the lead, will find themselves increasingly disconnected from the organization.

Share on FacebookShare on Google+Share on LinkedInTweet about this on Twitter