Being a Trusted BI Advisor

In his book called "How to Win at the Sport of Business," Mark Cuban makes an important point that your customers do not know what they don't know. 

If you are a Business Intelligence consultant, this means that your clients may give you the worst advice on how to build a BI application. They may understand their business problems, but not how to solve them with BI. Instead, you as an expert need to tell them how things should be done. 

Of course, if they are to listen to your advice, they must first trust you. Building trust on your BI engagements is one of the most important things you can do. David Maister once wrote,

"The key point is that trust must be earned and deserved. You must do something to give the other people the evidence on which they can base their decision on whether to trust you. You must be willing to give in order to get." 

You cannot demand trust; instead, you must work hard at developing it. Until you have the client's trust, they will more than likely force you to follow their own mistaken approaches instead of listening to your good advice. 

See this Entrepreneur magazine article for Mark's comments.   

BI Clarity

On a Business Intelligence initiative, much of your success will depend upon a very special word--"clarity." Here is how one online dictionary defines it:

noun: clearness or lucidity as to perception or understanding; freedom from indistinctness or ambiguity.

When performing a BI project, everybody must be clear in their understanding of what is going to be delivered, why, when, how, and at what cost. Ambiguities in the project's mission, tactical approach, or deliverables will lead to failure.

I have recently used a low-fidelity wireframe tool from a software vendor named Balsamiq to reduce ambiguity. This mockup tool helps to quickly define the BI initiative in a collaborative approach.

Not only can business users see an example of what is going to be built, they can interact with it. The Balsamiq Mockup application has simulated links for demonstrating button clicks, listbox pull-downs, drill-downs, and other common user interface controls.

Before any real application development starts, the users can test out your plans and bless the user interface design. Issues can be pointed out immediately, not after weeks or months of costly effort.

Mockups force everybody to put ideas into a quasi-working format and validate feasibility.

I am a fan of Balsamiq Mockups. At $79 USD, it is a cheap way to bring clarity to your BI project.

The Biggest BI Blunder

Many Business Intelligence projects struggle and fail. But there may be just one general reason why they go south: the individuals in charge did not treat the initiatives as software application development projects.

Instead, many people with failed projects saw BI as a business initiative or as a financial exercise and ignored decades of best practices of software development lifecycle.

Jim Collins talked about how great companies put the right people on the right bus going in the right direction.

If you try to run a BI project as something other than a software application development effort, then you are immediately starting off on a bad trip with the wrong people on the wrong vehicle.

Wrong Approach

By not understanding that your BI initiative is an SDLC project, you will not follow well-established practices that would improve your chance of success.

You might not even follow basic project management concepts such as the "Magic Triangle" to define the "scope" of what you promise to deliver by what time and at what cost. Experienced SDLC developers understand that in a project you are stuck with three interrelated variables.

Changing one of the following impacts the others. 

• Deadline (the length of time to complete the project)
• Functionality (the scope of the project, the features your provide)
• Cost (the resources on the project)

If you need your BI project to be done quickly, then you must add resources (increase cost) and/or reduce what will be delivered (reduce functionality).

If you want lots of functionality, then you must push back the deadline and/or add resources (increase your costs).

If you need to lower costs, then you have to be willing to eliminate functionality and/or push back the deadline and work with fewer resources. 

Ka-Boom!

Experienced SDLC developers also know the difference between the "Big Bang" approach and a phased, iterative roll-out.

BI initiatives are sure to fail when an influential person gives vague marching orders such as, "I want an enterprise dashboard where everybody in the company can access all data in any way they want. And I need to demo it at the annual conference in three months. Oh, and make sure it works on my iPad, Bob's Android phone, and Sally's iPhone, and Greg's BlackBerry."

Trying to create an entire complex application in one fell swoop is a sure recipe for failure. 

Burning Your Dollars

Successful SDLC projects follow a phased staffing approach. Not all individuals needed on the project should start on Day One. Yet that is a common mistake on failed BI projects.

Imagine building a house. On Day One, you only have a plan and an empty lot. The first step is to dig a hole and pour the foundation. If your custom home builder told the electricians, plumbers, dry-wallers, and painters to come on Day One and just have a seat in lawn chairs to watch and wait for their services to be needed, you would immediately recognize this as a waste of time and money.

Yet bringing in GUI dashboard developers to sit and wait for the database team to finish a repository is common on failed BI projects. It just is not as obviously a waste since IT people look more productive sitting in a Herman Miller chair in a cubicle rather than out front on a lawn chair. 

Wrong Expectations

Experienced software developers know they will not be successful unless they first have a clear definition of the application they are to build. Successful individuals are able to take a fuzzy idea from the business and work jointly with others to properly define and formally document the specifics of what will be done.

The worst BI projects are those where the ideas for the dashboard are in one person's head. This person is always frustrated that nobody else on the project is smart enough to understand what she is thinking. Without formally dumping ideas from that person's head into a well-articulated requirements document, the BI project is doomed. Yet it often happens.

Note: read the "Tappers and Listeners" section in the Heath Brothers' book "Made to Stick" for more on this topic.

The first problem of not knowing what you are building is that you will create the wrong thing; your project has to fail since it can never meet expectations.

There is another problem of not clearly defining what you are building: you will never be done. At some point, the executive paying for the BI project will start to scream about the time and cost. "Why is this BI project taking so long?"

Well, it was never designed with an end goal in mind and as a result the BI developers are continuously running a marathon on a circular track.

Wrong People

If you do not understand that you are running a SDLC project, then you will not employ the right people.

If you view this as a business initiative, you will naturally pick people who understand your business. It just seems to make sense to have the guy in Finance who calculates your business metrics to run the BI project. But without a formal SDLC understanding, Bob in Finance is not going to be able to run a successful software project.  

Repeatedly, I have a certain type of organizations fail in their BI initiative. These have been those where their IT groups have traditionally done operational support; they do not do SDLC projects. They are great at running the networks and keeping e-mail and packaged applications running, but they are not software developers. 

When these types of IT groups get complex BI projects thrust upon them, they typically fail. This is not what they know how to do. 

They should not be blamed; their organization is intentionally designed to provide operational support.

They are inexperienced in software development. Yet it does not occur to the executive that he or she is asking for the impossible from these employees. Instead, the project fails and the executive asks, "Why can't my IT development implement a dashboard?"

Plan for BI Success

Business Intelligence is a software application development endeavor; without that basic worldview in place, your BI project will fail. Instead, you will jump into a BI project with the wrong approach, the wrong expectations, and the wrong people. 

A "Just Do It" command from an executive will not make an enterprise dashboard magically appear. To be successful, you will need experienced people following the right methodologies and best-practices to create your clearly-defined BI application. 

Boris Evelson Predicts BI Future

Well-known Business Intelligence industry analyst Boris Evelson documented his Top 10 predictions for the 2012 BI market. You can read them on his blog.

Boris foresees a continuing trend of end users being able to serve themselves with better BI applications. To meet these needs, software vendors and IT groups must continue to innovate and provide what is needed for self-service BI.

Boris mentions innovations that you would expect, such as mobile BI, big data, BI-specific databases, and the cloud.

I found "Exploration" on his list to be interesting. That must mean "whatever you want to do" type of interaction with the data, going well beyond slicing and dicing a predefined cube structure or painting an ad-hoc report from a simplified data view. Boris seems to be pointing to "limitless BI" data discovery.

I also like Boris' comment about how enterprise standardization on a particular BI product or technology stack is not taking place. Instead, Boris says:

"Enterprises will learn to live with multiple BI tools. Forrester client inquiries about how to live with multiple BI tools far exceed inquiries about platform consolidations."  

As always, Boris makes good points.

BI Vendors use Communities to Serve Customers

BI vendors tend to primarily follow a software production business model and shy away from too much services. One reason is financial as they are often evaluated by a metric calculated using dollars divided by the number of employees. Because of high margins for software, this "revenue per employee" or "profit per employee" metric is much higher when a vendor focuses on selling software and not services.

For a nice visualization of this, see Pingdom blog from earlier in 2011 showing Apple with profits of about $420K per employee (almost tripled since 2008 when it was just $151K). Google ranked next with $336K in profits for each headcount; Microsoft followed with a measly $245K.

With that said, software customers still need help. At a minimum, software vendors must provide documentation and training services. They also need to offer a customer service desk to call when clients are in a pinch. Sometimes, customers do need onsite consulting services.

Some BI vendors have found a way to provide services without having lots of people on staff: they enable their customers to help themselves.

Qlik Technologies has a "QlikCommunity" of more than 62,000 QlikView BI users from around the globe. They report having one hundred new users sign up each business day.

Doug Laird, the VP of Global Marketing at Qlik Technologies, says:

“Today’s business users need to make quicker decisions backed by clean, relevant information. This means that queuing up and waiting for IT to generate a report just doesn’t cut it anymore. This is why QlikView’s self-service Business Discovery solution – backed by QlikCommunity – is providing a boost to these users by enabling them to quickly react to changing market conditions. This combination has taken our support to the next level, and is a true testament to the compelling content and engaging experiences enabled through QlikCommunity today.”

Information Builders, the vendor of WebFOCUS, has a similar service called "Focal Point." Their community web page states there are 5000 developers posting 88,000 messages on 13,000 different topics. Information Builders employs some individuals who watch over the postings, engage when necessary, but ensure that solutions are clearly documented for future searchers.

With this type of self-service question and answer forum, BI software customers can search for existing answers to their questions and interact with peers instead of opening a ticket with a support representative. This frees the software vendor to focus on what it does best--building software--and to be more profitable in the process. 

Text Analytics for Legacy BI Analysis

A stumbling block for businesses trying to modernize legacy computer applications is the sheer volume of program files. An IT organization may own thousands of code libraries, each with thousands of programs. Often, the whereabouts of the original developers of old applications are unknown. 

I have found this to be especially true for legacy end-user 4GL reporting tools such as FOCUS. 

A computer language developed by Information Builders in the mid-1970s, FOCUS became the industry standard as a multi-platform report writer for business end-user communities. With FOCUS, rather than ask the busy IT organization to develop reports, users could build their own.

But instead of being just a report writer, FOCUS was in reality a full application development environment originally designed to replace COBOL. Many enterprising users took advantage of robust features such as online screens, database maintenance, and batch processing  to build very sophisticated systems.

Two or three decades later, many IT shops now struggle to grasp what their FOCUS users developed. Trying to assess the purpose, functionality, usage, and complexity of these legacy applications by manually looking at each program is nearly impossible.

To assist with this type of time-consuming detective work, Partner Intelligence developed text analytics software called the "BI Consolidator." Written in C/C++ with a web browser graphical user interface as well as a command-line batch processor, the application has two main features: 1) automated textual discovery; and 2) automated translation into a new BI product.

For now let us consider only the automated textual discovery feature that called the "scanner."  

Text Scanning

Computer programs are not completely unstructured like an e-mail message or the prose found inside a Word document. Instead, almost all computer programs follow a particular formal syntax which forces them to be at least semi-structured text. This simplifies textual analytics since we know what to expect (for the most part, anyway, as there can be user syntax errors and a fair share of junk). 

Our textual analytic scanner is smart enough to figure out the code dialect, but we provide it with some starting instructions. For example, we can tell the application to perform a very specific scan such as looking for FOCUS-to-WebFOCUS conversion issues, FOCUS metadata to find data formats, SAS statistical features, JCL batch job features, HTML legacy CGI calls, Crystal Reports features, or to parse SQL commands. 

When in a curious mood, we can perform custom ad-hoc textual searches.

While the results pulled from the text can be just displayed on a screen, it is more useful to save these to a database and later analyze the answer set.

Online GUI and Batch Text Scanning

We started with a GUI front-end, but when working with a large number of libraries it quickly becomes tedious to repeatedly point, click, and run. As a result, we modified the Scan program to be alternatively run using a batch script from the command line.

Not only is it easier to use, the scanner runs much faster since we eliminate generating HTML for displaying results within the browser. On our current engagement, we scan close over 200 mainframe libraries containing over 80,000 programs within 15 minutes.

Keyword Frequencies

For many of the scans, the software performs keyword frequency counts. For example, to evaluate conversion issues related to green-screen application development, the scanner searches the text for a variety of FOCUS keywords whose either presence or absence would be significant: 

• MAINTAIN
• -WINDOW, -CRTFORM, -PROMPT, -FULLSCR
• CRTFORM, FIDEL, FI3270 (used within MODIFY)
• PFKEY, SET PF

To help with the accuracy of the scanning, we can apply a variety of criteria on searches such as: 

• Perform case-sensitive search (or uppercase all text first)
• Perform stand-alone search (or allow the token to be embedded within a string)
• Ignore blanks between search tokens (since developers often format code using spaces between words)

Pattern Recognition

Using the results of the keyword searches, we can group specific ones together help identify a pattern of usage within the application. For example, if we group keywords found during a legacy FOCUS 4GL scan, we should recognize one or more of the following archetypes: 

• Reporting App = high number of TABLE (report) requests but few MODIFYs (database updates)
• Online Reporting App = Reporting App with high number of -CRTFORMs (menu screens) or -PROMPTs
• Online Maintenance App = MODIFYs, CRTFORMs (transactional screens), and PFKEY usage
• Batch Maintenance App = MODIFYs with FIXFORM/FREEFORM (transactions) instead of screens
• Multi-Step Batch Job = JCL with various FOCUS and non-FOCUS steps (which implies this application may be difficult to port to a new platform)

Textual Parsing

For some textual analytics, we actually need to parse the semi-structured code and pull out more than just keywords. For example, we often find SQL (structured query language) embedded within reporting applications. Being structured, SQL follows a strict syntax of blocks of code in a specific order of: SELECT; FROM; WHERE; GROUP BY; HAVING; ORDER BY.

This makes it possible to parse the syntax and extract the names of databases, tables, and columns being used in the application. We can also distinguish between the columns showed on the report versus those being used in the selection criteria or for sorting and aggregation.

Standard Content Analysis

With these textual contents extracted and stored inside a database, we can then perform standard reporting as well as custom queries. For example, one well-known client used the scan results to perform a redundancy of their Business Objects environment to evaluate it being replaced with a new web-based solution.

The business sponsor was completely against a one-to-one conversion of these legacy reports. Instead, from the scanned contents of thousands of reports and SQL files, the client was able to identify commonalities and reporting redundancies which enabled them to categorize their BI needs into a dozen buckets. From there, they built a roadmap for replacing their legacy reporting environment with a collection of highly dynamic reporting solutions.

In addition to analytics, we have standard reports that help with the operational aspect of a modernization initiative such as parallel test plans. 

Building a Textual Analytics Engine

When companies need to modernize an application, they often view it as a one-time activity. With this mindset, they might not invest the time and money to build this type of textual analytics scanner and translator. Because we work with a variety of clients with this common need, it made sense for Partner Intelligence to create a reusable tool such as the BI Consolidator.

This application has evolved over time. When we first developed it, it handled SQL-based legacy tools. After that, we enhanced it for the NOMAD and FOCUS 4GL. Since then, we have added features for a variety of products such as SAS, QMF/SQL, Oracle Portal, and SAP Business Objects (Crystal Reports, Deski, and Webi).

In addition to the reporting tools, we have added features for handling complementary technologies such as metadata schemas, HTML web pages, and mainframe job control language (JCL). 

If you are interested in learning more, I would be happy to discuss the details of our software with you. Contact me at my DLautzenheiser at PartnerPS dot com address.

Survey shows European Decision Makers in Need of Better BI

Information Builders, the software vendor of the enterprise Business Intelligence product WebFOCUS, partnered with market research company Vanson Bourne for insight into how European companies make decisions and to evaluate the impact of those decisions upon business.

In this "No Barrier to Good Decisions" study, Vanson Bourne surveyed over 600 professionals in eight different countries across Europe. Almost all of the respondents (86%) felt that their companies could be making better decisions; 72% felt that a key to achieving faster decisions would be to have better access to information.

However, over half said it was not about more data, but rather easier access to the information already in the enterprise. Forty percent of the respondents wanted simpler BI tools and applications.

The study found that European managers spend over one hour each day looking for information. When viewed financially, this unproductive time costs European organizations with over 1000 employees about 8 million euros annually. The study found that marketing professionals spend even more time than managers looking for data--83 minutes per day (versus 67 for the managers).

Almost all of the people surveyed (88%) feel their organization does not give them the proper tools to do their jobs. 68% believe their company does not provide them with properly formatted information suitable for easy use.

The bottom line? Almost nobody in the survey felt they were capable of making good decisions based on the tools and data provided by their organizations.

Information Builders' Chief Marketing Office Michael Corcoran commented:

"If companies want to increase their competitiveness, they need to improve access to data across all levels of the company. They risk hampering any competitive advantage if they limit the availability of data for business decisions to desktop or laptop users and overlook other mobile devices. In addition, they have to be realistic about the level of control they will have over the devices the data is being accessed on.

At the heart of supporting this goal is the need to create the right infrastructure and support to enable broader access to data over an ever-increasing number of devices, either in real time or using a local device cache." 

Good Business Intelligence goes back to the simple formula of providing the right information at the right time to the right person so he or she can make the right decision and take the right action. Of course, this survey shows that executing a BI initiative effectively is the difficult part. 


For more information, see the press release. 

Starbucks Urges Fragile America to Drop Partisan Government Ways

Howard Schultz, the CEO of Starbucks, urges Americans to speak out against the partisanship in our government.

See his letter.

BI Professionals using Social Networking such as LinkedIn

In a recent study, Jobvite found that 80% of the people they surveyed said their companies use social media sites for recruiting, with the main information sources being LinkedIn, Facebook, and Twitter.

When Jobvite first did this survey four years ago, recruiting through social media was a new idea. Today, it appears to be the predominant method. In fact, almost all (95%) of the 800 U.S. respondents said that they had successfully hired an individual through LinkedIn.

My LinkedIn network has about 2000 direct contacts, many of whom are somehow related to the software industry. Unlike with my Facebook network, I am pretty open to LinkedIn invitations to connect. That brings my "friends of friends" LinkedIn network to over one million people. Of the 120 million people using LinkedIn, I can use my network to search about 16 million of them.

So I was curious to see the statistics for networked individuals displaying business intelligence keywords in their LinkedIn profiles. To keep it manageable, I decided to only look at those individuals living in the United States. I also removed vendor employees from the figures (e.g., for the keyword "Cognos," I ignored individuals who worked for either IBM or IBM Cognos).

Here are counts from my quick research of BI skills claimed by LinkedIn members in my network:

• SAS: 59,300 people living in the USA
• Crystal Reports: 43,600 people
• IBM SPSS: 37,000
• Business Objects: 34,500
• IBM Cognos: 28,700
• Microsoft SSRS: 17,000
• MicroStrategy: 8,300
• Oracle OBIEE: 4,200
• Actuate: 3,900
• QlikTech Qlikview: 1,400
• Information Builders WebFOCUS: 1,300
• BIRT: 1,200
• Pentaho: 900
• JasperSoft Jasper Reports: 800

Of these 14 BI products, 80% of the LinkedIn keywords clustered around two statistical analysis products (SAS and IBM SPSS) and the software products from two mega-vendors (SAP Business Objects and IBM Cognos).

The other mega-vendors Microsoft and Oracle were not far down the list. In fact, I might have slighted them by not being able to better search for the product keywords. 

Surprisingly, I saw a lack of R Stat people in LinkedIn despite the open-source statistical package being in the news and high demand of commercial products--only 126 people in the United States mentioned that skill in their LinkedIn profiles.

With so many people with SAS and SPSS skills, why would there be so few with an open source statistical package? 

Supply and demand must be at play here. If the market has limited demand for open-source products, fewer people will bother learning them. The other open-source BI products were also very far down on the list.

Of course, this is just a snapshot in time of the BI software skills in the United States. Some of these skills will decline in number and some will increase. For example, I believe that legacy tools such as Crystal Reports skills will move down the list and emerging products such as Qlikview will move up.

The bottom line for BI professionals is to maximize your opportunities by participating in social media--especially LinkedIn.

To get your own free copy of this Jobvite survey, click here. For a great infographic, see this.

To see my comments about Information Builders's BI product, go to my WebFOCUS blog. 

BARC Summarizes BI M&A

In a free analysis, BARC's BI Verdict site offers a nice summary of the business intelligence industry's merger and acquisition activities going all the way back to the 1994 purchase of Pilot Lightship by Dun & Bradstreet.

You can see it here.