Saturday, November 28, 2009

MicroStrategy Celebrates 20th Anniversary

Earlier this month, Matthew Ipri, MicroStrategy's Director of Marketing for the Americas, shared with me an e-mail from Sanju Bansal, MicroStrategy's COO, which coincided with their 20 year anniversary.

On 30 October 2009, Sanju wrote:
I’m delighted our quarterly revenues exceeded $100 million for the first time in our company’s history. This is a great accomplishment achieved through long-term commitment to delivering premium BI technology and service to our customers. Here’s the earnings.

This afternoon, InformationWeek and The Wall Street Journal both published positive online articles regarding our quarterly financials:

InformationWeek
The Wall Street Journal


Compare this year's Q3 performance with last year's and you will see a 39% increase in product license revenues, proving that companies are buying MicroStrategy's software.  In his e-mail, Sanju listed several factors leading to this success:
  • Prospects increasingly believe that MicroStrategy 9 is the best BI platform in the market today
  • We continue to augment our customer portfolio with industry-leaders. New customers this year include Facebook, Tesco Group, Honda Europe, SUBWAY® restaurant chain, Société Générale, Dean Foods Company, U.S. Department of Energy, and Toshiba Europe, just to name a few
  • As current customers expand their BI deployments, they are choosing MicroStrategy for their growing requirements
  • Influential industry analysts such as Gartner are proactively recommending MicroStrategy technology to their clients
  • Innovative offerings, such as the MicroStrategy Reporting Suite, are enabling us to penetrate new markets





Sanju also credited favorable analysis reviews from Gartner and offered a link to a report on MicroStrategy's website.  The document offers a strength, weakness, opportunity, and threat (SWOT) framework for assessing the MicroStrategy 9 BI Platform and related offerings.  Sanju said:
In short, Gartner addresses some of the strengths of our company, including:   
  • The impressive features and performance of MicroStrategy 9
  • Our world-class customer base (many with enterprise-wide BI deployments) and the high number of customers that choose MicroStrategy as their BI standard
  • MicroStrategy’s low total cost of ownership, thanks to our well-integrated platform
  • The high level of support that we provide to our customers

Congratulations to MicroStrategy on their 20th year anniversary and for their success in 2009.

Microsoft Going for Big Corporate BI

On the 23rd of November 2009, SearchDataManagement.com posted an article about Microsoft's latest foray into the Business Intelligence software space, competing with the other megavendors Oracle, SAP, and IBM.

Click on this link to read Jeff Kelly's article.

Back in the 1990s, IT shops did not see Microsoft SQL Server as a true enterprise database and went with vendors such as Oracle or IBM instead.  That attitude has since changed dramatically -- SQL Server now easily competes with large capacity databases such as Oracle, DB2, and Teradata.

Not content with terabytes, Microsoft is working on massive parallel processing capabilities which will take SQL Server into the petabytes size.  On the BI front-end, Microsoft is adding in-memory processing capabilities for fast online analytics.

Even though Microsoft Excel is probably the most common tool for analyzing data, is it really a BI software product? 

Once limited to 64K rows, Excel used to only be appropriate for use at the small departmental or individual level.  Bringing in enterprise data from disparate sources was well beyond the abilities of an end-user.  Even with a "PowerPivot" front-end and powerful database processing, will Microsoft really be able to convince companies that Excel is an effective alternative to true BI products such as Cognos, Business Objects, WebFOCUS, Actuate, and Microstrategy?

DAPPER Methodology for Retiring Legacy BI Software

To convert legacy BI products, Partner Intelligence employs our "DAPPER" approach, proceeding through the following phases:
 
D - Discover
A - Analyze
P - Perform Pilot Conversion
P - Plan for Full Conversion
E - Execute the Plan in a phased approach
R - Retire the legacy BI product
 
 
One of our first steps in a BI conversion is to get an accurate inventory of the existing legacy environment.  Using our automated scanning and inventory application, Partner Intelligence can quickly provide you with details for making a proper decision about a BI rationalization initiative.  Instead of paying analysts to manually look through your code over a period of time, you can use our software application to quickly scan, discover, and report on the functionality buried inside your legacy code. 

Partner Intelligence can quickly and easily customize the scanning tool for your particular needs. We can automatically scan applications written with a variety of software, such as:
  • 3GLs (COBOL, PL/1, etc.)
  • 4GLs (NOMAD, RAMIS, FOCUS, etc.)
  • Mainframe Job Control Language (JCL) and logic in batch steps (SyncSort, DYL280, etc.) 
  • SAS
  • RPG
  • SQL 
  • Crystal Reports

We use this automated scanning to size the conversion project and to determine the feasibility of automatically converting the legacy code into modern web-based BI software.

Contact Partner Intelligence for more information.

Friday, September 25, 2009

Shift Happens

See the latest version of "Shift Happens" for trends in technology and globalization.  It provides amusing (we're not reading newspapers in the toilet anymore), interesting (by 2020, mobile devices will be the dominant mode of accessing the Internet), and scary trends (if you're an American, that is).

Saturday, September 12, 2009

SAS iPhone BI

After reading my blog on using WebFOCUS to run iPhone BI applications, Sy Truong of MetaXceed sent me the following information on an iPhone architecture for SAS:
SAS iPhone App Architecture
The delivery of SAS data and reports to an iPhone application is unique and requires a different architecture as compared to traditional client server systems. This is similar to web applications in how it is delivered to a browser but in this case, the application replaces the browser. The diagram below illustrates the components of the iPhone SAS application architecture including: the iPhone application, a web server and the BI Flash server.

In this example, the iPhone application on the iPhone communicates through standard TCP/IP protocol to a web server. The web server then communicates to an application server which is actually a SAS session processing SAS programs and data. The output resulting from the SAS program is then delivered back to the iPhone in a similar way a web browser would access a web page the serer. The distinction however is that the iPhone application is not a web browser and the SAS session running on the server is more dynamic compare to a static page. The SAS data and macro program it executes may be simple and standard but facilitating the communication takes a little more effort. The request from the iPhone application and the delivery of information from the server is handled by BI Flash. This makes the experience more dynamic and delivers the full power of SAS on the server. The following steps are taken in order to facilitate the access of SAS data from an iPhone App.
 
Step 1 - Download iPhone Application

One of the most unique and successful aspects of the iPhone is how users can easily download an iPhone App directly from the iPhone App Store. There is a user friendly way of searching for and downloading the application to your iPhone. In this case, you can download BI Flash App which is the client component which enables the communication between the iPhone and the SAS server.

 Step 2 - BI Flash Application Server

More details on this step will be explained in the "Application Server" section but an administrator would execute the BI Flash server. This server functions as a listener waiting for a request from the iPhone. Upon receipt, it would process the request similar to how you would submit SAS programs from display manager. The server would generate a SAS log and output results in XML which is then sent back to the iPhone to be viewed.

Step 3 - Connect iPhone to Application Server

Most iPhone applications have configuration settings. This allows for users to easily configure settings during setup and then use the application without further changes unless future configuration changes are needed. In order for the BI Flash example iPhone app to access SAS, the user would need to configure the following:

 

 


Host Name - This the name of the server or an IP address of the SAS server.

 
User Name - A valid user name that has been defined on the server needed during authentication

 
Password - A user defined password to secure access

 
There may be other configuration options which will set the default behavior of the application but the parameters above show the minimum requirement in order to connect to a SAS server.

Step 4 - Run Application

The final step taken by the user to access SAS data is to execute the SAS macros from the iPhone. This request is initiated from the iPhone app and sent directly to the server with user selected options. The results are then returned to the iPhone displaying the most updated information on the server.

The system architecture in this example is rather simple compared to other systems that require multiple layers of middleware. This is similar to the SAS/IntrNet where users are on a web browser accessing SAS data and programs on the server through the broker and SAS application server. The difference however is that the client is not a browser, but rather a dynamic iPhone application.

Searching through the iPhone App Store, I could not locate the BI Flash application that Sy references.  Regardless, the architecture that Sy describes is very similar to that used by WebFOCUS which I will document in the near future.

Wednesday, September 9, 2009

Accidental iPhone BI

I admit it -- I converted to Apple. It started with an accidental iPhone purchase.

I never intended to buy an iPhone. Instead, I planned to replace my Treo with a brand new Palm Pre from the same carrier. Unfortunately for Sprint and Palm, the teenagers working at the strip-mall store lost my order. After weeks of strange phone calls with kids who could never find my order nor know when truck shipments would arrive ("Dude, if I could predict the future, I would be rich and not work for Sprint!"), I simply went to the Apple Store and came out with an amazing iPhone.

While waiting in the red velvet line (this was the weekend of the 3Gs release), I was introduced to the Apple Fan subculture. When not tapping away at the touch screen, iPhone users advocated their favorite downloaded apps with cohorts. Apple employees were professionals, passionate about their work.

Literally within seconds, the iPhone was accessing my personal and corporate e-mail accounts and calendars. I could easily connect to WiFi or the AT&T 3G network to browse websites with the Safari browser. Even secure sites were available through a VPN connection.

I even paid the $2.99 per month fee to listen to Sirius XM satellite music on the iPhone. (I also got a sick feeling about having just paid $150 for an MP3 player and $399 for a Sony Book Reader.) Everything I needed to be productive was now at my fingertips; I stopped waiting for the Windows bootup on my PC except for things that required heavy-duty typing.

So far, two individuals have walked away from my personal iPhone demos to go directly to the Apple Store to buy their own. Yes, the same day. Immediately. Others have changed their Christmas, birthday, and other gift-buying/gift-getting plans. One of my straight-laced employees responded to my iPhone enchantment with, "Sorry, I want my phone to be a phone. That's all it needs to do."

So I added some real sizzle.

On my iPhone, I logged into a secure executive dashboard for a well-known restaurant. The Safari browser visually showed corporate-wide key performance indicators for sales, speed of service, food costs, labor costs, and cash register operations. All right there in my hand, blazing fast. From the total level, I tapped down through the company hierarchy until I reached a specific store. I looked at their raw item inventory to see if they had enough cheese. I pointed out a red stoplight where the manager had failed to staff enough people on cash registers the previous day, causing a negative impact to speed of service and revenues. I tapped to get a trend of the past 20 days to see if that was just a new occurrence.

My friend responded, "Boy, I have to get one of those."


We had created this business intelligence application using the WebFOCUS software product from Information Builders. Initially, the plan was to have on-demand access through a PC web browser and push out static reports to e-mails, printers, and Blackberry devices.

Just like my personal use of the iPhone, the restaurant's corporate use of iPhone was accidental. During the development project, somebody tried to access the dashboard directly with a Blackberry but with no luck. It seems that the WebFOCUS BI Dashboard uses web techniques not supported by the Blackberry. Even if the Blackberry user could get past the initial dashboard screens, the reports did not display well on the handheld.

Not so on the iPhone. The Safari browser is top-notch and allows zoom in-and-out of the screen content. Even client-side scripts, cookie techniques, and secure LDAP integration worked on the iPhone. The iPhone even supported dashboard exports of data to PDF and Excel.

My accidental iPhone purchase has caused all sorts of changes for me and the people around me. I would write more about it, but I have to go sign for a FedEx package -- an Apple MacBook Pro just showed up.

Thursday, August 27, 2009

Take the Technology Job Trend Quiz

Here is a quiz you will not see in Cosmopolitan: How much do you know about today's technology job market?

In each of the following pairs of career activities, there is an A and B option. See if you can select the one in which you are most likely to find a job today:

1. Building computer applications using A) RPG or B) the iPhone programming language?

2. Writing A) COBOL programs or B) Business Objects ad-hoc queries?

3. Creating A) SQR reports or B) ESRI online maps?

4. Writing web applications in A) .NET or B) Java?


In 1980, I encountered my first computer programming job when a friend took me to his workplace to see RPG coding done on an IBM mid-range computer. For decades, RPG developers were in demand. However, if you had searched Monster.com for RPG job postings at the beginning of 2009, you would have only found 117 openings across the entire United States. The number of openings has steadily declined and in August of 2009 was down to 88.

The correct answer to Question 1 is B -- in August of 2009, there were over twice as many Monster postings for people who can develop handheld iPhone applications than for RPG programmers. Apple has sold millions of iPhones and the demand for new applications grows daily.


If you went to college for computer programming, you probably took COBOL. In 1997, Gartner Group estimated that 80% of the world's business applications were using COBOL. But if you picked A for Question 2, you would be wrong. Demand for COBOL programmers has been steadily declining. Monster started the 2009 year with 348 COBOL job postings and went down to 286, an 18% decline. The Business Objects business intelligence product, however, has seen solid demand despite being acquired by SAP, going from 729 job postings in January to 771 in August 2009, a 6% increase.


Specialized technologies for report writing have been popular. For example, the SQR language is a mix of SQL (structured query language) and COBOL. Today, however, you would have troubles finding a job writing reports using SQR. Monster postings declined from 86 at the beginning of 2009 to only 45 in August. Jobs to create geographic information systems using the ESRI technology, on the other hand, increased from 150 to 165 in the same timeframe.


Question 4 is tricky. The jury is still out for standards for building web applications. Possible winners are Windows products such as .NET, C++, and C# or non-Microsoft technologies such as Java. But if you base your guess on the number of Monster job postings, the verdict seems to be Java. At the beginning of this year, there were 3885 .NET job postings. Today, that number has dropped 9%. However, Java jobs have consistently stayed above 5000 most of the year (Monster caps the number of hits at 5000).


As you evaluate your skill sets, here are some things to keep in mind about today's technology trends:

Most companies prefer packaged applications over custom applications.
Companies want low headcount, hence they are willing to buy and implement systems developed by third-party vendors. Applications such as human resources, payroll, accounting, accounts payable, accounts receivable, and so forth are standard and easily acquired. Few firms are so unique that they cannot use a packaged ERP from a mega-vendor such as SAP or Oracle. You are better to have skills related to software package implementation, project management, and functional know-how than in custom application development (unless you want to work for the software vendors).

When companies have troubles finding resources for a legacy technology, they move away from it.
Organizations with mainframe computers were unable to find technical people who knew how to run their large platforms. The result? They looked for alternatives such as small, distributed systems. IBM was forced to try to switch from their proprietary operating systems to an industry standard such as Linux which had a broader knowledge base. It may be counter-intuitive, but you are better to have fewer skills in emerging technologies than be an expert in legacy ones -- don't get suckered into being the old-time guru who is honored today and discarded tomorrow.

To reduce costs, companies consolidate multiple technologies.
Over time, companies collect a variety of technologies. There comes a time when they have to clean house. For example, large firms purchase a myriad of report-writing technologies -- for example, COBOL, RPG, EzTrieve, FOCUS, Crystal Reports, SAS, and Oracle products. It becomes too expensive to deal with multiple vendors, pay lots of annual maintenance fees, perform multiple upgrades, and support and train employees on so many different technologies. At some point, companies select a winner and eliminate the other similar products. You are better to build expertise in the clear leader within a technology segment.

Companies are moving away from host-based and client/server technologies.
Today's applications must be integrated with web technologies and ubiquitous desktop products such as Microsoft Office. Stand-alone platforms will continue to decline. Even the popular client/server reporting tool Crystal Reports experienced a 30% drop in Monster job postings this year. If your skills are limited to legacy technologies, it is time to renew yourself.

Companies are moving away from paper-based and manually-intensive systems.
Systems that are heavily paper-based and require people are quickly becoming obsolete. Online systems and forms will replace paper. If a person's manual process can be automated, it will be. If your job is repetitive and time-consuming, find a way to automate your tasks and move yourself into a more essential and protected position.

Companies are offloading work to the customer through web-based and mobile applications.
For example, instead of having employees take telephone orders, pizza shops provide customers with an online ordering website and an iPhone app in order to reduce operating costs. Keep an eye on emerging cost-saving technologies and change with the times.

Companies will still build applications if those will save them money.
Application development is not dead.  Companies will gladly fund development initiatives that promise to save them money.  For example, executive dashboards with key performance indicators that visually point out positive and negative trends are popular projects.  If you love creating computer applications, be sure to have current web application development skills.



If you picked B for all 4 questions, you are aware of technology trends and probably keep your job skills fresh and in demand. If you guessed A for any of the options, you should investigate current technology trends and consider making some changes to protect your job (and stop wasting your time with quizzes in Cosmopolitan).

Tuesday, July 28, 2009

Another BI Vendor Acquired

The big software vendors continue to consolidate the BI market.

Today, IBM announced plans to acquire statistical analysis software vendor SPSS for $1.2 billion in U.S. cash. IBM and SPSS had already been partners to integrate predictive analytics features into the Cognos BI product.

We know that consolidations will continue, so let's just get them over with. Here is some advice on how to move things forward faster.

Which of the original companies are still standing in the weeds after the big boys pushed them off the field to play the BI game?

Information Builders and Actuate both offer BI products oriented toward developers to build BI applications -- Information Builders with its proprietary 4GL and Actuate using Java. MicroStrategy, on the other hand, has an end-user adhoc reporting environment that is popular with companies having large data stores, such as those in retail or insurance.

Because Actuate uses Java and Oracle (after acquiring Sun) is aligned with that technology, let's give Actuate to Oracle. Yes, Oracle already has plenty of BI tools, but they will find a good purpose for Actuate, even if it is only used internally to build BI features into Oracle applications.

IBI and IBM. Over the decades, thousands of receptionists confused IBI reps with IBM when they showed up for sales calls. IBM already resells Information Builders' WebFOCUS product under the name "DB2 Web Query" on the Series i platform. Information Builders has strong mainframe technology and a large client base which IBM can easily leverage. Of course, WebFOCUS competes with Cognos, which IBM acquired and ported to the mainframe. Regardless, each product can be positioned into different niches and made to play nicely together.

For MicroStrategy, let's give them to SAP which earlier partnered with the very-large database vendor Teradata. Many of the MicroStrategy clients already store their data in the Teradata product, so this makes sense. Even though SAP already acquired competitive product Business Objects, they can put BO into an adhoc user niche while they position MicroStrategy as an industry-specific application for organizations with VLDBs.

After these mergers, the only vendors not included will be the open-source BI youngsters jumping up and down shouting, "I wanna play! I wanna play!" The rest of us can take our seats and watch the game.

Thursday, July 2, 2009

The Trend for Business Intelligence Software Jobs

In a 2009 June issue, Software Development Times wrote on the scarcity of software developers. Using figures from the U.S. Department of Labor and from Evans Data, SDTimes showed a decline not only in the number of individuals working in application development roles, but in the number of companies producing software products.

Fewer software jobs and fewer people to do work is a worldwide trend. While Evans Data had predicted in 2009 there would be 15.2 million developers globally, they reduced their estimate by 4%. They point a finger at the obvious culprit: the economy.

Is it possible that Business Intelligence software products might go against this bigger trend?

Throughout the 2009 year, I have tracked U.S. jobs for the various Business Intelligence software products using Monster. Today, I see a 10% decline in BI job postings from January through June of this year.

Hardest hit was Crystal Reports, which has 35% fewer job postings than at the beginning of the year, dropping from 985 in January to 639 to now. This decline may reflect a shift from desktop to web tools as well as uncertainty in Crystal's future after being acquired by Business Objects and then by SAP. Changes in licensing costs might also drive customers away.

On the other hand, peer Business Objects jobs saw a 2% growth in spite of a general downward trend. While Crystal Reports is a strong tool for building standard production reports, Business Objects offers a more flexible tool for ad-hoc reporting. Companies may be changing from Crystal Reports to Business Objects as a move toward flexible end-user architectures instead of having developers build user reports (especially if there are fewer software people).

The leader in BI job demand continues to be SAS, although their 1195 postings in January declined 4%. Another top BI product is IBM's Cognos, which shed 14% of its open jobs, going from 823 to 706 postings. Openings for Microsoft Reporting Services jobs went down 3%.

Declining demand for Crystal Reports developers is consistent with that for the legacy COBOL programming language (now 50 years old), which also saw a 35% decline. COBOL jobs went from 348 Monster postings in January to 226 now. C++ jobs also went down -- 3759 to 3049 -- for a 19% drop.

Web-based development positions are not exempt from this downward trend. While Monster did not show the actual number of Java jobs in January -- just that fact there were more than 5000 -- today the postings are at 4948. Openings for Microsoft .NET programmers went from 3885 in January down to 3351 now, a decline of about 14%.

In the Top 5 List of BI openings, the landscape is changing. While SAS remains at the top with the most demand for jobs and Microsoft stays at number 5 with Reporting Services, Crystal Reports dropped from number 2 to number 4 on the list. Business Objects took advantage of its peer's decline by moving up two notches in the list, going around Cognos to take the number 2 spot.

At the middle of 2009, the top 5 BI products in demand are:
  1. SAS (increasing as a percentage of overall BI jobs -- now 25% of the BI jobs that I track; was 23% at the beginning of 2009)
  2. Business Objects (increasing -- 16%; was 14%)
  3. Cognos (steady/declining -- 15%; was 16%)
  4. Crystal Reports (declining -- 14%; was 19%)
  5. Reporting Services (steady/increasing -- 11%; was 10%)
In my work, I have seen demand for BI application development in two main areas. First, companies want to save money by eliminating legacy tools. Because of this, companies are consolidating tools and even changing computer platforms in the process.

Also, companies want to build BI applications such as KPI metric dashboards where cost-savings can be visually identified and acted upon.

Either way, it's the economy driving Business Intelligence software jobs. Companies want to save money by supporting fewer software products and employing fewer people.

Friday, June 12, 2009

Information Builders' Summit 2009

Information Builders' annual users' group for 2009 is over. As a sponsor of the event, I can say that that it was a great week in Nashville with several hundred customers attending. Speakers from a variety of companies did an outstanding job explaining how they are using Business Intelligence software to improve their organizations.

USTRANSCOM told how BI has enabled them to better transport wounded soldiers while reducing the number of beds and other medical needs. Ford Motor Company lowered operating costs by making warranty costs openly available on the web to their automobile dealers. Nationwide Insurance built an urban school portal to address student proficiency -- while Nationwide would not take credit, the city increased graduation rates from less than 50% to 75% and is on its way to a goal of 90% graduation rate.

You can read more about Summit 2009 on my WebFOCUS BI blog.

While there, I presented case studies on automating BI application development and consolidation tasks. For those of you who could not be at my sessions, I made a copy available on the web.

Monday, April 27, 2009

SAP and Teradata Announce Partnership

Today at the Teradata Universe event in Istanbul, SAP and Teradata announced a partnership to have the SAP Business Warehouse run on the Teradata database computer.

Henry Morris, the senior VP for worldwide software and services research at IDC, made this comment:


"CIOs committed to driving greater IT efficiency are seeking to consolidate IT investments across the enterprise. The integration of SAP NetWeaver BW and the Teradata database (optimized for enterprise data warehousing) with SAP
BusinessObjects business intelligence tools supports such a consolidation strategy. This is a significant development that will be welcomed, especially by the many large companies who are joint SAP-Teradata customers. They can now drive operational efficiencies by running their enterprise data warehouse and SAP NetWeaver BW on the same scalable Teradata platform."

After the BI market consolidation, nobody needed a crystal ball to see this coming. In the old order, SAP used databases from IBM, Oracle, and Microsoft. When these four companies all moved into the BI marketspace, they became competitors.

SAP needed an industry-strength database from a non-competitor and Teradata was the logical choice.

Tuesday, April 7, 2009

The Slow Demise of 4GLs

On a regular basis, web searchers come to my site looking for advice on how to eliminate legacy 4GL applications.

In the late-1970s, some innovative software vendors came out with "fourth-generation computer languages" with the goal of eliminating COBOL, the standard "3GL" of the day. Needless to say, these companies missed their marks.

Three decades later, COBOL is still alive and kicking. Funny thing is, so are some of the 4GLs.

At the time, it seemed pretty obvious that data processing professionals would love to reduce the complexity of building software applications. The 4GL vendors offered programmers the same capabilities as COBOL but in a more succinct and easy language. But COBOL coders gave the vendors the cold shoulder.

When you have a product nobody wants to buy, you become creative.

"Okay," these 4GL vendors said, "we understand that you want to stick with COBOL. But how about your business users? Users can't handle power tools like you can. Let's give them something safe and easy to use -- just imagine, they could build their own report programs and leave you alone!"

And so, companies all over the world started giving business users 4GL application development tools for DIY reporting purposes. In the 1980s and 1990s, many organizations researched and selected one of the three major 4GLs: RAMIS, FOCUS, and NOMAD.

Today, firms are trying to figure out how to clean up decades of unmanaged end-user 4GL application development. One of my clients estimated that rewriting the hundreds of 4GL libraries used by their business groups might cost over $4 million USD. Another was shocked to discover somebody had used what management considered to be just a reporting tool to develop a mission-critical shop-floor scheduling application.

In my career, I used 4GLs to write many critical applications that were downright difficult to replace. I am willing to bet that 80% of my American readers have seen consumer product goods coupons generated by one of my 4GL applications. Another easy bet is that almost all of the readers have drunk a brand of coffee whose purchasing, quality testing, blending, and inventory processes were managed by 4GL applications.

There are problems with these 4GL applications. First, they were most likely written by somebody who left the company years ago. The code is handed down from one poor soul to another until it becomes unsupportable. If the IT organization turned a blind eye to end-user 4GL development, no standard app dev methodologies were ever followed. To make matters worse, not a single college today teaches 4GL application development to its students.

But the major problem today is that business users still depend upon 4GL output.

If you are in this situation, I can help by providing software that can inventory and scan your 4GL applications, automatically analyze them for functionality, and even convert them to other BI products. A spreadsheet generated by the scanning process will give us the first step in planning a cost-effective 4GL elimination strategy.

If you are interested in learning how I can help you replace your 4GL, contact me at DLau....

Saturday, March 28, 2009

BI Giant has Clay Feet

Jean-Michel Franco opens his Information Management's March 2009 Special Report by pointing out a problem with today's corporate BI usage:


It took more than 30 years for BI to reach maturity, a little longer than it took for other key components like enterprise resource management and customer relationship management. But whilst the latter have seen the creation and redesign of entire information system landscapes in a big-bang mode, BI investments have taken place in a gradual and often ad hoc manner. In spite of unifying concepts such as the data warehouse, each decision support project often generated its own tools and selection of service providers, architectures, data models and standards. As a result, despite the fact that BI and performance management represent more than 10 percent of the typical IT budget, it can be compared to a giant with clay feet: strong footprint, but sparse foundations.

While uncoordinated, decentralized Business Intelligence implementations and usage may not have been an issue in the past, Franco says that has changed -- BI now provides companies with mission-critical functionality that requires centralized strategy and management.

Franco urges companies to "review and streamline their decision support architecture" and define "a cohesive BI roadmap." As part of the strategy, Franco recommends establishing a corporate team dedicated to Business Intelligence.

Read Franco's whole report at Information Management's website and, if you have not already read it, see my blog on establishing good BI foundations.

Thursday, March 26, 2009

Business Intelligence Foundations

When I stopped at Steve's house, his wife directed me to the backyard where I found him standing knee-deep in a muddy trench he had just dug along his newly-built family room.

Clearly frustrated, Steve climbed out of the pit and gave me the low-down on his outsourced home addition project. He thought the hired builders knew what they were doing and everything seemed fine at the time.

A few weeks later, however, Steve noticed that the windows didn't open or close smoothly. They stuck slightly when you slid them up or down. Strangely, one of the windows cracked. Coming home from a week-long vacation, Steve discovered another window had broken. With the help of a carpenter's level, Steve began to figure out the mystery.

To prove a theory, Steve grabbed his shovel and dug down to the room's concrete footings. He discovered that the main wooden beams for the new room were not even on the concrete foundation; they were off to the side.

Evidently, the concrete person poured the foundation supports in the wrong place. When the next guys on the building project discovered the error, they must not have wanted to wait for a fix. Instead, they just let their wooden beams use the ground for support.

The builders' solution could be considered short-term, at best (just long enough for Steve to pay them).

After a few rains, the weight of the room caused the beams to sink into the dirt. The walls shifted and windows began to break. If allowed to continue, more serious structural damage was sure to happen.

The cost of fixing the problem after the room was built was going to be much higher than had it been addressed properly during construction.

So it is with Business Intelligence applications. If you build on a weak foundation, you will end up paying more later to fix your sure-to-happen future problems.

If you are thinking about implementing BI, what should you establish as the BI Foundation? I see three main components:

  • Business Intelligence Team (People)
  • Business Intelligence Standards (Policies and Procedures)
  • Business Intelligence Infrastructure (Technology and Data)


If you want a successful BI initiative, critical to the success is obviously executive sponsorship for the endeavor. Once you have that, your first step needs to be establishing a team of individuals to support the future BI infrastructure and users.

You could establish your BI team in different fashions. It might be a formally recognized group with its own organizational structure and funding (such as a BI Competency Center or Center of Excellence) or it may just be an informal “community of practice” supporting BI as a spare-time activity. Regardless, you should define these individuals’ responsibilities, accountabilities, and how their activities are funded.

Next, identify potential resources for membership within the BI team. Depending on how robust you would like to make your BI support, you could consider positions such as: BI Director; BI Architect; Subject Matter Experts (data, operational systems); Analysts (Database, Data Quality, BI); and Mentors and Trainers.

Your BI Team's first responsibility will be to establish formal standards for the future BI environment, including a shared corporate language, formal agreements, and service-level agreements (SLAs).

Your organization must formally define a corporate language of business terms, which will then be implemented as a standard within the BI environment. For example, you would define the term “profit,” and identify where it resides within the operational systems (or how it is calculated if it is not a physical element).

With an enforced shared language, all BI reports created within the organization will have terms of the same meaning – for example, no report should stray from corporate standards and calculate an alternate version for “profit.”

Your BI Team also needs to formal reach certain agreements about the BI environment. For example: Communication protocols between IT, Business, and BI Team; Change control procedures; and Responsibilities and Accountabilities; and Security.

Your BI Team should establish service-level agreements on important BI topics. These SLAs establish expectations for IT, Business, and BI Team as to their performance. For example, SLAs may be created for guarantees on: Data life-cycle (latency, length of storage, historical archiving, purging, etc.); Data quality (validity of data, guaranteed synchronization with operational systems, etc.); and Services (response times, quality of service, etc.).

With the BI Team and BI Standards in place, now it is time to consider technology and data -- the BI Infrastructure.

Conceptually, I break out the BI Infrastructure into three layers: 1) the user interaction/presentation component; 2) a centralized BI repository of data and metadata; and 3) an integration mechanism.

Users interact with BI in two main ways: by automatically receiving information (“push”) and by accessing online applications as needed (“pull”).

Automatically, users can receive regularly-scheduled standard reports, exception reports, and alerts sent to destinations such as printers, e-mail, or hand-held devices. Online users have access to interactive BI applications, reports, queries, analytics, and ad-hoc report writing features. This presentation layer will integrate with Microsoft Office for easy use of tools such as Excel spreadsheets.

The next layer is the centralized BI Repository. This is the company's “single version of the truth” used for all BI needs. Your BI Team will design the repository specifically for reporting to mask the complexity of the underlying operational systems, and fully document it for the users.

Depending on your company's BI needs, the BI Repository can consist of multiple data structures, such as a main relational database for general reporting, multi-dimensional cube structures for online analysis (OLAP), and star schemas for ad-hoc reporting needs.

In addition to structured data, a BI Repository might contain unstructured data such as scanned documents, Microsoft Office documents, and other files that are free-form in nature. These types of files typically require special BI capabilities such as content indexing, document management, and search engines.

Also in the BI Repository will be the organization’s definition of abstract terms (hierarchies, Master Data Management, virtual calculations, etc.) and a metadata layer.

Lastly is the BI Integration Layer, which performs the functions of accessing the complex operational data, transforming data into understandable information, and loading the BI Repository. Here, your BI Team will define integration rules such as: Mapping of physical data (Source to Repository); Rules for calculating virtual columns; and Rules for Extract, Transform, and Load operations.

Yes, this sounds like a lot of work for just the foundation.

If you are considering skipping this critical step and jumping right into building BI applications, think about Steve standing in a muddy pit. Then picture yourself in a similar situation in the near future.

If you still decide to move forward without first implementing a solid BI foundation, be sure to have your own level and shovel nearby.

Sunday, March 1, 2009

SAS: Great Revenues in a Bad Economy

Jim Goodnight, the CEO of SAS, is happy. Not only did Gartner rank his company as a leader in the Enterprise BI Platform category, he saw a 5% revenue increase over the previous year and 2600 new customers.

In their February 2009 press release, SAS reported $2.26 billion in 2008 revenues. Goodnight says,

We achieved our 33rd year of revenue growth in the worst economy most can remember. This growth is a direct result of being a stable privately held company, which allows us to invest in long-term relationships with employees and customers.


Unliked some other BI software vendors, SAS created new jobs. At the end of 2008, SAS employed over 11K individuals, an increase of 3.5% from 2007 (240 people joined the SAS ranks as part of the acquisitions of Teragram and IDeaS Revenue Optimization).

You can read the entire press release on SAS's website.

Declining Business Intelligence Jobs in 2009?

For some time now, I have used Monster job searches to gauge demand for Business Intelligence technologies. What I failed to do, however, was keep a good record. Had I done that, I might have been able to spot BI trends. So that became my 2009 New Year's Resolution: do a consistent monthly BI job search and ponder the results.

Comparing Monster U.S. job postings in February 2009 to the previous month, I see a marked decline for BI keywords. For the standard products (e.g., SAS, Cognos, Business Objects, Reporting Services, etc.), my unscientific study shows 22% fewer job postings (5,923 openings in January and 4,594 in February).

SAS has the biggest demand within BI and it appears to be growing, at least as a percentage of the total BI jobs in the United States. In January, SAS jobs accounted for 20% of the total standard BI jobs and increased to almost 26% in February.

The open source keywords (BIRT, Pentaho, Jaspersoft, and QlikTech) experienced a much bigger decline -- 36% -- although the small number of job postings for these technologies probably makes the change meaningless (25 and 16 for January and February, respectively).

I also looked for a job posting trend for programming language keywords such as COBOL, RPG, C++, Java, and .NET. While demand for these also declined, the drop was less than that for BI products -- down only 6% (13,109 postings in January to 12,308 in February). Within this category, demand for Java far exceeded any other programming language.

Now that I have committed this New Year's Resolution to you, I will try to keep it up each month (perhaps I should have blogged about a weight loss/exercise plan!).

Wednesday, February 11, 2009

How to Choose Between Cognos and WebFOCUS?

A Google searcher from New Jersey arrived at my blog with the question, "What are the differences between Cognos and WebFOCUS?"

Perhaps a good way to begin answering his or her question is to look at the vendors' histories.

Cognos (which started as Quasar) and Information Builders (IBI), the software vendor of the WebFOCUS BI product, have similar origins in the 1970s and then, after a decade or two, experienced a major divergence from each other. Both began as providers of next-generation application development tools hoping to replace the common 3GLs, such as COBOL and PL/1.

Cognos' PowerHouse suite offered tools such as QUIZ for report writing, QUICK for screens, and QTP for batch transaction processing. IBI created the FOCUS language which had facilities similar to those found in Powerhouse, but they also included their own proprietary database structure.

Cognos focused their PowerHouse tools on the midrange market (HP3000, VAX, and AS/400) while IBI went after the mainframes and then expanded to almost every platform imaginable (Tandem and Wang included). IBI also invested heavily in building data adapters to access the various types of data structures.

In the 1980s both Cognos and IBI investigated the artificial intelligence (AI expert system) software market which did not materialize. In 1986, Cognos went public; IBI never did.

When Cognos and IBI tried to move their app dev products to the Wintel platform, they struggled. In the 1990s, both companies released app dev product enhancements to support the new Internet.

The major split between Cognos and IBI came late in the 20th century when Cognos started to make a break with its app dev past. Basically shedding its old PowerHouse midrange tools, Cognos focused instead on new end-user Business Intelligence products running on client-server architectures.

In the early 1990s, Cognos introduced Impromptu for report writing and PowerPlay and a multi-dimensional cube structure for OLAP (online analytical processing). Cognos recently has reworked their BI products for the web, releasing Cognos 8 with new browser-based Studio tools (Report Studio, Analysis Studio, Query Studio, and Metric Studio).

In 2000, Business Week magazine named Cognos one of the Top 100 IT companies in the world, standing right in line behind a different powerhouse -- Microsoft. Cognos grew by acquisitions, buying up vendors of complimentary BI products and applications. Their annual revenues skyrocketed to over $1 billion. In January of 2008, IBM acquired Cognos with plans to double that sales figure.

Bringing in about $300 million each year, IBI remains privately-held with the same owners. Even with a small organization, however, IBI provides top-notch technical support and consistently ranks high for customer service.

The WebFOCUS product, based on the FOCUS 4GL, is truly an enterprise BI tool, able to run on almost any platform and access virtually every data structure. In February of 2001, IBI split out the nuts and bolts of their underlying integration layer, offering it as a separate product under the name iWay Software. Reworking the valuable data adapters to be accessible by SQL (IBM's Structured Query Language) instead of the FOCUS 4GL, IBI created another source of revenue outside of their traditional BI customers.

The Cognos 8 Studios are easy-to-use browser-based tools designed for end users and developers of BI solutions. Providing easy front-ends, however, typically means that some hard work must be done on the back-end. A reporting repository of properly structured data along with multi-dimensional cubes is needed before the end users can start playing. Because of this, Cognos is best suited for an environment with relational data repositories running on a centralized server.

WebFOCUS, on the other hand, is more suited for IT technicians to use to build BI applications. The underlying 4GL has powerful tools for complex reporting and analysis. With WebFOCUS, developers can get "under-the-covers" and manipulate the source application logic, something not possible with most BI tools.

[2010 January comment: of course, Information Builders is working hard to provide easy end-user query and analysis features.  See their InfoAssist product.]

As an example, WebFOCUS developers can build row-oriented financial statements, multi-step logical processing, and highly dynamic self-building logic; you would not do this with Cognos. Unlike any other BI product, WebFOCUS can access data scattered across the enterprise on various platforms, pulling things together from different locations to be displayed in the same report output. You still have data locked away in IDMS, IMS, Total, VSAM, Datacom, NonStop SQL, or Model 204? No problem with WebFOCUS.

Those are technical differences. Another consideration is the availability of resources. If you search the Monster job board, you will see close to 900 postings for Cognos people and only about a dozen for WebFOCUS. Because of large differences in customer base and market demand, there are more experienced Cognos developers available.

Which BI product is right for you?

Well, if you have an end-user community needing to build their own reports and queries with minimal outside assistance, and you have an IT database team that could easily create a reporting repository, you should consider Cognos.

On the other hand, if you have a complex environment with several different platforms and databases, and you have an IT development staff that could easily create web applications, you would lean toward WebFOCUS.

Good centralized reporting repository, go with Cognos. Lots of disparate data and some available application developers, go with WebFOCUS.

[An interesting development occurred in January 2010, when IBM began selling WebFOCUS as a web-based BI product for System z DB2 data warehouses.  Why would IBM choose WebFOCUS instead of their own IBM Cognos product which also runs on the z platform?  See more information here.]

Of course, I have given a simplistic answer to a complicated question. If you want to talk in more detail, contact me.

Monday, February 9, 2009

Business Intelligence and Finance

Carl Weinschenk is addressing Business Intelligence topics from the viewpoint of those in Finance. You can get on-demand access to his printed material and podcasts at "The IT-Finance Connection" website or have information delivered to you by registering for his bi-weekly newsletters.

As an example of his IT-Finance topics, see the podcast of Gordon Burnes, the VP of Marketing for OpenPages, explaining why the current financial crisis provides good reason for adding a BI layer to your risk management and compliance systems.

Click here to visit The IT-Finance Connection.

Sunday, February 8, 2009

Why Use Reporting Repositories for Business Intelligence?

I regularly review web statistics with interest to see how Google searches route people to my blog. One person from Indiana came looking for reasons why you should use a data mart. The searcher was probably disappointed since the answer was not really there as advertised. I feel bad, so let me fix that and add some content -- perhaps he or she will return.

Rather than use industry business intelligence terms pregnant with political meaning like Data Warehousing and Data Marts, I am using "Reporting Repository," which I hope is free from bias.

While there are proponents for doing reporting directly against the operational systems' data (see another blog for some information), it is generally speaking just not a good idea. The main argument is that operational data has live, real-time information. But there is a long list of disadvantages.

Instead of using operational data for reporting purposes, you should provide your end users with a centralized, shared copy created especially for BI: a Reporting Repository.

With this term, I mean that the operational and external data feeds have been restructured and stored in a way to make end-user reporting as simple and easy as possible. In addition to changing the data structure, the organization has added an abstract layer (master data management, hierarchies, and so forth) as well as a metadata layer.

A team of technical specialists must bear the burden of understanding the complex data one time while creating the reporting repository and the necessary integration processes. Otherwise, the end users are yoked everyday with the burden of trying to write reports against the complex operational data (which is not meant for reporting).

Here are some reasons you should not use operational data for your end-user reporting:
  • Because of critical nature, operational system must be isolated and protected
  • Data is structured for ease of data maintenance, not reporting
  • Since not designed for end-user reporting, there is rarely documentation for doing that
  • Typically, operational system does not physically store all data (uses business rules inside application code)
  • Typically, operational system does not store historical data (only a snapshot of current situation)
  • Typically, operational system does not have an abstract layer for end-user reporting
  • Typically, operational system does not have a metadata layer for end-user reporting
  • Typically, operational data not accessible by common BI tools

Reporting Repositories provide a corresponding solution to each of those operational data problems:
  • Intentionally designed for access with considerations for security, performance, etc.
  • Data is structured for ease of reporting
  • Intentionally designed and documented for end-user user
  • Either physically stores calculated columns or exposes virtual columns
  • Captures historical information for auditing, comparisons, trending
  • Provides abstract layer to provide context for understanding information
  • Provides metadata layer to provide explanations for accessing and using information
  • Designed to be accessed by BI tools

But what are you going to put in your Reporting Repository? Well, you will figure out the design and content based on the needs of your business decision makers. If you want to build an enterprise repository to serve any and all reporting purposes, you are looking at a Data Warehouse (Bill Inmon approach). If you go with a very specific repository for a unique purpose, you are considering the Data Mart (Ralph Kimball).

Here are the steps for designing your Reporting Repository:
  • Your decision maker needs to take important business action
  • Which requires specific information.
  • Decision maker also needs specific method of interacting with the information (information delivery or interactive, on-demand access)
  • Which determines the proper storage of information in the repository
  • Which determines the requirements for integration with operational systems
  • Which determines the requirements for capturing and storing data within the operational systems


If you came here through Google looking for answers to your BI questions, I hope this helped. If you have any questions, please contact me.

Thursday, January 29, 2009

Gartner's 2009 Q1 Magic Quadrant for BI Platforms

Some people anticipate the Super Bowl, try-outs for American Idol, the Oscars, or the season premiere of Lost. Me? Well, I wait for the publishing of Gartner's Magic Quadrant for BI Platforms. If you are in the market for a Business Intelligence product, Gartner clearly identifies which software vendors you need to consider.

Gartner just released their first report for 2009 and you can get a copy compliments of SAS, who wants you to know that Gartner still considers them the most visionary of the leading BI software vendors.

Of course, Gartner has its own version of a red-velvet rope. Not just anybody can get into the quadrant; a vendor must sell at least $20 million annually of a product that really classifies as BI according to Gartner's list of a dozen checkmark items.

Gartner then separates the BI vendors into four different boxes: Challengers, Leaders, Niche Players, and Visionaries. Inside each grouping, they position vendors according to ability to execute and the completeness of vision.

Back in 2006, Gartner identified three pure-play BI vendors as Leaders: Cognos, Business Objects, and SAS. Information Builders was on the fence, trying to move from being a Challenger to a recognized Leader. In the Challengers' corner were big database and application vendors -- Microsoft, Oracle, and SAP -- just waiting to pounce.

And pounce they did. Acquisitions by the mega-vendors in the last few years now paint a different picture in the 2009 Leaders quadrant. In order of ability to execute, Gartner ranks the 2009Q1 BI Leaders as: IBM, Microsoft, Oracle, SAP, SAS, Information Builders, and MicroStrategy.

IBM popped onto the BI Leader scene after buying Canadian-based Cognos, which had already acquired Niche player Applix. Microsoft moved over with its acquisition of Niche player ProClarity. Oracle gobbled up both Hyperion Solutions and Siebel Systems, which Gartner had considered BI Visionaries. German-based SAP took over the French company Business Objects.

It was like a big game of global domination using BI companies instead of plastic armies!

But Gartner says you should not think the BI market is all about these mega-vendors. Don't forget about the innovative, independent BI platform vendors who are still out there. One example is QlikTech, steadily moving up within the Visionary quadrant and hoping to break across the line into the box where the big boys play.

In later blogs, I will take a closer look at these BI players as well as some who are still waiting on the other side of Gartner's red-velvet rope.

Sunday, January 4, 2009

Need for Speed

A friend posted on Facebook pictures of his motorcycles, adding a comment that his Kawasaki could go 185 MPH -- his commutes to work should not take long. Of course, he never claimed that he ever went that fast; he merely stated the bike's top speed. We seem to want zippier technology, even if it exceeds our real ability to actually utilize the enhancement.

John Lienhard wrote about technological improvements in his book, "The Engines of our Ingenuity," which is based loosely on his Public Radio show of the same name. John writes:

We are told that necessity is the mother of invention, but history does not bear that out. The true mother of invention is a powerful, driving internal need to invent. We invent because we want to invent. The parents of invention are those inexorable internal needs for self-satisfaction and freedom.

Dr. Lienhard includes in his book an interesting time graph showing the continuously improving speeds of vehicles for two hundred years starting in 1800. Within 100 years, inventors had so improved steam cars and locomotives that instead of puttering about on land at under 20 MPH, they were now barreling along at over 100 MPH.

From 1900 to 2000, engineers worked on the internal combustion car and continually picked up speed, increasing from 100 MPH to over 400. Switching to jet engines, engineers got the cars up to almost 800 MPH -- now there's a fun commute!

Vehicles in the air saw a similar trend in speed improvements. In the late 1800s, gliders started out flying very slowly, but inventors added propellers and had planes exceeding land speed records around 1930. Later improvements -- such as jet engines -- enabled planes to go faster than 1000 MPH; fixed-wing rocket crafts went even faster.

Lienhard points out that both the land and air speeds exponentially increased over time; they were not influenced by wars or other major historical events. He puts it this way: "human ingenuity…is remarkably impervious to external pressure."

So there is really no practical reason for a middle-aged man to approach 200 MPH on two wheels; it just has something to do with "self-satisfaction and freedom."

How does this relate to Business Intelligence software, you ask? Well, in the early days of building BI applications, we would putter along at the speed of keystrokes, typing in a text editor. Going 80 WPM between coffee breaks was pretty much a breakneck speed for report developers.

Hoping to increase the efficiency of app dev, software vendors came out with visual environments where the developer could "paint" the results and let the tool generate the code. This improvement was an obvious improvement over manually typing cryptic instructions.

For reasons similar to "self-satisfaction and freedom," I decided to take BI application development to the next level -- strapping a jet rocket on the tool, if you will. Using the metadata of the underlying database as a driver, I built a mass code generator that can spit out, in seconds, hundreds of dynamic web launch pages with accompanying parameterized online queries (two well-known firms take a closer look at the software during this first month of 2009).

But like the Kawasaki, just because the technology goes that fast does not mean that you will. I developed this tool last year as part of a mass BI conversion for another client. I soon realized that humans place restraints on how fast you can really create BI applications, despite the speed of the technology.

Once you generate hundreds of reports, you still have to wait for somebody to review them, test them, approve them, and move them into production -- sort of like a BI speed limit.

About Me

My photo

I am a project-based software consultant, specializing in automating transitions from legacy reporting applications into modern BI/Analytics to leverage Social, Cloud, Mobile, Big Data, Visualizations, and Predictive Analytics using Information Builders' WebFOCUS. Based on scores of successful engagements, I have assembled proven Best Practice methodologies, software tools, and templates.

I have been blessed to work with innovators from firms such as: Ford, FedEx, Procter & Gamble, Nationwide, The Wendy's Company, The Kroger Co., JPMorgan Chase, MasterCard, Bank of America Merrill Lynch, Siemens, American Express, and others.

I was educated at Valparaiso University and the University of Cincinnati, where I graduated summa cum laude. In 1990, I joined Information Builders and for over a dozen years served in regional pre- and post-sales technical leadership roles. Also, for several years I led the US technical services teams within Cincom Systems' ERP software product group and the Midwest custom software services arm of Xerox.

Since 2007, I have provided enterprise BI services such as: strategic advice; architecture, design, and software application development of intelligence systems (interactive dashboards and mobile); data warehousing; and automated modernization of legacy reporting. My experience with BI products include WebFOCUS (vendor certified expert), R, SAP Business Objects (WebI, Crystal Reports), Tableau, and others.