In this article, we would like to review the differences between BI Tools and SAP, what each does best, and how these two can efficiently cooperate.
Abstract
Put it simply, Business intelligence (BI) is about delivering relevant and reliable information to the right people at the right time to achieve better decisions faster.
When asked, people working at organizations running SAP often retort that their Business Intelligence (BI) is the organization’s sole portal for reports. They usually add that all SAP-related reports were already developed at the BI side, and state that all new requests will be handled that very same way. Given the advantages of BI tools, these statements could hardly come as a surprise.
However, for some reason instead of leveraging BI tools abilities, BI developers tend to extend the tool out of its natural reach.
In this article, we would like to review the differences between BI Tools and SAP, what each does best, and how these two can efficiently cooperate.
Standard vs. locally developed Z-Reports.
An SAP vanilla system is shipped to customers with a set of reports that help end-users to query the application. Standard SAP reports are ABAP-based executable programs that read data from the database, manipulate it, and generate an output based on the filter criteria selected by the end-user. Alas, often those SAP standard reports are falling short to fulfill all customers’ needs. Using the ABAP integrated development environment (IDE), customers will employ developers to code their own reports. Those Z-reports, named after the customer namespace (letters Z & Y) assigned by SAP, will be used by the organization’s end-users instead of the SAP standard ones.
What is BI
BI is all about harnessing the data that the business generates in all of its different activities. Then, using BI, we analyze and visualize this data to clearly understand it and gain valuable insights into how the business is performing. These insights, in turn, allow one to make better informed and smarter business decisions to help the business evolve and grow.
Let’s check closely the stages BI tools are composed of.
Data – In today’s modern business world, companies generate a lot of data. All of this data is separate in silos making it hard to easily get a global 360 view of how the business is performing in all of its activities without having to look at multiple reports in different places.
BI brings these disparate data sources together and analyzes them together to get a clearer picture.
Analysis & Data visualization – The end product of the analysis usually takes the form of reports or dashboards containing visual representations of your aggregated data. BI dashboard contains your business’ metrics or KPI’s like your revenues, stock levels, or how much engagement you’re getting on social media. When you track and benchmark your KPIs you can measure your business’s performance.
Visualizing your data makes it easy to read, understand, and digest. A dashboard gives you an at-a-glance snapshot of performance and helps you identify areas where you’re doing well and those that might need some more attention or investigation.
Data visualization is a discipline all unto itself and effective visualization is the key to making sure that your data is as easy to understand as possible.
Insight – Finally, once we’ve got all the data together, analyzed, and visualized it, it’s
time for the insight stage where we read the results of the analysis and see what
we can learn. Your dashboard can help you spot trends, see how metrics correlate
with each other, and also compare several periods to see how things are evolving.
Quickly enough, BI tools adopted the Extract ability used by ETL tools to be able to combine data from different source systems. Each separate system may also use a different data organization and/or format. BI tools are streaming of the extracted data source and loading it on-the-fly when no intermediate data storage is required.
BI Overview
Origins – Almost all big corporations are using relational databases to store their data. But with the adoption of relational databases came a problem. Companies found out that relational databases were great at storing data, but made it difficult to generate management reports from transactional data. On the hardware, such reports would often slow down the entire system to the point where it became unusable for critical business operations.
The solution was Online Analytical Processing – OLAP for short. The idea behind OLAP was to pre-compute all of the totals and subtotals needed for reporting, allowing instant access. The totals are stored in a special database, called an OLAP Cube (multidimensional dataset). That is to say, an OLAP Cube is a snapshot of data at a specific point in time defining a business problem. It is worth mentioning that although OLAP is an important phase int the BI evolution, the seeds of BI were planted before the 1990s around which the OLAP concept was invented.
Technological bubble – With measures (simply, the thing that’s being totaled) which are categorized by dimensions (e.g., product, time, city, category, etc,) and aggregated in a hierarchy (the elements of a dimension can be organized as a set of parent-child relationships), the OLAP cube is a data structure optimized for very quick data analysis.
Note, that a cube is not a “cube” in the strict mathematical sense. But this term is used widely.
With the notation of the cube in mind, slice is a term for a dimension that is held constant (one plane of the cube holding specific value for one of its dimensions) so that multidimensional information can be shown in an n-1 dimensional ‘cube’. Following the cube geometry, dicing is when two or more dimensions are selected resulting in the creation of a sub-cube.
Slicing and dicing the OLAP cube, business users became able to manipulate large data sets, and apply summaries such as SUM and COUNT across custom groupings in real-time. A better term is ‘ad-hoc analytics’ because slice and dice were originally used purely for tabular data.
Market review – In the recent past, it was very expensive to implement a Business Intelligence solution. BI projects would often take many months to complete, and involve large numbers of highly-trained IT professionals, to design and extract data into the OLAP Cubes. With time both prices of BI suites and implementation time dropped dramatically.
According to Grand Review Research, the overall Business intelligence market is expected to grow from USD 17.09 Billion in 2016 to USD 26.88 Billion by 2021. Increasing adoption of cloud, growth of advanced analytics, adoption of data-driven decision-making, and the emergence of the Internet of Things (IoT)-enabled technologies are the key factors driving the growth of this market.
No wonder, then, that new BI tools often emerge promising something new that revolutionizes the BI domain. Yet, shortly after, old and new BI tools are adding that feature/ability/technology into their toolkit, leaving the BI market in equilibrium.
In terms of usage, Excel [by Microsoft] is by far the most commonly used “BI tool” out there. Nonetheless, Excel is the least favorable tool for the job by the organizations’ IT team. Having said that, Power BI [by Microsoft] is the BI tool of choice by most organizations.
BI tools at SAP sites
Surprisingly, many SAP sites use more than a single BI tool. Moreover, it is not rare to find an organization running SAP with three different BI tools, side by side. Business units pick their BI tool of choice, ending up with Different BI tools being used by different business units. It looks as if this phenomenon is a mixture of reasons.
- One reason is seeded back in the days when SAP BW and its SAP Business Explorer (BEx) reigned that category. Some organizations running SAP still actively use SAP BW and its leftovers could be found in others. To stay relevant, SAP keeps pushing its (own) BI tools (e.g., BusinessObjects, SAP Analytics Cloud) – generating additional BI islands within the organization.
- Also, the relatively small prices of BI tools bring BI companies salespersons to directly address department’s managers, offering them tailor-made BI solutions for their department.
- Finally, the fact that companies running SAP are usually huge enterprises, geographically dispersed, makes it easier for them to accommodate more than one BI solution. Thus, one may come across SAP sites that use SAP BW (with both BEx and BO) together with Qlikview [by Qlik] and Tableau [by Tableau Software].
Yet, another surprising piece of information is having those various BI teams work very closely with (and under the supervision of) their business units. IT is often left out of the ‘BI party’ leaving business units to manage the BI teams directly. This phenomenon, as well, is rooted in a mixture of reasons.
- One can relate this situation to the fact that IT often does not take part in any phase of the BI tool adoption process. Department managers pick the software and pay it from their budget. They often buy professional services from the BI company for the onboarding phase (and sometimes beyond). Since BI reports are the front window (‘face’) of the department thus, managers like to be able to influence it at will.
- Technology-wise, new BI tools use light coding or even no coding at all. This way, using drag & drop and (usually proprietary) scripting technology, BI developers achieve their goals. This technology is often new to the organization (IT has not too much to offer) and is not shared between other frameworks in the organization – eventually discouraging both ends (IT and business) from too much investment/involvement.
- In many organizations for too many years, bad blood is running between Business and IT. Even though these are two sides of the same coin, inherently they hold opposite standpoints. While the Business needs as much functionality as possible in the lowest possible budget, IT aspires for control blend with comfort and job security. It is not that there are no organizations where these two entities leave in harmony and synergy. However evidently, the big picture has changed dramatically during the last years, where budgets for IT projects moved from IT to Business. Nowadays in many organizations, ‘The Business’ decides what their money will buy (IT-wise). Picking BI tools on their own is merely another step in that direction.
State of SAP-BI Coupling
In theory, the coupling of SAP alongside BI tools looks like a match made in heaven. These two machines marvelously complement each other. SAP with relational DB is optimized for operation while BI with multidimensional datasets is optimized for analysis. Business end users handle transaction and business processes from SAP while upper management gains insights from sliced and diced data beautifully displayed at the BI end. OLTP and OLAP working together allow organizations to eat the cake and have it too, so to speak.
“In theory, there is no difference between theory and practice. But, in practice, there is.” [Benjamin Brewster]
In practice, working with many many SAP projects that use BI tools, this coupling is far from being heaven. The following are the most common problems manifested with this coupling.
- BI projects at SAP sites are often still very expensive and have a very long time-to-market.
- Discrepancies are often found between the BI reports data and their equivalent standard-SAP reports. For example, a sales report at the SAP side showed different values than its counterpart at BI. Back in the days, BI developers ascribed these discrepancies to the fact that extractors (data replication processes from SAP to BI) took place once/twice a day. Yet, today with near real-time data replication tools, this excuse could not be used any longer.
- IT and Business often argue about what should be developed and at what end. When information means power, each side wants to have a bigger chunk of that power.
All in all, Instead of taking advantage of the strengths of these two tools, decisions are made according to who will have the power and influence over the result – Ending up with expensive and lengthy BI projects with inaccurate data.
Not surprisingly, the above problems are tangled together in a ‘cause & effect’ relationship. Moreover, the problems that lead to the current situation reside on both sides – SAP and BI alike.
BI behavior
BI developers use raw data (information taken from SAP DB tables & views) rather than application-level information (reports’ results). Using the somehow limited coding abilities offered by BI tools, they try to reverse-engineer standard SAP reports. People think it’s a simple task – Nothing could be further from the truth.
When SAP ships a standard report SAP is held liable for that report in relation to the accuracy, scope, performance, and authorization. That very same report will serve all SAP customers that are using it in different business scenarios, in different languages, with different currencies and units of measures. Technically speaking, based on user input, SAP reports gather data from the DB server and in thousands of thousands of lines of code, manipulate it in the application server (business logic) so it will be presented accordingly at the presentation server (UI). Needless to say that SAP invests heavily to make sure that each report it ships, will hold the standard. Yet, even if SAP failed to meet all these requirements, the enormous number of users’ invocations will force SAP to perfect that report with time. Years filled with heavy use by customers, cycles of OSS incidents and notes, yield an, almost Darwinian, mechanism of perfection.
Does it make any sense that BI developers plan to reverse-engineer – in days – something that took the great SAP years to accomplish?
In practice and without a doubt, the part of reengineering and enhancing standard-SAP reports at BI end takes most of the time and yields the majority of problems. Needless to say, that in case various Business departments use different BI tools, each will spend the time to reverse-engineer something that was already done by SAP.
Why is the wheel reinvented time and time again?
SAP deficiencies
The profound question would be: Why do not BI developers use application-level (reports’ results) data to start with? The answer is a mix of reasons.
- Firstly, there is no SAP standard mechanism for a periodic extraction of applications-level information. Instead, to extract reports’ data out of SAP, customers use the following workaround. They schedule a print job of a report. Then, a utility program that is periodically invoked and looks for the file within the print background Job’s Spool. In case such a spool is found, the utility converts it and saves or emails it. Creative as it is, rarely this hack is used to feed BI tools. Furthermore, trying to use more than a single selection-variant of the original report to feed another BI tool, forces doubling the trouble.
- Second, the above workaround does not support enhancing the underlying SAP-standard report. For example, adding content from DB tables (Z tables included) or from another report. This means the destination BI tool must do that extra mile on its own, nullifying the advantages or reuse.
- Third, in case BI tools need to access SAP application-level information, ABAP developers are often asked to interfere. Using ABAP coding language, the latter often replicate the code behind the needed SAP-standard report and enhance it as needed. The results are then saved into DB tables from which BI connectors could periodically gather it. The overhead in coding the report again, enhancing it, and saving the results in DB makes this option even worse than coding the whole report on the BI side.
All in all, there is currently no way to reuse standard-SAP reports by enhancing them to fit BI tools requirements and to easily share them.
However, the world of SAP is always more glamorous down the “Roadmap”. Let’s investigate if and how SAP will challenge this fundamental deficiency of being unable to enhance standard SAP reports and easily share them periodically with BI tools.
People may argue that with the newly introduced CDS (Core Data Services), the whole issue of application-level data becomes irrelevant.
CDS is an extension of the ABAP Dictionary that allows one to define semantically rich data models in the database and to use these data models in ABAP programs. CDS is a central part of enabling code push-down in ABAP applications minimizing the application server part. Thus, selecting data from the DB using CDSs IS the report itself – saving everybody, BI developers included, from rewriting the business logic.
- CDS is not a report. Funny as it sounds, when CDS will evolve to handle all of the application server tasks, it would be an OLAP cube as well. Superfast DB with abilities to display logic quickly enough, as if it was data, means duality – relational DB and OLAP cube simultaneously. In that case, stand-alone BI tools will become a burden.
- Only a small fraction of SAP legacy was already manifested as CDSs. And still, standard SAP reports further manipulate the CDS data at the application server level.
However, at the customers’ side, the situation is different. While SAP is heavily investing in moving legacy business logic to CDSs, customers cannot afford to rewrite their entire legacy. In addition, with logic stored at the CDS level, code should now be regarded as infrastructure. In that case, before having any report in place, the relevant CDS must be constructed to perfection. Yet, companies running SAP are not software companies in the ERP domain – They have neither the skills nor the resources to do so and cannot afford such a process.
SAP Analytics Cloud is another approach SAP is taking down its Roadmap. Yet again, when
Organizations running SAP are left out with BI tools that want to redevelop the legacy SAP business logic at their end (disregarding the complexity of such a task) on one side. And from the other side, with SAP deficiencies to enable enhancement of the legacy SAP business logic and sharing the outcome with relevant BI tools.
These two great powers are acting together to prevent the SAP-BI match from entering heaven.
Whether the SAP-offered solution resides in CDSs, SAP Analytics Cloud, or a newly invented SAPUI5-based solution, for SAP customers the question is always summed up in: Where and When. SAP customers need those solutions Here and Now – not down the Roadmap or after migrating to a never-ending-version of external-to-ERP solution.
Suggested alternative
The majority of ABAP-based reports use SAP ABAP List Viewer (ALV) as their foundation. The ALV is a framework that enables displaying tabular data in a user-friendly way. The ALV is the most common way by which SAP and its customers display user reports.
Our approach is based on InsightSAP Suite – an SAP-certified product, natively coded in SAP using ABAP. InsightSAP is a platform that allows business analysts, consultants, and business end-users to create, enhance, and publish ALV Assets – All done directly in Production, within SAP and w/o coding.
One may need some elaboration regarding the term “ALV Asset” herein (widely) used. ALV asset could be SAP report, locally developed (Z) report, ABAP query, DB table, or view. Even those assets that were not exposed as ALV, could easily take a turnabout to become ones. This way SAP application-logs data (T-code SLG1), job data (T-code SM37), and others could be presented as an ALV-report making them accessible for the InsightSAP platform.
Description Of Operations – InsightSAP executes SAP ABAP list viewer (ALV) reports suppressing its UI interaction (“dark” mode), wherein results of data retrieval and associated metadata (“data model”) are stored in computer memory. Computer method and software further embodies data processing tools for processing and enhancing said data model by forming a new – persisting – data model. All processing and enhancing instructions are defined without programming efforts ready to run in a live environment. Namely, using InsightSAP any ALV asset could be enhanced in terms of content (new data columns) and presentation (coloring, grouping, and graphs to list a few).
The benefits are enormous in terms of accuracy and efficiency. One must wrap her mind around the fact that by using an existing ALV asset as its baseline, 90% of the coding is saved. Yet, the cost of coding is only the tip of the saving iceberg:
- The resulted Insight is as matured and error-free as the original report/query
- If an error was found in the baseline, this is the problem of the original code creator (in case of SAP-standard reports/queries, this is SAP responsibility)
- In case the baseline was modified (bug fixing or modifications), the outcome will instantaneously propagate to the Insight model
In case the Insight Maker uses SAP Standard report as a baseline, she – actually – Starts where SAP ends.
New content – The new columns added to the baseline could be of several types (Formula, DB Lookup, Input, and aggregation column kind to list a few). Each of the column kinds corresponds to a certain typical task Insight makers are required to. This way, to add a column with the amount with VAT, a Formula Column will be used. By the same token, to enrich the report with a field from a Z DB table, the DB Lookup column type will be used. Column by column, makers enhance the baseline with additional needed content. All is done w/o a line of code and directly on a live system.
Presentation – On top of the content (baseline + new content), a new presentation layer could be added. This way a Painting Rule Engine could be employed to set a set of colors at any level (cell, column, or row). The look and feel of the result could be now enriched by adding features like grouping and graphs.
Sharing – Insight could be shared in multiple ways using both push and pull techniques. Using SAP jobs, the content could be periodically published in several formats and via several media.
Snapshot – Every said Insight could be saved by internally Publishing it as a Snapshot. This way, business end-users may review that very report in key dates in the past. Just like in BI, the snapshot could be used to buffer reports’ content, allowing business users to instantaneously view the time-consuming reports. Last but not least, the snapshot could be used to enrich other Insights. Namely, this Vlookup-like functionality enables a join at the application level by which reports results could be interlinked.
Thus, InsightSAP serves as a unification environment for almost any ALV asset. An environment in which Makers could enhance and enrich an existing asset to fit the Business end-users’ needs. Business end-users invoke the outcome InsightSAP report (which is yet a new ALV asset) like any other ALV report, a report that behaves just like any other ALV report. Nonetheless, as soon as the Rainow report is in-place it could be viewed as a Fiori-like application (called from the Fiori launchpad) or shared by others in various formats (Excel, text, HTML, PDF, XML, and others).
All done w/o code, in hours, directly in a live-system (production).
Operational reports, Interfaces (incoming and outgoing), and Business Process Automation all could be easily achieved by InsightSAP. The InsightSAP platform is a tool of choice for the long-tail of processes every manager yearns to automate within the SAP environment but could not afford it.
SAP-BI Coupling using InsightSAP
As the point that InsightSAP could enhance any existing ALV asset was well established, sharing the result with BI-tools is the only gap.
Coupling SAP with BI tools using InsightSAP could use two approaches
- Push – Using the publishing abilities of InsightSAP, the result could be saved in a place of choice (file directory, FTP, and others) in one of the various formats (e.g., XLSX, TXT, XML, etc.). Periodically, the BI-tool using its extracting abilities could easily pick up the content file.
- Pull – InsightSAP is publishing a single REST service (in JSON) of any Insight report for others to consume. This very same service is used by the Fiori-like application to render any Insight report using SAPUI5. BI tools could easily consume the REST service to get the needed information at will.
Summary
With SAP from one side and BI tool from the other, organizations running SAP should immensely benefit from this coupling. Yet, in practice, this is not the case – not at all.
Instead of taking advantage of the strengths of these two tools, decisions are made according to who will have the power and influence over the result – Ending up with expensive and lengthy BI projects with inaccurate data. SAP deficiencies in terms of enhancing existing reports and sharing them with BI tools, make it easier for management to move the responsibility over to the BI side.
InsightSAP suit was suggested as an alternative that moves the responsibility for application-level (operational) data back to SAP allowing each side, SAP & BI, to concentrate on what they are best at.