System integration methods and approaches. Why does it matter?

System integration methods and approaches. Why does it matter?

Overview

As a rule, the integration of systems is aimed at improving the efficiency of the company’s business processes where information systems are applied. 

In this article, we will consider basic information system integration approaches which will help us to demonstrate possible ways of solving various business tasks related to the need to organize the interaction of systems.

Basic system integration approaches

• No integration between systems

The company uses three independent information systems: Warehouse system (accounting and analysis of warehouse turnover), CRM system (accounting and analysis of sales and other customer relations) and Accounting system (accounting and financial analysis). There is no information exchange between them.

• 

As a result, sales managers have to print out copies of invoices and take them to the accounting department after they have invoiced customers. In the accounting department, invoices are registered in the accounting system. The accounting department registers the receipt of money on the account. Sales managers have to check with the accounting department on a daily basis whether money from customers is received. That is required as they don`t have the ability to check payments automatically in the CRM system. In this situation, there is large document circulation between managers, accounting, and warehouse units, as well as double registration of actions (once in the warehouse system and the second time in the accounting system).

If we calculate the cost of employee time that should be paid by the company for performing duplicate processes in different systems (highlighted in red in the diagram), we will see that it is a significant part of the total expenses of the firm.

• Vertical integration

This approach presupposes that systems are integrated according to the principle of their functional expertise. For example, in this case, two expertises are highlighted: operational accounting and accounting. In this case, accounting is vertically above operational accounting. In our example, the operational accounting subsystem delivers data to the accounting subsystem.

• 

This allows significantly reducing labor costs for duplicate activities and paperwork, but there are two caveats that you need to bear in mind:

1. Firstly, this system is quite difficult to expand functionally. For example, a company may want to create a subsystem expertise Analytics, which will be vertically located above the expertise Accounting. The examination is largely based on Operational Accounting data. That’s why in addition to the development of the subsystem Analytics, it will be necessary to finalize a subsystem Accounting to ensure that it gets and stores additional information from the Operational accounting.
2. Secondly, there are big opportunities for integration within single functional expertise. 

• “Many-to-many” integration (star, spaghetti)

Under this approach, each of the company’s subsystems can, if necessary, access the functionality of any other subsystem, and each of the subsystems can also be used by any other subsystem. This type of relationship between elements is called “many to many”.

• 

In this case, we have almost unlimited possibilities for integrating subsystems with each other (of course, if from the technical perspective subsystems allow us to do so).

But, on the other hand,  the cost of supporting such an integration scheme grows exponentially as the number of integrated subsystems can increase. For example, if in our case you need to change something in the accounting subsystem (for example, to change its object model), it may lead to the need to modify all other subsystems that use it, because the old object model will stop working. If we have three interacting systems, it may not be so critical, but if we have, for example, thirty – that’s a very different story. 

• Horizontal integration

This approach includes the use of specialized middleware – the so-called enterprise service bus (ESB). The main task of this software is to keep a repository of the functionality of corporate apps connected to it and ensure that these features can be used by other apps that are also connected to this ESB. Interactions between apps can, for example, take the form of exchanging messages or calling published features as Web services. The connection of the system to the ESB is ensured by creating a special adapter for each system. After that, the “published” features of the system become available to other connected systems.

• 

For example, a CRM system when connected to the ESB publishes its features to work with the customer database. The ESB provides the possibility of using accounting and warehousing systems. The CRM system can use the accounting and warehouse data.

The benefit of this approach is the following: systems themselves can be replaced within the existing specification of the published features. No changes to other systems are required. Also, the connection of a new system is quite standardized and simplified. For example, it is possible to connect a new system Analytics, which will get access to all the other systems at once. 

• No need for integration

The best integration is the lack of need for it. For example, all of the above subsystems can be presented as functional modules of a single ERP system of any vendor. In this case, there is no need for integration. As the system is originally unified, it provides much greater connectivity between functional modules than any of the above-mentioned options for integrating different systems.

• 

Objects and methods of system integration

For describing system integration approaches, we considered each information system as an “indivisible” object. However, an information system is a set of several components. That’s why when we are discussing the integration of information systems, it is more correct to talk about the integration of their components. 

• 

Integration of information systems is the integration of one or more components of integrated information systems (integration of objects):

1. Platform integration
2. Data integration
3. App integration
4. Business process integration

• Platform integration

Goals of platform integration:

1. To provide interaction between apps running on different hardware and software platforms (for example between apps running on Windows, Solaris, Linux servers, etc.).
2. To ensure the possibility for apps developed for one hardware and software platform to run on other hardware and software platforms (e.g., Windows apps on Linux platforms, Solaris, etc.).

There are several ways to achieve these goals. Each of them is based on different technologies:

1. Remote procedure call (RPC, Web services, REST, etc.)
2. Middleware (Microsoft.Net, Java Runtime)
3. Virtualization

Remote procedure call technologies (app functionality) allow publishing a procedure and making it callable (transferring input parameters and getting output results) for apps running on other platforms.

The concept of middleware (framework, implementation environment, virtual machine) presupposes developing app software without using services of a specific operating system (e.g., Windows API), but using services of the middleware. The middleware developers create implementations for different operating systems that translate calls of the corresponding framework features into calls to the corresponding operating system. A typical example is the Java Runtime Environment technology. Apps developed for this technology work on any hardware and any software platform (Windows, Linux, etc.) without any modifications introduced to the apps themselves. Similar capabilities are provided by Microsoft .Net Framework.

Virtualization is an interesting and modern concept. It allows significantly simplifying the use of different platforms and, accordingly, the use of systems that require specific platforms for their functioning. If without virtualization it is possible to run N operating environments on N servers at the same time, the use of virtualization technologies allows the functioning of N operating environments on M servers. If N > M – it allows reducing hardware costs by using it more efficiently. If N < M it is an easy way to increase the performance of the systems. For example, virtualization allows several operating systems to be deployed and used at the same time on a single physical server: Windows, Linux, etc. On each of these “virtual” servers, the relevant systems can be deployed and will be available at the same time. Examples of virtualization technologies: Microsoft Hyper-V, KVM, OpenVZ, Virtuozzo, VMware, Xen, etc.

• Data integration

The information system works with the data. Most often the system has a database to store them. Data integration is the sharing of data with different systems. Data integration may be easier than app integration because the industrial DBMS, where information systems typically store data, has advanced software capabilities to access data from other apps. Apps in this case may have a quite limited ability to programmatically (outside their own user interface) allow external systems to use their functionality.

Approaches to data integration:

1. Universal data access
2. Data storage

Universal data access technologies allow uniforming access to data from a different DBMS. In order to work with a particular DBMS in this case, the intermediary acts as a driver for the corresponding DBMS. For example, the same SQL query to select data SELECT * FROM TTABLE can be used to select data from the table TTABLE stored in another DBMS. This allows you to abstract away from the specifics of a particular DBMS and easily integrate data stored in different DBMSs. The most common technologies of this class are ODBC, JDBC, ADO.NET. Object-relational mapping (ORM) technologies are widespread today as well, they also allow abstracting from the details of interaction with specific DBMSs. Examples of such technologies are Entity Framework, Hibernate, NHibernate, Flexberry ORM.

The concept of data warehouses is based on the creation of an enterprise data warehouse. A data warehouse is a database that stores data collected from databases of various information systems with a view to conducting further analysis. For example, a single company data warehouse can be created to collect information from accounting, operating systems and external systems of the company’s partners. Basically, this is done to improve the performance of complex analytical queries in accordance with many parameters (multidimensional queries). The approaches to creating and populating data warehouses are reflected in the ETL paradigm (extraction, transformation, loading). Technologies and tools to analyze large volumes of data in order to identify patterns in the subject area are related to the concept of “Data Mining”, a term for a set of data warehouse technologies and tools that translate transactional business information into a readable form suitable for business Intelligence.

• App integration

App integration is the use of off-the-shelf app features by other apps. For example, developing an electronic document management system, it is possible to use MS Word as a text editor within this system instead of developing your own text editor. Or, for example, the call center software, after receiving an incoming call from a client, has the opportunity to call the billing system feature to check the balance (input – the subscriber’s phone number, output – its current balance). Depending on the state of the balance, it can connect him/her to an operator or automatically inform him/her about the need to top up the account. In this case, the structure of the database billing system is its inside information and published specific features that allow other systems to work with specific data. Only specific functions that allow other systems to work with specific data are published.

Approaches to the app integration:

1. App programming interfaces
2. Messaging (ESB)
3. Service-oriented architecture
4. Integration of user interfaces

The software interface of the system is the “published” functionality of this system, which can be used externally. Functionality can be published as a set of features (for example, Windows API) or as an object model (objects with properties and methods, for example, Microsoft Office app object models). 

In most cases, the integration of multiple systems is the transfer of information between them. For example, in the request-response form. If systems function in heterogeneous distributed environments, it is of fundamental importance to provide a guaranteed, secure, manageable transfer of information between apps. These and other principles are realized in corporate messaging systems. In this case, we are talking about the exchange of messages between apps, not people, as in the case of, for example, e-mail services or messengers. The functionality of these systems is quite transparent: receiving a message from one application, transporting it according to given rules, and transmitting this message to another application. By doing so, the following possibilities can be ensured:

1. Message encryption (to make it impossible to read the data in transit)
2. Digital signature (to protect against intentional changes to the data during the message path)
3. Subscription setup (to send a single message to multiple applications at once)
4. Definition of metadata for messages (for easy use of messages with complex content structure), etc.

Service Oriented Architecture (SOA) is a logical extension of the Web Services concept. It consists of publishing the feature blocks of an application in a way that allows them to be accessed by another application over the Web. Web (protocol HTTP) in this case is very appropriate because of the possibility to use it and, accordingly, to use the published web applications functionality on any hardware and software platforms. A web service is a small software add-on to an app’s functionality. It converts calls received through the web into internal app feature calls and returns the results back.

The basic ideas of SOA are:

1. To publish corporate app functionality in the form of Web services.
2. To organize published services as a catalog.
3. To build new apps based on Web Services by combining them.

The cost of creating new apps based on existing Web Services is significantly lower than of developing apps from scratch or of integrating them with other systems. 

Let’s have a look at an example. A company (a telecom operator) has a Service Desk system (technical support for subscribers) and a billing system (tarification services). The company is facing the challenge of building a new system called “Subscriber’s Personal Account,” in which the subscriber could view the status of his/her account via the Internet and report a malfunction. Instead of creating a “Personal account” with its own database synchronized with the database of the billing system and the Service Desk system, the company uses ready-made web services “subscriber card” (published functionality of the billing system) and “Create a request for technical support” (published functionality of Service Desk). Сlearly, all the work on the new “My Account” application includes creating a Web user interface on the company’s website.

The approach of integrating user interfaces is used often as well. For example, to create one-stop applications. The simplest example is frames on a web page. There is a separate web application within each frame. Frames allow all of these applications to be displayed simultaneously on the screen. Web application user interfaces are easy to integrate. However, there are opportunities to integrate “classic” user interfaces and their fragments (ActiveX).

• Business process integration

The most holistic approach to system integration is business process integration. Within the business process integration, there is application integration, data integration, and people involved in the business process. Integration of business processes is the most “natural” for organizations because their work consists primarily of business processes, not applications, databases, or platforms.

The ideas behind business process integration are quite simple:

1. It is necessary to make a scenario of the organization’s business process. Inside, to describe the operations of user interaction with various systems and systems among themselves. A business process is an element that logically integrates various systems. A scenario is created using specialized software, which will then manage the course of the business process according to the set scenario itself.
2. Operations of interaction with systems within a business process are described in detail in terms of information exchange: exchange formats, services used, applications, events, rules, policies, etc. 
3. It is required to integrate software, by means of which the business process scenario is described. The integrated systems involved in the business process by means of adapters are connected. This makes an automated information exchange between systems possible.
4. Readiness for implementation of a business process is displayed on the manager’s “control panel”, through which he/she can start and stop business processes, monitor their status, enter data and make decisions on individual operations of business processes that require his/her participation, etc. Interactions between systems that do not require the participation of an employee are carried out automatically by the integrating software.