How Oracle reduced Sun's IT messaging costs 6x using Fusion ...

Reducing IT Messaging Costs: How Oracle reduced Sun’s IT messaging costs 6x using Fusion Middleware

An Oracle Technical White Paper February 2013


Contents Abstract ..................................................................................................... 3 The Importance of IT Messaging............................................................... 3 The Need to Reduce Messaging Costs at Oracle ..................................... 3 The Approach to Simplifying Messaging ................................................... 5 Results from Oracle’s Messaging Implementation .................................. 12 Lessons Learned ..................................................................................... 15 Conclusion............................................................................................... 15

2


Abstract Before Sun Microsystems was acquired by Oracle, IT Messaging costs accounted for 12% of Sun’s IT budget. Oracle’s were 2%. With Sun came 1500+ information highway interfaces and 1000+ applications whose functionality needed to be integrated into Oracle. These applications included several regional messaging solutions from multiple vendors and were the cause of support headaches, communication issues and difficulties adopting new solutions due to integration complications. The integration project was initiated in Jan 2010 and completed in 11 months. Today, Oracle runs one Fusion Middleware integration solution for all Application to Application (A2A) and Business to Business (B2B) messaging for customer support, handling of partner orders and fulfillment. This solution has helped bring the post-acquisition cost of messaging back down to 2% of the overall IT budget. Further growth within the same cost parameters is also straightforward since there is now a common set of components for Error Handling, Message Routing, Reporting, and Metrics. This white paper describes how Oracle integrated Sun’s IT Messaging solutions and simplified the combined solution using Fusion Middleware integration. Best practice recommendations that customers can use to bring down costs when implementing and integrating IT messaging solutions are also included.

The Importance of IT Messaging IT Messaging refers to the transport, translation and error handling of data between applications through A2A and B2B integrations. A2A integrations transport data between intra-enterprise applications, such as synchronizing Service Requests (SRs) between a company’s Enterprise Resource Planning (ERP) application and Support application. B2B integrations extend data from one company’s applications to external customers, partners and suppliers. An example of B2B messaging is the communication of invoicing data between a company’s ERP application and a supplier’s. Both forms of messaging are critical for automating manual processes and increasing efficiencies. For example, Oracle estimates savings of $250 for every Purchase Order (PO) message that comes in as XML and instead of having to be manually entered. Oracle also saves $200 on every invoice that is sent to customers electronically, rather than processed and mailed manually. In addition, automated IT messaging allows POs to be processed in minutes rather than hours. These examples highlight the benefits of IT messaging. However the Sun acquisition also required Oracle IT to look again at the cost side of the IT messaging equation.

The Need to Reduce Messaging Costs at Oracle Oracle’s acquisition of Sun in January 2010 came with the immense challenge of consolidating the IT infrastructures of two large public companies. Sun brought a hardware business that Oracle had never been in before. As if this was not a sufficient challenge, that business was run on 1000+ legacy applications whose functionality needed to be integrated into Oracle, connected by 1500+ interfaces using a number of messaging solutions from multiple vendors. Before the acquisition, both Oracle and Sun used different messaging solutions for different Lines of Business (LoBs). Oracle used a limited implementation of Oracle Application Server (OAS) 10.1.2 for IT Messaging to and from E-Business Suite (EBS). But this setup was on

3


limited hardware, did not offer high availability, and did not support messaging for Supply Chain and Customer Support. Sun on the other hand used advanced B2B messaging for Customer Support, Supply Chain and Partner Ordering. However the large number of third party messaging components in use at Sun led to support and maintenance headaches as well as high costs amounting to 12% of Sun’s total IT budget. Table 1 shows the differences between the Sun and Oracle Messaging solutions in place before Sun was acquired by Oracle.

Area Messaging Architecture

Oracle (before acquiring Sun)

Sun (before Oracle acquisition)

Limited messaging implementation that

Complex implementation with 1500+

did not offer high-availability and was not

interfaces and multiple partner entry points.

deployed on a SOA architecture Transmission Formats

OAG XML was used for several

Over ten different XML and EDI flavors

Messaging implementations at Oracle, but

were used for messaging and several third

did not play well for messaging to/from

party tools were in place to perform message

non-Oracle applications due to limited

translations.

external adoption B2B Messaging Areas

Quote to Fulfillment – Partner Orders, Product/Price Update Finance – Customer & Supplier Invoicing

Field Service: Field Dispatch for service Quote to Fulfillment: Partner Orders, Product/Price Update Supply Chain: 3 Logistics Provider Integrations, External Manufacturer Messaging Finance: Customer & Supplier Invoicing

A2A Messaging

Gateways

No standards were in place for A2A

A separate messaging hub was in place for

solution and A2A bridges used

A2A Messaging and for Integration with E-

heterogeneous standards.

Business Suite

Integration to ERP was dependent on a

Three different gateways were in use for

XML Gateway which required a dual setup

XML and EDI transmissions.

of Trading Partners Table 1 – Snapshot of the Sun and Oracle Messaging solutions before Oracle’s Sun acquisition

Figure 1 provides details of Sun’s messaging architecture before its acquisition by Oracle. Sun provided multiple entry points for messages from trading partners and had three translation tools at different points in its architecture. The B2B-XTS interface used JCAPS for translating B2B XML messages, the EDI interface used GenTrans for translating B2B EDI messages and the Electronic Information Highway (EIH) interface translated A2A messages to XML. These translation tools required different expertise to maintain and extend, so each tool had a dedicated development team, resulting in high support costs. The 1500+ interfaces and numerous third party tools in use at Sun also added to support and maintenance costs, since even minor backend system changes had to be tested against all the components that they could potentially impact.

4


Figure 1 –Sun’s Messaging Architecture before the Oracle acquisition

Both cost and scalability requirements dictated that messaging be standardized across both Sun and Oracle’s existing business. In addition, the Oracle IT team took this integration as an opportunity to combine both B2B and A2A Messaging infrastructures into a single architecture. In addition, business processes had to be reworked and new partners on boarded and integrated. Oracle’s mandate was to complete the overall Sun consolidation process in one year. This required an aggressive implementation timeline for a unified messaging solution.

The Approach to Simplifying Messaging Selecting the Messaging Platform

The Integration team chose Fusion Middleware 11g (11.1.1.2) as the messaging solution platform. Two factors drove this selection. First, Fusion Middleware 11g enabled the team to build common sets of reusable code on SOA core composites for both A2A and B2B communications. Second, Fusion Middleware’s integration mechanisms were sufficiently flexible and scalable to provide a framework for integrating applications from future acquisitions into the messaging solution. Both requirements were critical to enable Oracle to maintain the needed functionality and keep messaging costs down in the face of an ongoing stream of additional acquisitions.

5


Re-architecting the Messaging Solution

The first task for the Integration team was to replace the 1500+ Sun interfaces with a smaller, more manageable set on one Fusion Middleware 11g Patch Set 2 Integration. This was not a purely technical task. Instead it involved replacing legacy Sun business processes with simplified ones, and then building a technical solution to fit these process needs. Sun’s legacy architecture was designed around the message formats used, rather than the functional purpose of the messages being transmitted. This focus had led to multiple hubs, each geared towards handling a specific transmission format. To correct this ad hoc structure, the primary source and target applications that were to be part of the final messaging solution were identified and interfaces built based on functional purpose. For example, a Supplier PO B2B interface was put in place to deliver POs from a manufacturer’s Supply Chain Management (SCM) application to Oracle’s Global Single Instance (GSI) of EBusiness Suite (EBS). The next step was to migrate Sun’s old EBS instances on R10.7 and R11 to Oracle’s R12.1 ERP instance so that there would be a single instance of ERP that interfaces could be built against. This migration was actually part of the larger Sun integration, but was a required step in the development of the messaging solution. Finally, the Sun Messaging Architecture and all associated third party tools were taken offline and replaced with the new A2A and B2B bridges. Table 2 below summarizes the changes.

Turned Off

Turned On

1500+ Sun Information Highway interfaces

One Fusion Middleware 11g PS2 Integration

Sun 10.7 ERP Instance Order, Fulfilment, R11 ERP

Migrated Sun to Oracle’s R12.1 ERP instance

support instance Sun Messaging Architecture, Third party messaging tools

18 new A2A 15 new B2B bridges as of Jan 2011(end of the consolidation project) 25 OAG and EDI messages

Table 2 – Before and After

The following sub-sections describe the design and implementation steps taken to reach this end state.

Designing for Scalability and Availability

All messaging traffic is not equally important. For example, a PO is far more important to a business than a Business Applications Management (BAM) reporting message. Therefore Oracle’s new messaging solution achieved scalability and availability in part by physically and logically separating high and low priority transactions. Java Virtual Machines (JVMs), Nodes and Domains were added so that low priority transactions could be kept separate from high priority transactions. The number of JVMs was increased from 2 to 20, Nodes from 2 to 4, and Domains from 1 to 4. Details on how transactions were distributed among Domains are shown in Figure 2.

6


Figure 2 – The Final Middleware Architecture (Jan 2011)

The separation of transactions ensured that lower priority ones such as Business Applications Management (BAM) reporting messages did not interfere with crucial ones, such as those for POs. In addition, this allowed all patching to be done hot. A node could be patched without impacting the other nodes. JVMs could be bounced in a rolling fashion so that at least one JVM would be running at all times ensuring that Domains rarely went down. To further protect availability, a separate disaster recovery instance was also put in place at the Oracle data centre in Utah. This allowed Oracle to switch over to the instance within 30 minutes without any transaction losses. The implementation of the new architecture was completed along with the EBS upgrade from 12.0 to R12.1 on the live production environment six days after the quarter close and during the quarter end freeze. This was a patching freeze period during which no downtime for cold patching was allowed. All patches were applied hot with zero downtime.

Building Bridges for Communications

To replace the functionality of the old Sun Messaging Architecture and its Third Party tools, the team built three groups of B2B bridges totaling 15 interfaces and five groups of new A2A bridges totaling 20 interfaces.. The B2B groups handle messaging between GSI and systems for Field Support (FS), Supply Chain Management (SCM) and Order Management (OM). The FS-GSI interfaces synchronize Partner Service Requests between the two systems. The SCM-GSI interfaces support messages for hardware manufacturing, delivery and invoicing. The OM-GSI interfaces support partner ordering and provide customers and resellers with updated Price data for Products. Details about the bridges are provided in Table 3.

7


Bridge

Interface Name

Source

Target

Group FS-GSI

ORION/MOS C2C Partner

Date

Average

Peak Daily

(2010)

Message Size

Volume

IBM

MOS

Oct

2K

75

IBM

GSI

Oct

5K

1125

SR Create/Update FS-GSI

GSIC2C Partner Task/Note Create/Update

SCM-GSI

Supplier PO

GSI

Supplier

Jan

20k

7200

SCM-GSI

Supplier Change PO

GSI

Supplier

Jan

5k

100

SCM-GSI

Supplier ASN

Supplier

GSI

Jan

5k

7200

SCM-GSI

Supplier SO Shipping Docs

GSI

Logistics

Jan

500k

20500

Jan

20k

2500

GSI

Jan

5k

2500

Supplier SCM-GSI

Supplier Ship Instruction

GSI

Logistics Supplier

SCM-GSI

Supplier ePOD

Logistics Supplier

SCM-GSI

Supplier Invoice

Supplier

GSI

Jan

20k

200

SCM-GSI

Customer Invoice

GSI

Customer

Jan

50k

5000

OM-GSI

Customer PO

Customer

GSI

Jan

350K

130

Customer

Jan

15K

130

Jan

350K

130

Jan

350K

400

Jan

200K

1

Total

47191

/Reseller OM-GSI

CBOD

GSI

/Reseller OM-GSI

Acknowledge PO

GSI

Customer /Reseller

OM-GSI

Show Sales Order

GSI

Customer /Reseller

OM-GSI

Product/Price File Update

GSI

Customer /Reseller

Table 3 – Sun System Integration: B2B Bridges

The A2A groups handle messaging between GSI and systems for Agile (AGI), Manufacturing (MFG), Global Deal Management Tool (GDMT), Global Trade Management (GTM) and My Oracle Support (MOS). Details about the bridges are provided in Table 4.

8


Bridge

Interface Name

Source

Target

Group AGI-GSI

Validate Engineering Change

Date

Average

Peak Daily

(2010)

Message Size

Volume

Agile

GSI

Jan

75K

100

Agile

GSI

Jan

75K

100

Orders AGI-GSI

Create/Update Engineering Change Orders

AGI-GSI

Create/Update Item (N/A)

Agile

GSI

Jan

N/A

0

AGI-GSI

Generate New Item Number

Agile

GSI

Jan

3K

2000

AGI-GSI

Publish Item Attribute Updates

GSI

Agile

Jan

5K

2000

AGI-GSI

Publish Engineering Change

GSI

Agile

Jan

3K

1000

GSI

MFG

Jan

1.5K

320

GSI

MFG

Jan

1500K

320

GSI

MFG

Jan

12K

6500

Orders Updates MFG-GSI

Publish Unit Data for the System Serial Number

MFG-GSI

Publish the BOM for the System Serial Number

MFG-GSI

Publish Unit Data for Failure Analysis

MFG-GSI

Print the System and Ship Label

GSI

MFG

Jan

1K

120

GDMT-

Publish the Quote Details to

GSI

GDMT

Jan

55K

600

GSI

GDMT

GTM-GSI

Sales Order DRPL export DRPL

GSI

GTM

Jan

70K

16000

check GTM-GSI

Supplier Customer DRPL check

GSI

GTM

Jan

2K

72000

GTM-GSI

Item, Customers and Suppliers

GSI

GTM

Jan

12K

5000

MOS-GSI

Create/Update SR header in GSI

MOS

GSI

Oct

2K

1500

MOS

GSI

Oct

1K

1400

MOS

GSI

Oct

5K

1200

based on MOS CR MOS-GSI

Create task in GSI based on MOS CR

MOS-GSI

Create task Note based on MOS CR

MOS-GSI

Sync GSI task creation to MOS

GSI

MOS

Oct

1K

500

MOS-GSI

Sync GSI note creation to MOS

GSI

MOS

Oct

1K

6700

MOS-GSI

Sync GSI task update to MOS

GSI

MOS

Oct

1K

17000

Total

134040

Table 4 – Sun System Integration: A2A Bridges

9


Enabling “Hot Swap” of Partners

The integration of Sun into Oracle included replacing some Sun legacy partners. The messaging architecture had to be flexible to support rapid on-boarding of new partners. In fact, Oracle switched Logistics partners twice during the 11 months of the project due to performance issues, each time with a 30 day notice. Each of these involved modifications to the message translation maps, to error routing and to partner notifications. Because Fusion Middleware provides out-of-the-box integrations and support for several messaging languages, these modifications were performed with simple configuration changes that did not involve patchings or code changes.

Enabling Integration Options for Inbound and Outbound Messages

By building its messaging solution around its SOA suite, Oracle was able to get a 100% standards based, hot pluggable infrastructure with a wide range of Out of box (OOB) integration options available when adding new applications. OOB adapters are available that fit every kind of integration parameter asynchronous or synchronous, single or batch, small or large. Adapters available for inbound and outbound messaging are shown in Figures 3 and 4.

Figure 3 –Integration options for outbound R12 ERP/11g

10


Figure 4 – Integration options for inbound 11g SOA/R12 ERP

Providing Mechanisms for System Management

System Management is important for monitoring messages and the transactions that they are part of. Three different mechanisms are used for System Management of the Messaging Infrastructure at Oracle depending on the information needed. The Fusion Middleware Console in Oracle Enterprise Manager (EM) 11g is used to provide system level monitoring of composites, components, and B2B endpoints. In addition, it allows log files and end-to-end flows to be viewed. The Administrative Console in Oracle WebLogic Server 11g is used to manage environments such as JVMs and Servers. Oracle also uses BAM for real-time monitoring of business processes and services so that events can be analyzed as they occur and acted upon either automatically or manually. An example is the monitoring of a PO to provide information on the process stage of a transaction.

11


Figure 5 – Clockwise from top left: System level management with EM, Server management with WebLogic Server Console, Service Request monitoring with BAM, Pipeline management for Oracle’s Access Provisioning System (APS) with BAM

Results from Oracle’s Messaging Implementation By simplifying the Messaging solution as covered in the previous section, the team was able to bring down Messaging costs. Messaging costs, as a fraction of the IT budget, went from pre-acquisition figures of 12% for Sun and 2% for Oracle, to 2% for the combined entity. This section discusses results of the IT Messaging integration and simplification process undertaken by Oracle. Figure 6 shows the implementation timeline for the project. The technical and architectural pieces had been put in place before the first User Acceptance Testing (UAT) in September 2010 and additional functional modifications were completed in time for a second UAT in November 2010, allowing the new messaging solution to be up and running by January 2011.

Figure 6 – Implementation timeline for the IT Messaging project

12


The reduced complexity of the final messaging infrastructure brought down the size of the team developing and supporting the messaging infrastructure 5x and reduced the time needed to onboard new partners 6x as shown in Table 7 below.

Area

Before

Messaging costs as a

After

12%

2%

Over 50 people

10 people

Time needed to onboard a

Six months or more in cases of logistics

Less than 30 days for each of the logistic

new partner

partners due to the complex messaging

partners onboarded during the project

fraction of the IT budget Size of team developing and supporting the Messaging infrastructure

infrastructure. Table 7 – Before and after picture for the Messaging Solution

Average messaging loads handled for the three areas of Customer Support, Partner Orders and Fulfillment in 2011 are provided in Table 6 below.

Area

Load

Customer Support

80k B2B and 750k A2A transactions processed on average each month

Partner Orders

Over $1bn in XML orders processed in the first year. The Orders went straight into GSI offering real automation with no need for double entering.

Fulfillment

18k POs sent to manufacturers on average every month. 25k shipping instructions sent to the logistics provider on average every month.

Table 6 – Messaging loads by area

The End State Architecture of the Messaging Solution

The end state of the combined B2B and A2A messaging solution as of January 2011 used only Oracle products and is shown in Figure 7. A 100% Oracle Integration Architecture resulted in low support and maintenance costs. Fusion Middleware (11g) and R12 EBS provide mechanisms for common Error Handling, common Processes, out of the box Message Definitions and Integration, and common Metrics and Dashboards through BAM.

13


Figure 7 – The End State of the combined B2B and A2A Messaging solution

Common processes were possible through the definition of generic composites built using Service Oriented Architecture (SOA) Suite 11g since the composites were re-usable by all simple process flows. The Generic Composites were used in several bridges to provide standard error handling, retry, routing rules, metrics and reporting. This simplified deployment and support. It also provided a generic transformation and routing framework. Common error handling composites allowed errors to be captured in a standard way across all components of the end-to-end solution, providing a single work list for errors. The composites built were reusable for both XML and EDI formats across B2B and A2A messages. Table 7 shows how the Fusion Middleware architecture addressed problems that existed before the consolidation.

Frameworks Messaging Architecture

Description Consolidated SOA end solution based on Fusion Middleware 11g that offers highavailability, lower support costs and easy extensibility

Transmission Formats

Only four formats were in use at Go Live in January 2011. OAG XML output is easily translated to standard XML/EDI protocols with an XSLT Translation call to B2B engine

B2B Messaging Areas

Customer Support, Partner Orders, Fulfillment

A2A Messaging

There is a single entry point for all A2A and B2B messages. Common SOA standards are used for all bridges.

Gateways

A single gateway supported by single team.

Table 7 – Snapshot of the Oracle Messaging solution after the Sun integration

14


Lessons Learned Oracle’s use of Fusion Middleware in implementing a robust and scalable messaging solution has provided lessons that are relevant to enterprise customers with similar requirements. Oracle IT found it most productive to leverage standard and re-usable components within Fusion Middleware for all translation, system monitoring and error needs. This allowed the team to narrow development work on the communication bridges to implementing specific functional business logic. The team also found that, where several components are in place for a messaging solution, an all Oracle end state provided the best scalability and lowest maintenance costs when setting up or expanding integrations. Oracle also suggests keeping to as small a set of messaging and protocol standards as possible. Maintenance costs go up proportionately with the number of protocols used, due to additional regression testing requirements. This means being firm with partners where possible, to limit the protocols they want to use for messaging. To ensure high availability, Oracle IT recommends providing both physical and logical redundancy. This approach has allowed Oracle to add JVMs and nodes hot since go live without issues or downtime. In addition, separating lower priority transactions from higher priority ones ensures availability for critical transactions. The above approaches will ensure that the messaging solution is flexible and scalable. Finally, Oracle IT recommends customers make use of Fusion Middleware’s partner downtime features. If a partner’s systems are ever offline for regular maintenance or upgrades, the downtime period can be noted in the B2B messaging system so that new transactions and retries are not sent during that window. Instead, transactions simply get queued during the downtime. This saves system resources and provides cleaner handling.

Conclusion Oracle, like many of its enterprise customers, must constantly integrate new partners and acquisitions into its messaging solution. The Sun acquisition presented a challenge, but also gave Oracle the opportunity to put in place a high-availability, flexible and scalable messaging solution. The team was able to bring down Messaging costs as a fraction of the IT budget, from pre-acquisition figures of 12% for Sun and 2% for Oracle, to 2% for the combined entity. Oracle realized savings primarily because its Fusion Middleware-based solution required substantially fewer development and support resources than Sun’s fragmented legacy systems. Oracle’s experience indicates that both scalability and cost savings can be achieved by architecting solutions around reusable SOA composites, reducing the number of components and transmission formats in use, and by largely avoiding the use of third party components. Fusion Middleware 11g provided the capabilities required to build such a solution.

15


Reducing IT Messaging Costs:

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the

How Oracle reduced Sun’s IT messaging costs

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other

6x using Fusion Middleware

warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are

February 2013

formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Authors: Barry Geraghty, IT Director; Vinay Dwivedi, Principal Product Manager; Jeffrey Pease, Vice President; Dave Stephens, Senior Vice President World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.

UNIX is a registered trademark licensed through X/Open

Company, Ltd. 0611

Worldwide Inquiries:

Phone: +1.650.506.7000

Fax: +1.650.506.7200

oracle.com

16