IT SOLUTION PROVIDER

CASE STUDIES                            PROJECTS       

Case Study

CalPERS Backbridge Decommission Project

Client Profile

The California Public Employees’ Retirement System (CalPERS) manages pension and health benefits for approximately 1.5 million California public employees, retirees, and their families. As of June 30, 2006, it has provided benefits to 1,048,895 active and inactive members and 448,271 retirees. CalPERS membership is divided approximately in thirds among current and retired employees of the state, schools, and participating public agencies.

Problem/Challenge/Requirements

During the mid-nineties, the California Public Employees Retirement System (CalPERS) undertook a major initiative to revamp its systems which resided on an IBM mainframe. The initiative’s main goals were to:

  1. Create an enterprise-wide, relational corporate database (CDB) implemented in Oracle that all systems could utilize
  2. Migrate applications to an object oriented platform that could easily adapt to new or changing business rules

This initiative, the CalPERS Online Member and Employer Transaction System (COMET), was only partially completed. While many systems were incorporated into COMET, a handful remained on the mainframe. The underlying corporate database, CDB, was the system of record for member and employer data, but legacy systems remaining on the mainframe had no means to directly query these data.

In order to enable systems in separate universes to share the same data source, CalPERS devised the Backbridge solution by which updates to CDB would be sent during nightly batch runs to mainframe member (MBR), employer (EPR), and member address (MBR ADDR) data files (key-sequenced indexed sequential). A series of batch PL/SQL programs extracted updates made during the day from CDB, formatted them to match the Virtual Storage Access Method (VSAM) record layouts and sent the data via FTP to the mainframe. Another program then conducted further transformations like packing fields and dealing with ASCII to EBCDIC translation. This solution worked reasonably well but had three main imperfections:

  1. Legacy programs did not have access to up-to-the-minute data, but only information from the previous day.
  2. Record deletions in CDB could be submitted by the batch processes, but they were not applied correctly to the VSAM file
  3. Significant costs were incurred supporting this infrastructure and dealing with data discrepancies and data cleanup it required

After considering alternatives, CalPERS determined that in order to retire the BackBridge infrastructure, legacy systems would have to be reengineered to utilize a middleware layer that would directly call CDB.

Trinity Technology Group began the Backbridge Decommission Project faced over 200 batch and online COBOL and Natural projects that access VSAM databases. Some of these batch programs required several hours to run. Access to the CBD had to be maintained 24/7. Performance had to be maintained as a resident data source was replaced by network access, a relational database replaced native flat files, and sequential and skip-sequential reads were eliminated. In addition, online requests had to maintain a < 1 second response time.

Approach/Implementation

Trinity Technology Group devised the Backbridge Decommission (BBD) to enable legacy programs to retrieve data directly from CDB. Trinity built a new middleware solution that allowed legacy online and batch programs access the CDB directly. The overview of this process is shown as follows:

The EPR and MBR databases were decommissioned and the solution was deployed in a manner complete transparent to business functionality. Deployment included a “safety net” strategy that allowed for seamless rollback if necessary.

During an initial proof of concept (POC) phase (2-3 months), Trinity accomplished the following:

  • Built and validated a middleware solution for legacy online and batch programs
  • Implemented error management strategy
  • Demonstrated load balancing
  • Accommodated problematic programs (skip/sequential)
  • Proved the solution can handle production volumes with less than 30% increase in completion times for production batch processing and negligible difference in production online response times

In addition, the actual implementation phase (6 months) accomplished the following:

  • Acquired team commitments and management support
  • Established development and test environments
  • Conducted comprehensive testing
  • Refined infrastructure
  • Minimized modifications to existing programs

Trinity’s solution utilized the following physical architectural components:

  • z/OS 1.5
  • COBOL and Natural
  • Common Modules (Assembler)
  • Common Modules (COBOL)
  • SoftwareAG EntireX Broker and RPC Client
  • Windows 2000 SP4 VMWare, 2.2Ghz CPU X 4, 1Gb RAM
  • SoftwareAG EntireX RPCServer implemented in Java
  • 4 Online Servers
  • 1 Batch Server
  • JDBC thin driver
  • HPUX 11i 800 MB CPU X 8, 12 Gb RAM
  • ORACLE 9.2.0.5
  • PL/SQL Packaged Procedures
  • 1 Gigabit Network – TCP/IP

This architecture is illustrated as follows, showing new components in orange:

Results

Shortly after implementation, CalPERS CEO Fred Buenrostro wrote:

“On November 11th, the Legacy Enrollment Database and its associated "backbridge" process were decommissioned. For those of you who have lived the pain of reconciling data discrepancies between COMET and the legacy systems, the significance of this achievement is huge! The solution went into production seven weeks ahead of schedule and well under budget (the budget was originally estimated at $4-5 million, but delivered for only $1.2 million) and the ongoing annual savings to CalPERS is estimated at $500,000 - $700,000. However, the greatest benefit to this project is that we've eliminated a major source of our data integrity and redundancy problems.”

Trinity Technology Group concluded this project, accomplishing the following:

  • Deployed on time and on budget, with no production downtime
  • Created a stable environment with no production downtime or significant issues
  • BackBridge successfully decommissioned
  • Although batch performance was 1.2 to 3 times longer in the new infrastructure, the overall impact did not translate into a similar increase in total batch processing time.
  • No noticeable change in online performance.
  • All legacy programs, including “problematic” programs that perform sequential reads, are accommodated in the new infrastructure.
  • Enhancements included 11 annual programs

Future

The success of this project underscores Trinity Technology Group’s ability to provide technically feasible, advanced middleware solutions for production applications. Trinity crafts cost-effective solutions that can accommodate legacy programs performing both direct and skip/sequential database accesses with minimal or no changes. Furthermore, this solution is scalable to handle large volumes of database calls under actual production conditions, and it can be configured and tuned to achieve similar performance results in production as during testing.

 

Case Study

Enterprise Service Bus (ESB) Implementation

Client Profile

The California Public Employees’ Retirement System (CalPERS) manages pension and health benefits for approximately 1.5 million California public employees, retirees, and their families. As of June 30, 2006, it has provided benefits to 1,048,895 active and inactive members and 448,271 retirees. CalPERS membership is divided approximately in thirds among current and retired employees of the state, schools, and participating public agencies.

Problem/Challenge/Requirements

CalPERS employed Enterprise Architecture Integration (EAI) products to integrate business to business (B2B) and Application to Application (A2A) functionality. EAI product allows CalPERS investment managers and custodians, such as State Street Bank to send investment data to CalPERS for updating internal financial applications. EAI product also allows CalPERS to exchange health-related data with health carriers.  CalPERS also uses EAI/ESB product for Government to Government (G2G) integration such as File/Data exchange between State Controller’s Office and CalPERS

CalPERS wished to extend its architectural base by adding service oriented architecture (SOA) components. SOA components would allow CalPERS to develop web services that allow authorized users to draw data and business functionality from standalone legacy applications across the Internet.

Approach/Implementation

Traditional EAI is an asynchronous message-based communication from one location to another. It employs “loose coupling;” that is to say, EAI products can interact with multiple data sources and applications regardless of their operating system or native technology. EAI products also include the option for synchronous messaging through remote procedure calls (RPC).

Enterprise Service Bus (ESB) is a commercial off the shelf (COTS) software product. However, it is not a single application but rather a combination of SOA tools that provide the same benefits as EAI, but extends it to add web service support. With web service support added, any authorized user can be given access to native databases and applications through a web browser. ESB utilizes the SOAP, Java messaging service (JMS) and xml standards.

Because ESB is a combination of tools that were to be added to an existing environment, Trinity consultants performed a thorough evaluation of the suitability of grafting ESB onto the CalPERS enterprise before proceeding with an implementation. Once ESB was functioning, additional services were created and made available for further application development.

The following diagram illustrates how ESB provides interconnectivity to legacy applications and databases. It includes CalPERS’ custom-built services, Logging, Cryptography, and Cross Reference Lookup.

 

Results

ESB functionality was achieved in a totally transparent manner. Currently, CalPERS enterprise can make use of its functionality.

Future

In addition to the developed services which are now available for use within CalPERS, additional services will soon be available to extend this functionality.

 

Case Study

SOA Implementation

Virtual Information System Architecture (VISA)

Client Profile

The California Public Employees’ Retirement System (CalPERS) manages pension and health benefits for approximately 1.5 million California public employees, retirees, and their families. As of June 30, 2006, it has provided benefits to 1,048,895 active and inactive members and 448,271 retirees. CalPERS membership is divided approximately in thirds among current and retired employees of the state, schools, and participating public agencies.

Problem/Challenge/Requirements

In 2005, CalPERS’ leadership determined that its IT environment was no longer adequate to meet rapidly changing business needs. Its current environment consisted of numerous standalone systems and databases that were increasingly difficult to maintain, costly to support, and unresponsive to growing needs.

CalPERS further determined it would be best served with IT environment that facilitates the reuse of infrastructure and business services, which any authorized party could use to respond to business needs. Publishing these services and providing a coherent framework within which to manage them would accommodate current and yet-to-be-defined business requirements.

This new enterprise architecture was to be based on an industry standard architecture approach known as Service Oriented Architecture (SOA) and was named CalPERS “Virtual Information Systems Architecture” (VISA).

Because SOA implementation represents a leading edge technology, CalPERS first engaged Trinity Technology Group to launch a prototype development phase. This prototype phase would determine whether full scale development was feasible.

Upon the prototype’s acceptance, VISA Phase I was designed to setup the SOA infrastructure suite that includes hardware and software. VISA Phase I implements production quality common infrastructure services such as Logging, Cryptography and Cross reference lookup.

Approach/Implementation

As successful existing SOA implementations were still few in number, Trinity Technology Group began at the ground level, asking very basic questions about how SOAs function in the real world. Trinity researched and evaluated the then-current state of the art concerning:

§  Available SOA standards

§  Security and identity management (SAML, X509, and LDAPv3)

§  Enterprise Service Bus (ESB)

§  Orchestration of services and user activity using Business Process Execution Language (BPEL)

§  Messaging (JMS)

§  Service registration and repository (UDDIv3)

§  Granularity (fine grained versus coarse grained services)

§  Frameworks

§  Quality of service using Web Services Management tool

§  Governance using Web Services Management tool

Specifically, Trinity sought to determine how an SOA implementation would function alongside CalPERS existing system.

The following diagram illustrates the deployment configuration for this prototype implementation:

Results

In the six month prototype phase, Trinity successfully crafted a working SOA environment able to function within CalPERS existing environment.

The prototype project leveraged existing JCAPS ESB technology and utilized Open Source alternatives to proprietary software. Using this approach, the Trinity saved CalPERS approximately $400,000.

Trinity determined that achieving the espoused benefits of a service-oriented architecture requires understanding of how to use the architecture appropriately. Trinity also determined that application of standard legacy application-oriented analysis and design techniques would not yield the appropriate use of the architecture.

Rather, Trinity recommended CalPERS adopt and promote a service-oriented analysis and design methodology. The methodology must provide a framework that drives analysis from a business process through to a service identification and specification level. The methodology must provide a logical basis for how to identify and categorize services from descriptions of required function. This approach, coupled with appropriate guidelines and principles for designing stateless, independent, and highly reusable services are critical to success.

As a result of the successful acceptance of VISA Prototype phase, Trinity continued with the successful development of VISA Phase I. Trinity selected and implemented the best solution for managing and securing services using a policy-based security mechanism, using standards such as BPEL, WSDL, LDAPv3, UDDIv3, SAML and X509. This application has been accepted by CalPERS and is now available for use within CalPERS enterprise architecture.

Future

As of 2Q 2008, Trinity continues with the development of VISA Phase II, which will provide additional services for CalPERS enterprise within the structure successfully created in Phase I..

The success of this project underscores Trinity Technology Group’s ability to provide technically feasible, advanced SOA solutions for production applications. Trinity crafts cost-effective, leading edge solutions that can accommodate legacy programs’ continued use within a state of the art configuration.