CASE STUDIES                            PROJECTS       

Case Study

CalPERS Backbridge Decommission Project

Client Profile

The California Public Employees’ Retirement System (CalPERS) manages pension and health benefits for approximately 1.5 million California public employees, retirees, and their families. As of June 30, 2006, it has provided benefits to 1,048,895 active and inactive members and 448,271 retirees. CalPERS membership is divided approximately in thirds among current and retired employees of the state, schools, and participating public agencies.

Problem/Challenge/Requirements

During the mid-nineties, the California Public Employees Retirement System (CalPERS) undertook a major initiative to revamp its systems which resided on an IBM mainframe. The initiative’s main goals were to:

1. Create an enterprise-wide, relational corporate database (CDB) implemented in Oracle that all systems could utilize
2. Migrate applications to an object oriented platform that could easily adapt to new or changing business rules

This initiative, the CalPERS Online Member and Employer Transaction System (COMET), was only partially completed. While many systems were incorporated into COMET, a handful remained on the mainframe. The underlying corporate database, CDB, was the system of record for member and employer data, but legacy systems remaining on the mainframe had no means to directly query these data.

In order to enable systems in separate universes to share the same data source, CalPERS devised the Backbridge solution by which updates to CDB would be sent during nightly batch runs to mainframe member (MBR), employer (EPR), and member address (MBR ADDR) data files (key-sequenced indexed sequential). A series of batch PL/SQL programs extracted updates made during the day from CDB, formatted them to match the Virtual Storage Access Method (VSAM) record layouts and sent the data via FTP to the mainframe. Another program then conducted further transformations like packing fields and dealing with ASCII to EBCDIC translation. This solution worked reasonably well but had three main imperfections:

1. Legacy programs did not have access to up-to-the-minute data, but only information from the previous day.
2. Record deletions in CDB could be submitted by the batch processes, but they were not applied correctly to the VSAM file
3. Significant costs were incurred supporting this infrastructure and dealing with data discrepancies and data cleanup it required

After considering alternatives, CalPERS determined that in order to retire the BackBridge infrastructure, legacy systems would have to be reengineered to utilize a middleware layer that would directly call CDB.

Trinity Technology Group began the Backbridge Decommission Project faced over 200 batch and online COBOL and Natural projects that access VSAM databases. Some of these batch programs required several hours to run. Access to the CBD had to be maintained 24/7. Performance had to be maintained as a resident data source was replaced by network access, a relational database replaced native flat files, and sequential and skip-sequential reads were eliminated. In addition, online requests had to maintain a < 1 second response time.

Approach/Implementation

Trinity Technology Group devised the Backbridge Decommission (BBD) to enable legacy programs to retrieve data directly from CDB. Trinity built a new middleware solution that allowed legacy online and batch programs access the CDB directly. The overview of this process is shown as follows:

The EPR and MBR databases were decommissioned and the solution was deployed in a manner complete transparent to business functionality. Deployment included a “safety net” strategy that allowed for seamless rollback if necessary.

During an initial proof of concept (POC) phase (2-3 months), Trinity accomplished the following:

• Built and validated a middleware solution for legacy online and batch programs
• Implemented error management strategy
• Demonstrated load balancing
• Accommodated problematic programs (skip/sequential)
• Proved the solution can handle production volumes with less than 30% increase in completion times for production batch processing and negligible difference in production online response times

In addition, the actual implementation phase (6 months) accomplished the following:

• Acquired team commitments and management support
• Established development and test environments
• Conducted comprehensive testing
• Refined infrastructure
• Minimized modifications to existing programs

Trinity’s solution utilized the following physical architectural components:

• z/OS 1.5
• COBOL and Natural
• Common Modules (Assembler)
• Common Modules (COBOL)
• SoftwareAG EntireX Broker and RPC Client
• Windows 2000 SP4 VMWare, 2.2Ghz CPU X 4, 1Gb RAM
• SoftwareAG EntireX RPCServer implemented in Java
• 4 Online Servers
• 1 Batch Server
• JDBC thin driver
• HPUX 11i 800 MB CPU X 8, 12 Gb RAM
• ORACLE 9.2.0.5
• PL/SQL Packaged Procedures
• 1 Gigabit Network – TCP/IP

This architecture is illustrated as follows, showing new components in orange:

Results

Shortly after implementation, CalPERS CEO Fred Buenrostro wrote:

“On November 11th, the Legacy Enrollment Database and its associated "backbridge" process were decommissioned. For those of you who have lived the pain of reconciling data discrepancies between COMET and the legacy systems, the significance of this achievement is huge! The solution went into production seven weeks ahead of schedule and well under budget (the budget was originally estimated at $4-5 million, but delivered for only $1.2 million) and the ongoing annual savings to CalPERS is estimated at $500,000 - $700,000. However, the greatest benefit to this project is that we've eliminated a major source of our data integrity and redundancy problems.”

Trinity Technology Group concluded this project, accomplishing the following:

• Deployed on time and on budget, with no production downtime
• Created a stable environment with no production downtime or significant issues
• BackBridge successfully decommissioned
• Although batch performance was 1.2 to 3 times longer in the new infrastructure, the overall impact did not translate into a similar increase in total batch processing time.
• No noticeable change in online performance.
• All legacy programs, including “problematic” programs that perform sequential reads, are accommodated in the new infrastructure.
• Enhancements included 11 annual programs

Future

The success of this project underscores Trinity Technology Group’s ability to provide technically feasible, advanced middleware solutions for production applications. Trinity crafts cost-effective solutions that can accommodate legacy programs performing both direct and skip/sequential database accesses with minimal or no changes. Furthermore, this solution is scalable to handle large volumes of database calls under actual production conditions, and it can be configured and tuned to achieve similar performance results in production as during testing.