Forests of wooden derricks sprang up in a rural California valley when oil was discovered there in 1899. Today those towering structures are gone, replaced by the more familiar pumpjacks and steam-injection systems necessary to coax the crude from the earth. Most of the valley’s “black gold” is heavy crude with a viscosity much like molasses.

In 2013 a major California petroleum producer pumped 177 Mbbl/d from the valley. The company’s California business unit comprises six separate oil fields. The typical well site includes steam generators used to apply the pressure and heat necessary to force heavy crude above ground.

Some sites incorporate cogeneration systems, which use natural gas turbine/generator sets to produce electricity that supports site operations. Cogeneration in the California valley produces enough energy to supply the power needs of more than 1.5 million homes.

Challenge
Such complex processes and so much capital equipment demand a huge number of control assets. The company’s process control network (PCN) uses about 2,800 control devices distributed among the six oil fields.

For years, the quantity of devices, diversity of its valley operations, geographic distance between them and absence of an effective technology had complicated the company’s effort to create a comprehensive system for identifying, tagging and monitoring its control devices.

“They had a manual database of this type of information, but it was incomplete,” said Joe Zaccaria, reliability program manager for Rockwell Automation. The oil and gas company has a large installed base of Rockwell Automation control devices as well as equipment from multiple other vendors.

“The oil company had spent a lot of time and money putting people in the field to document part numbers, quantities, etc.,” he explained. “But this manual collection method couldn’t keep up with such a large and dynamic PCN installation, which includes several generations of mission-critical assets.”

The manual collection process itself carried risks for the company. Employees in the field could encounter hazards ranging from rattlesnakes to desert environmental conditions to the potential for exposure to hydrogen sulfate gas.

Concerns about a newer and potentially more devastating threat—a cybersecurity attack—loomed even larger.

The tipping point came in late 2011. A corporate SCADA audit identified cybersecurity risks related to the lack of reliable control asset data-keeping by the valley business unit. The audit was prompted by a new companywide cybersecurity policy based on the International Society of Automation 99 (ISA99) standard for industrial control systems.

“Until the early 2000s, control systems and devices were essentially immune to cyberattacks for a number of reasons,” he said. “One, the networks weren’t as open. And two, there were few hackers. Now plenty of them are looking for industrial vulnerabilities, especially where they can cause considerable collateral damage.”

Outsiders weren’t the only potential threat. Numerous system components were missing and considered stolen from control cabinets across the valley.

Solution
The oil giant’s California business unit turned to Rockwell Automation for a technological solution to centralize information gathering and continuously monitor the company’s assets. The automation company already was piloting a new Logix-based application that would do exactly that.

“Over the course of several months, we went through discussions with their IT department, their SCADA department and other key decision-makers,” Zaccaria said. “They all agreed that our new approach was the best method because it was safe and secure and applies to Allen-Bradley and third-party devices.”

At the heart of the automation solution is an innovative software-as-a-service technology. The new diagnostic reliability (DR) system automatically identifies, interrogates and monitors control hardware by continuously scanning the PCN. Proprietary code encrypted into the Allen-Bradley SoftLogix 5800 controller directs the DR system to locate and tag every device on the PCN—including programmable logix controllers and everything connected to them in the rack and subcomponents—using native industrial protocols to talk to each device. The DR system also retrieves detailed information about each piece of equipment, including but not limited to the part number, series version, serial number and firmware version.

The system exports all that information to a structured Microsoft SQL Server and displays it to the end user using FactoryTalk VantagePoint software from Rockwell Automation. This provides the oil company with the most essential tool for maintaining a cybersecurity strategy around a master inventory of all the devices on the PCN. The availability of a comprehensive and accurate control asset master inventory is essential to the effective creation and sustainability of a PCN cybersecurity strategy. Using FactoryTalk View SE HMI software, the DR system also takes a deep dive into each asset to understand its status, health and parameter changes.

The Rockwell Automation team has continuously added more tracking capabilities to the DR system, including scans for security vulnerabilities. For example, the system continuously checks the key switch status on processers. When a key switch is open, a hacker could potentially change the programming, an attack that could cause downtime and production losses or create safety and environmental hazards.

The DR system also reports if a device was moved from one location to another or if it goes missing. If the DR system shows that a device that was formerly on the network is gone, the system produces an exception report. Company personnel receive this report and follow up to see if the device has been removed or if it is some type of failure or fault.

An exception report is just one of many created in the FactoryTalk VantagePoint software based on information culled from the DR system. The software provides context to the data with web-based dashboards, allowing different users to view actionable information important to their roles.

In addition to the key switch status reporting, maintenance personnel can see if the processor has a low battery and dispatch a field employee to replace the battery before a failure occurs. The DR system data also are used to produce monthly scorecards detailing the PCN’s overall performance and reports on network anomalies.

Two Rockwell Automation field service engineers work onsite at one of the oil producer’s California campuses, helping the company’s team troubleshoot and correct issues identified by the DR system as well as build out its capabilities.

“We’re embedded with the customer to continuously develop more decision-support tools,” said Ray Spangler, field service engineer at Rockwell Automation. “The DR system is expanding as we explore, identify and implement new ways to use it.”

The PCN compliance program continues to evolve based on new objectives that are prioritized and incorporated into the service program, expanding the DR system’s capability. The oil company’s business unit is already in full compliance with the new corporate cybersecurity policy based on the ISA99 standard for industrial control systems. By moving from a manual database to automated real-time data collection, the company also can perform proactive maintenance. This is helping lower the cost of manpower in the field and yield more barrels of oil per day.