Pioneering virtual server solution in plant control system upgrade at Cementos Argos Cartagena

Bringing virtualisation to the cement industry

After ten years of successful operation of our ECS/ControlCenter™ V7 control system, the system was starting to show its age. Cementos Argos was therefore looking for a new solution to secure the digital future of the Cartagena plant, based on our latest control software, ECS/ControlCenter V8 .

We certainly provided that. But we didn’t stop there.

Enter into the story Hasmukh Patel, Technical Manager at FLSmidth. For the previous year, Hasmukh and his team have been working to bring the benefits of virtualisation – long enjoyed by the IT industry – to the world of operational technology (OT). And to do so in a standardised way that would reduce cost and overcome implementation challenges.

Achieving the (almost) impossible

Virtualisation had long been thought almost impossible in an OT environment because it can involve a degree of latency – or delay – in the system. While some latency is OK in most IT applications, e.g. with office-based applications, such as email, databases, internet browsers, etc., there is no such luxury or leeway allowed in the OT environment of a manufacturing facility, such as a cement plant.

“In OT, you need the full power of the computer hardware, software and network resources available all the time. Any latency in performance or software errors due to hardware resource constraints are simply not acceptable. When a control room operator pushes a button to start/stop equipment or change setpoints in an industrialised automation environment, you need those control actions delivered instantly, without delay,” Hasmukh explained.

Conventional wisdom did not deter Hasmukh and his team, however. They developed a virtualisation solution tailored to the OT environment and optimised ECS/ControlCenter V8  platform for virtualised environments along the way.

Why make this effort? Because Hasmukh and his team believe virtualisation holds significant potential to benefit our customers. Benefits that were ultimately persuasive to the forward-thinkers at Cementos Argos, who gave Hasmukh and his team the chance to demonstrate what they could deliver with their futuristic technology solution.

This turned a traditional control system upgrade into something new: the first FLSmidth virtualised automation system, following the new OT optimised standards developed in-house.

The team at the Cementos Argos Cartagena cement plant, from left to right: Jamer Mercado - Control Room Operator, Elkin Osorio - Plant Process Lead, Jose Blanquicett - Plant Process Lead, Alejandro Ortiz - Process Director

Forget the technical jargon. What are the benefits?

Built-in backup for all, not just critical, systems

A virtualised system delivers much greater peace of mind to plant operators. Backup (or redundancy, in technical terms) is built-in for all systems, rather than just those deemed critical. At Cementos Argos, this means that the plant automation team can rest easy, knowing the new system’s three virtual host servers will always deliver and keep the automation applications running 24/7, without manual intervention, even if one of the physical host servers fails.

Easily managing maintenance and security updates

It’s also easier to keep the new system secure. Unlike in traditional set-ups, where security updates must be installed on each server individually, there is one central operation console for the virtualised system. On top of that, the solution has an easy-to-use backup and restore function, adding additional security and ensuring system uptime.

Another potential benefit of the new virtual solution is a ‘Test Scenario’ for applications during future software upgrades. This allows customers to test software upgrades before deployment for plant operations, helping to minimise the risks involved, as customers can easily revert back to previous application versions, if any issues occur.

“There are a lot of expected savings in the ease of maintenance, including applying security updates and application updates, as well as in the high availability and inherent automatic fail-over redundancy of all applications. Avoiding just one unscheduled stop during production is an important benefit to us. With this robust, high-availability, fault-tolerant solution, we expect uptime of our plant automation systems to be 99.99 percent,” said Elkin Gustavo Osorio, Process Coordinator at Cementos Argos.

Building cooperation between IT and OT teams

Another benefit comes from the need for IT and OT teams to work together with the new system to find new synergies and complement each other’s skills. In this project, the Cementos Argos corporate IT department participated in the Factory Acceptance Test, alongside their OT colleagues, at our office in Bethlehem, Pennsylvania, USA.

IT-OT collaboration was also instrumental in implementation, as all automation network switches had to be reconfigured to allow connection of virtual network switches. Other implementation challenges included validation of IP addresses on all nodes to make sure that no duplication or conflicts occurred, and that hardware had to be installed in existing rack cabinets.

Reduced energy consumption

Virtualised solutions require less power to operate and cool the hardware. This fits both with Cementos Argos’ ambitions to reduce the CO2 emissions from cement production and our own sustainability programme, MissionZero , which aims to significantly lower emissions from the cement industry with the target to operate zero-emission cement plants by 2030.

The bottom line: these benefits combine to offer a significantly lower total cost of ownership over a 3-5 year period.

One project but with implications far beyond Colombia

The success of virtualisation at Cartagena is an important step forward – both for the plant but also for the wider cement industry. A moment from testing at the plant is perhaps the best way of capturing the benefits. During the site acceptance test, one server in the three host-server virtual environment was shut down, followed – a deep breath later – by one more server being shut down.

“This was the defining ‘aha’ moment,” concluded Hasmukh. “In this new virtual platform, applications migrate seamlessly between host servers. When we pulled the power cable, applications just moved on to the next available host server, maintaining uptime performance without interruption. While we had redundancy of some critical applications before, this solution puts it on another, higher level, ensuring all applications continue to run flawlessly and thereby making sure the plant is always capable of achieving its production targets and desired performance benchmarks on emissions, quality, optimisation, operational costs.”

This was mission accomplished. And a game changer: proof that the benefits of virtualisation are now available in OT environments.

Technical highlights of the virtualised solution

• 3-2-1 architecture (3 servers, 2 switches, 1 central storage) based on DELL hardware.
• VLAN configurations for networks - VM Management, PLC/HMI systems, integration with existing plant network and office network, secure remote access.
• Supporting the recommendations of Cyber Security Standard IEC62443.
• Applications hosted on the platform:
  o ECS1 (primary control centre). 
  o ECS2 (secondary control centre). 
  o PDM (historian/plant data management). 
  o QCX (quality control system with Robolab and Auto-sampling). 
  o RAS (remote access server with Windows terminal services). 
  o PXP1 (mills optimisation). 
  o PXP2 (kiln and pyroprocess optimisation). 
  o CEM (Cemscanner). 
  o Engineering station (programming terminal). 
  o Management console for virtualised solution. 
• 5-year production level mission critical 24/7 support.
• 5-year PlantLine maintenance and service contract.

What is virtualisation?

According to Microsoft , virtualisation “creates a simulated, or virtual, computing environment, as opposed to a physical environment, that often includes computer-generated versions of hardware, operating systems, storage devices, and more. This allows organisations to partition a single physical computer or server into several virtual machines. Each virtual machine can then interact independently and run different operating systems or applications, while sharing the resources of a single host machine. By creating multiple resources from a single computer or server, virtualisation improves scalability and workloads, while resulting in the use of fewer overall servers, less energy consumption, and less infrastructure costs and maintenance.”