Stop the clocks: How disaster resilience can protect pharma manufacturers

Chris Whitehead, managed platform product manager at SolutionsPT, looks at the importance of disaster resilience and the solutions available to help protect pharmaceutical manufacturers from the loss of critical data and interruptions to production.

Changes to IP protection laws have left pharmaceutical manufacturers facing fierce competition and urgently looking for ways to streamline production, boost productivity and minimise downtime whilst adhering to a strict regulatory framework. With many pharmaceutical plants now moving towards continuous operation, the threat of disruption to production looms large and is costlier than ever before.

Unlike other industries, pharma has not always had to pay close attention to manufacturing costs.  With generous margins and limited competition, the high price tag of finished drugs often compensated for any inefficiencies in production.  However, the intellectual property (IP) landscape in pharmaceuticals has changed. New competitors are emerging, increasing the pressure on manufacturers to fast-track new drug innovation, increase the speed at which they introduce new drugs to market and claw back profits with improved margins and better operational efficiency.

As a result, innovators in the pharmaceutical industry are quickly waking up to the fact that, as manufacturers, they must go back to basics and adapt and improve their processes in the same way automotive and fast-moving consumer goods (FMCG) businesses have before them. The first task is to embark on an improvement campaign to increase efficiency and protect plants from downtime.

As a tightly regulated sector in which downtime equates to potentially huge financial losses, manufacturers in the pharmaceutical industry typically experience comparatively less unplanned downtime than those in other sectors. However, when disruption does happen, it is much costlier, leading to potential drug shortages, fines and compliance failures.

Pharmaceutical manufacturers are heavily regulated in terms of traceability and batch serialisation, and, as such, any downtime during a batch run will almost certainly ensure the entire batch has to be discarded as there would otherwise be no continuity in the batch data.  Waste such as this has huge cost implications in terms of materials, time and profit.

Threats to continuity do not only impact those who manufacture by batch.  Many of the industry’s manufacturing leaders are exploring continuous manufacturing as a route to increase quality, process throughput, and profit. Pharmaceuticals that are made using continuous manufacturing are moved non-stop within the same facility. This new approach is delivering significant benefits to pharmaceutical businesses but from an operational perspective, it is also creating a reliability challenge. During batch production, quality testing would normally be carried out at regular intervals. However, during continuous manufacture, the production process doesn’t stop, so reliability and product quality must always be guaranteed.

Machinery failure is a common cause of downtime in pharmaceutical manufacturing, but increasingly one of the most pressing issues is cyber security. The 2017 attack on Merck saw a ransomware attack that hit every location within the company, including its industrial control systems. The pharma giant suffered a total worldwide disruption of its operations and had to halt production of new drugs - negatively impacting on the company’s bottom line.

It is estimated that 96% of ransomware victims lose access to their data for more than one-day, heavily impacting production. With such a heavy reliance on OT systems to maintain uptime and productivity within the pharmaceutical space, it is essential to have disaster resilience and business continuity systems in place to ensure minimal impact across the plant, should a ransomware attack strike. 

From a patient safety perspective, pharmaceutical manufacturers have a public responsibility to ensure that the production of drug substances and final drug products are not disrupted. Beyond the need to ensure the uninterrupted supply of vital medicines, drug companies have a responsibility to mitigate threats to their business operations and assets, which is why building disaster resilience into the pharma manufacturing environment, rather than relying on back-ups, is essential.

Traditional backup solutions are often not specifically designed for OT systems and may present issues. For example, they often only perform a CRC check on the backup data which doesn’t provide assurance that systems could be fully recoverable. In addition, these backups are often only tested in a recovery scenario by support teams.

The latest managed disaster resilience services can highlight anomalous file level changes through continual monitoring. Backups are rigorously tested on a defined schedule by recovering the system in a ‘offline state’ while checking key services, dependencies and where applicable, integrity of critical databases and data sets. With ‘hot standby’ technology, systems can be quickly recovered with minimal of downtime on primary or backup hardware. Initial infection and lateral movement can also be tracked and contained, ensuring that critical pharma processes can continue while support teams investigate the outbreak and remediate the affects.

Compliance demands, a highly competitive environment and the delivery requirements of a complex range of end customers, including hospitals, insurance firms, supermarkets and pharmacists, means that operations continuity is crucial in pharmaceutical manufacturing. As a result, having a good b business continuity and disaster resilience process in place is essential and this isn’t just about having the ability to retrieve files but knowing how and how soon you can carry on production.

Pharma businesses need to ensure they don’t leave disaster resilience too late. It is essential to plan for what will happen when the worst happens and to test that plan out on a regular basis. Then, when something does go wrong, you’ve been through the procedures, you’ve been through a disaster resilience shake-down test and the people and the teams are all in place, so everyone knows what needs to be done to get things back up and running as quickly as possible. If this is something that your operation would struggle with, considering an externally managed service could be the solution.