It is a common misconception that blockchain’s inherent security obviates the need for secure end-to-end computing infrastructure, especially for production consortium permissioned networks in regulated environments. Enterprise blockchains, such as the Linux Foundation’s Hyperledger Fabric, do provide crypto operations such as: encryption, hashing, digital signing, authentication and verification, however, they are not sufficient. Production networks require secured peers, ledger, state database, smart contracts, channels, and all internal and network communications. Traditional approaches to protecting the entire environment rely upon selective encryption of a security perimeter (e.g., disk, network traffic, application, and databases).  In addition to the substantial cost and administrative burden of this approach, the moment data leaves its designated perimeter, it is exposed.

Blockchain solutions also rely on traditional technologies for back-end processes like authentication, data processing and APIs.  In essence, the blockchain infrastructure is much like an armadillo: hard shell on the outside, but soft and vulnerable on the inside, especially on commodity hardware.  The blockchain may be secure, but if attackers can find a way in through poor access controls, for example, the whole infrastructure may be vulnerable. These components included everything from the network IP address, the blockchain layer, back-end components such as databases and object stores, the APIs, the externally-facing IP addresses and the mobile applications associated with the network.

Access control is critical to blockchain security; often permissions and access controls are implemented at the application level but not at the blockchain layer. If hackers compromised any application and accessed internal networks, they could hit any number of APIs associated with the blockchain. With no access controls, a malicious actor could literally do anything and everything they wanted on the chain itself.

Another common security problem: data stored off the blockchain and accessed through reference codes on the chain. Although the unique ID (hash) on the blockchain that references the data cannot be changed, a malicious insider could alter the original data in the off-chain database, and the blockchain reference code would then access the changed data; which could lead to elevation of privileges, account hijacking or accessing of information from an unauthorized perspective.

An often-overlooked element is the security of smart contracts. A smart contract is not just a piece of code; it is a representation of business logic that may secure a house on the blockchain, assure a digital identity, or even represent an escrow transaction between people buying and selling a used car. It is important that a smart contract is reliable and always does what it says it will. Smart contracts and other deployed applications should undergo a trusted build process complete with independent software attestation. Without a secure CI/CD process, someone could write malicious code that, once executed, would have granted command-level access to the hosting server and deploy backdoored smart contracts.

IBM’s open source Linux cloud server called LinuxONE draws upon the architectural lineage of the IBM mainframe but designed only for serving open source Linux (in all the standard distributions). Over 70% of today’s credit card, bank, airline or telco transactions are processed on this technology. LinuxONE also provides system availability measured in decades without a single point of failure and built with the world’s fastest commercial processor with dedicated I/O processors to move data with uncompromised integrity.

Pervasive Encryption

Not unlike the magical gadgets provided to James Bond by Q Branch, LinuxONE features distinct technologies that are particularly useful for securing blockchain nodes from external, insider, and developer attack vectors. IBM’s point of view is that all data at rest and in flight should be encrypted without the need to modify applications or databases. IBM LinuxONE’s Pervasive Encryption differentiates from Intel x86 by providing an industry unique real-time, end-to-end, holistic security encryption apparatus for the entire production environment.  Blockchain applications can benefit from the end-to-end encryption of data, for both data-at-rest (e.g., the ledger stored on the disk) and data-at- flight (e.g., transactions and peers’ communication). Traditionally, the largest barrier for running full-scale encryption has been the cost of the encryption and the performance load put upon the computing platform. Indeed, bolt-on security solutions can take up over 60% of system capacity. It would take 12 times the number of x86 servers and 7 times the capacity to execute the workload on a single server. In contrast, LinuxONE’s ability to encrypt in bulk provides 18 times performance at only 1/20th the cost of a similarly x86 based solution.

Logical Partitions

IBM’s LinuxONE is designed for Common Criteria Evaluation Assurance Level 5+ (EAL5+) certification. This level is achieved through the use of logical partitions (LPARs).  An application running in one operating system image in an LPAR cannot access an application running on a different operating system image on another LPAR.  LPAR’s enable separation of blockchain application elements, database components, into strictly defined, isolated security zones. LinuxONE’s isolation provides for near ‘air-gap’ separation of appliance environments on a single server footprint (air gapped servers still have network connections between that could be breached). Compared to x86, LinuxONE reduces the threat surface by nearly 92% according to Solitaire International.

True Random Number Generator

Cryptographic algorithms rely on random seeds. Pseudorandom number generators commonly found on x86 systems have been successfully brute force attacked. True Random Number Generators (TRNGs) stop this class of attacks. IBM LinuxONE systems incorporate built-in TRNGs into the processor cores and IBM Crypto Express 6S. The TRNGs are designed to support the most demanding cryptographic security requirements.

Firmware Tamper Protection

IBM LinuxONE systems comply with National Standards and Technology (NIST) Special Publication 800-147B and raise an alarm if anyone attempts to tamper with firmware booting. Dedicated Hardware Management Consoles support multi-factor authentication (per IETF RFC 6238) and separate role-based security profiles. These dedicated HMC’s prevent operators from exceeding defined responsibilities.

Time Source Security

Accurate timekeeping is a critical element in maintaining security. Timekeeping is vital for checking and rejecting expired credentials and certificates. Some blockchain attacks have taken advantage of poor clock synchronization. IBM LinuxONE systems incorporate high-quality system clocks in redundant form, with continuous, automatic error checking. This redundancy allows for the isolation and removal of any failed clock component before it produces an inaccurate time. Even the highest quality clocks require synchronization with the world’s most trusted time bureaus. LinuxONE systems are equipped with Server Time Protocol (STP), a system feature that makes regular inquiries with trusted time bureaus. STP can also check the validity of time information using symmetric key and Autokey authentication per IETF RFCs 1305 and 5906, to assure that time messages are not falsified or altered. STP also helps the system handle the periodic leap seconds that global time standards mandate.

Hardware Security Module

Hardware Security Modules (HSM’s) are dedicated components designed to hold, protect, and secure master crypto keys. HSMs are critical for blockchain cryptographic operations including encryption, access control and data protection.  However, not all HSMs are built the same and some are vulnerable to exploits.  Without HSM’s, encryption keys would be typically held in x86 main system memory, vulnerable if an attacker breaches application security. LinuxONE’s Crypto Express 6S features a FIPS 140-2 Level 4-compliant tamper-proof secure key infrastructure for storing private keys and certificates.  This classification is highest available on the market today. There are significant differences between Level 3 and Level 4 certified HSM’s. While Level 3 HSM’s respond to tampering with their doors or covers, they can still be attacked through other paths. It’s like securing your doors and windows, only to have an intruder drill through the floor below. In banking, should a Level 3 HSM be breached, new keys would be generated and transactions reversed. However, reversing transactions is not possible on blockchains.

IBM HSM’s support a concept known as “cryptographic domains”. Secure keys generated and wrapped with the master key in one domain are not usable by another domain using a different master key.   Crypto Express 6S supports over 300 cryptographic algorithms including Elliptic Curve Digital Signature Algorithms.  It detects any physical intrusion such as drilling, probing, chemical attacks, temperature attacks, and offers protection from side channel attacks and x-ray proton beaming of ASIC and FPGA to trigger bit flips.  Should Crypto Express 6S detect such tampering, it will self-destruct before giving up key material. It also provides double asymmetric encryption acceleration for TLS handshakes.

IBM’s CryptoExpress 6S also employs multiple key generation engines that feature parity checks and error code correcting that is unique to this HSM.  All other HSMs risk key generation issues should they encounter a CPU fault. LinuxONE is able to process over 350,000 encrypted transactions per second by wrapping the Linux clear key with the HSM’s trusted key into a “protected key” using hardware-based cryptographic acceleration. Thus Protected Key provides the FIPS 140-2 level 4 trusted key guarantee at scale. Protected keys are stored outside the Linux kernel preventing raw key material from being visible to the operating systems and applications. Protected Key guarantees an additional level of security not available on any other platform

Secure Service Container: 16 TB of protected memory.

Working with Crypto Express 6S is LinuxONE’s Secure Service Container (SSC). It provides an FIPS-197 AES 256 bit encrypted, isolated, and trusted runtime for deploying the blockchain infrastructure. SSC completely conceals Fabric’s data and prevents non-authorized access—both from inside and outside threats. IBM’s production blockchain networks all run in LinuxONE SSCs as part of the IBM Blockchain Platform Enterprise Plan (below illustration). The SSC provides the base infrastructure for integrating the operating system, middleware and software components into an appliance, which works autonomously and provides core services and infrastructure focusing on consumability and security.

No alt text provided for this image


A malicious actor who has Docker group level access rights can perform a Docker container permission exploit to obtain root level access to the Linux kernel and obtain access to smart contracts held in another container on the same virtual machine ( Should a malicious actor obtain root access rights to an unprotected x86 Linux kernel, they could perform a memory core dump which would expose the private key seed material and perform smart contract attacks.  Should the same actor perform the same core dump with the blockchain application running in the LinuxONE SSC, they would obtain nothing, just useless encrypted data. The LinuxONE SSC appliance is secured by a trusted firmware boot sequence before the software deployment. The appliance is made tamper-resistant during the installation and runtime. After the appliance is built, it can be accessed only by remote APIs. System administrators cannot access the memory or processor state; there is no direct host or OS level interaction.

The LinuxONE SSC creates a trusted execution environment (TEE) for running containerized workloads and databases, such as blockchain, in up to 16TB of memory.  By using a TEE, one does not have to trust the host system which runs the blockchain node. Intel’s Software Guard Extensions (SGX) also establishes TEE enclaves, however, they come with several important constraints that make them suboptimal for securing blockchains.  Intel’s SGX has a memory limit of only 128 MB of which roughly 90 MB is usable. Due to SGX’s size limitation, developers must re-factor code to designate which parts will be secured/encrypted. With only a portion of the application secured by SGX, neighbouring applications can attack the SGX application with a side-channel attack, which has been done successfully.  SGX enclaves are protected memory areas located inside an x86 CPU. When the enclave stops, restarts, or crashes, the internal state is lost and cannot be recovered. SGX uses a data sealing approach that encrypts and authenticates data before leaving the enclave where it is stored externally. When the enclave restarts, it loads and decrypts the data preserving data confidentiality. However, it does not prevent rollback attacks whereby properly sealed data operates on blockchain state data that could be stale.

Rollback attacks on stateful applications running in TEEs pose serious risks, unless the state continuity of an application is ensured. For example, if a malicious blockchain node is able to influence the execution order of smart contract transactions in an SGX enclave, it could learn information about those transactions without being able to decrypt them. The node running a sealed-bid auction blockchain application could execute a bid evaluation transaction multiple times and then reset the enclave again afterwards every time a new bid has been committed to the ledger. Thus, the malicious node could make inferences about other bids violating the integrity of the system without having to break encryption in the SGX enclave. The SGX architecture is also vulnerable to state continuity exploits. Since the malicious node has access to the key value store database, that cannot be accommodated by the SGX enclave due to size limitations, the node could intercept, modify, reorder, discard, or replay smart contract operations by feeding the enclave manipulated state inputs or even drop messages or halt the enclave and prevent consensus altogether. State continuity and rollback attacks can be prevented if the state input to the smart contract enclave always corresponds to the unique, committed blockchain state. One way to guarantee the desired state continuity would be to run the whole blockchain node, especially its protocol logic and the state maintenance, within the enclave such as possible running then entire node in LinuxONE’s SSC.

Ensuring a Trusted Build – Secure CI/CD

Without a trusted DevOps CI/CD process, someone could write malicious code that, once executed, would have granted command-level access to the hosting server and deploy backdoored smart contracts.  In IBM’s Secure CI/CD process, the build process runs within the LinuxONE SSC appliance that builds the image, generates a key, signs it, and pushes it to a Docker Hub repository using the Docker Content Trust framework. Docker Content Trust is commonly used to make sure that a container binary image is from an original developer, not from a malicious developer.  If signing the key were to be stolen, a malicious person could replace the image on Docker Hub with another image with malicious code.  The LinuxONE SSC secure build protects the signing key and source code manifest that ensures we can trust the binary image.

Third party auditors perform code reviews and testing prior to deployment to ensure the integrity of the build. Auditors review the manifest that describes all the ingredients to produce the image binary, such as the source code, and a base image, and so on. The manifest is signed inside the LinuxONE CICD Service to make sure that it is not replaced with something else.  Each auditor checks the manifest to establish confidence that no malicious code exists in the final image. The attested image is deployed into the LinuxONE SSC runtime environment by build endorsement service via an endorsement policy, like a multi-signature or multi-factor authentication. LinuxONE SSC only runs a Docker container if the image is trusted in the Docker content trust framework and if the manifest meets an endorsement policy for this particular application.

The LinuxONE’s unique security capabilities of pervasive encryption, FIPS 140-2 Level 4 HSM, and the SSC make it not only suitable, but the preferred platform for permissioned blockchain solutions.

Editors Note

This post was penned by Peter Demeo who is a WW subject matter expert on the intersection of Blockchain, Digital Assets and Secure Infrastructure. Peter is literally re-defiing how the industry thinks about the deployment of secure Blockchain networks and his approach to securing digital assets will provide the trust so needed as the world of finance reimagines itself. I highly recommend you follow Peter on Twitter