Endianness and IBM LinuxONE

Are you concerned about LinuxONE being “big endian”? Do you understand what “big endian” means? This article attempts to clarify what “endianness” means and why you shouldn’t be overly concerned about it.

History

In 1725, Jonathan Swift unintentionally created a useful term in his work of fiction known commonly as Gulliver’s Travels [1]. In the book, Gulliver visited the island empire of Lilliput, where breaking an egg from the big end had been outlawed by the grandfather of the then-current emperor. The emperor from two generations earlier had cut his finger while breaking an egg from the big end after which he made it a criminal offense for anyone to break an egg from the big end. Most people complied but there were, of course, people who refused to change, who revolted or emigrated and continued their practice. The emperor referred to these people as “Big Endians”. War almost broke out over the Big Endians wanting to stay Big Endians.

More than 250 years later, the Internet was being formed. Different computer systems and networks were getting connected together. There was a need to share or pass along information, but there was a problem: different systems stored and shared information in different sequences. 

Some systems stored information using the most-significant byte of a word (the one having the largest value) at the smallest memory address and the least significant byte (the one having the smallest value) at the largest memory address. Mozilla.org makes the analogy of the European date representation of 7 June 2021. Others stored the least-significant byte at the smallest address, Mozilla’s analogy is the International Standards Organization (ISO) date format of 2021-June-7. People on both sides thought theirs was the best way and that the others should change. A war was brewing about the bit order for representing data. It was almost like Lilliput. 

In his 1980 paper “On Holy Wars and a Plea for Peace”, computer networking scientist Danny Cohen pleaded with the community to simply recognize and accept the bit-order differences. He reused Swift’s terms Big-Endian (BE) and added the implied Little-Endian (LE) to indicate which way the data was represented (BE would be 2021-June-7, and LE would be 7 June 2021). His plea was that both should be acceptable, and that systems be made to accommodate endianness. In the end, a compromise was made to use a mix of endianness on the Internet, in both the computer systems and the networking systems.

Today

Forty years after Cohen’s paper, there is mostly harmony and acceptance that we exist in a mix-ed endian environment that isn’t likely to change soon. IT companies went through their endianness issues years ago, but there’s still the occasional comment that a difference in endianness could bring business to a halt. 

Those same people don’t tend to mention that MacOS is BE, that Linux servers running using NFS or any RPC-based applications are using BE, even on an LE computer. Dr. Ulrich Weigand, IBM Distinguished Engineer told me, “In addition to the core network packet format and most higher-level network protocols (that were already mentioned), the on-disk metadata for some file systems including XFS and GFS2 is big-endian.  Also, the Java native byte order is big-endian, i.e., Java binary files are encoded in big-endian. But I think the most central point to make should be that for an end user, endianness should be completely transparent.  It is at most application developers that *may* possibly be affected.”

Migrating databases to LinuxONE

IBM LinuxONE is a BE system, as are a few others (IBM AIX, IBM Z, HP-UX, Oracle SPARC Solaris, Apple MacOS). Most other servers in a datacenter tend to be LE. Communicating between the two endians is something I was doing 40 years ago. Lots of people smarter than I am (because they made money from it whereas I didn’t) have written tools over the years to handle this.

Let’s say you wanted to move data from a set of x86 servers running IBM Db2 for Linux/UNIX/Windows (LUW) to Db2 LUW on LinuxONE. You could use IBM InfoSphere, IBM MQ, IBM Integration Bus, IBM Cloud Pak for Integration, IBM Cloud Pak for Data, third-party tools or open-source tools to perform the data migration. Or grab a Comp Sci student and have them write it for you. Make it interesting for them by suggesting they compare doing it in different programming languages.

What if it’s Oracle Database? Oracle owns some BE systems and has supported BE and LE systems for years, so they offer several options. I asked IBM Technical Specialist and Certified Oracle DBA David Simpson what he uses to help customers migrate to LinuxONE. He said he’s used Oracle GoldGate or Oracle Data Pump tools, or in some cases he’s done it manually by using Transportable Database, Transportable Tablespace, or RMAN. IBM also has some options for migrating an Oracle Database environment to LinuxONE: IBM InfoSphere Change Data Capture and/or the IBM Migration Factory tool, Xenobridge.

Migrating applications to on LinuxONE

Transporting data from an LE system to a BE system like LinuxONE is no big deal, but What about migrating applications to LinuxONE? I asked IBM Global Technical Sales Leader for LinuxONE, Dr. Andrea Corbelli. He told me “If talking about Oracle WebLogic, after solving the usage of the correct and updated WebLogic version and libraries, then Java is Java, and doesn’t suffer from endianness issues. When you have ISV applications, endianness is solved when the app is ported to LinuxONE. When you have custom-developed apps, and you need to do the porting, that is where endianness hits.”

I asked Elton de Sousa, a cloud architect at IBM, “Is endianness ever an issue when moving containerized workloads to the LinuxONE architecture?” He responded, “Very rarely and mostly for C/C++ workloads or libraries that are trying to do clever things to improve performance. It was a problem for the runtimes too (Node.js/Java etc.), but once we got the runtimes to understand endianness, everything on the top just worked.” 

Notes

  • Last updated March 12, 2021: https://support.oracle.com/knowledge/Oracle%20Database%20Products/371556_1.html 

    Starting with Oracle Database 10g, you can transport tablespaces across platforms. In this note there is a step-by-step guide about how to do it with ASM datafiles and with OS filesystem datafiles.

    If your goal is to migrate a database to different endian platform, the following high-level steps describe how to migrate a database to a new platform using transportable tablespace:
  1. Create a new, empty database on the destination platform.
    1. Import objects required for transport operations from the source database into the destination database.
    1. Export transportable metadata for all user tablespaces from the source database.
    1. Transfer data files for user tablespaces to the destination system.
    1. Use RMAN to convert the data files to the endian format of the destination system.
    1. Import transportable metadata for all user tablespaces into the destination database.

You could also convert the datafiles at the source platform and, once converted, transfer them to the destination platform.

[1] Gulliver’s Travels on https://www.gutenberg.org/files/829/829-h/829-h.htm


One thought on “Endianness and IBM LinuxONE

  1. Modern macOS isn’t big endian in any comprehensive way, Although there are evidently some small aspects of macOS that retain big endian behaviors, macOS is now little endian. MacOS was big endian when it ran on the first two Macintosh processor architectures (the Motorola 68000 series, PowerPC) then switched to little endian for X86. (The first X86-based Macs had 32-bit processors, then they switched to 64-bit X86.) Apple is now, as I write this, mid-way through the Macintosh’s transition to its fourth processor architecture, to little endian ARM64. But these transitions also illustrate how “endianness” isn’t really much of an issue for most people.

    Another processor architecture that frequently operates in big endian mode, including for most Linux distributions, is the MIPS processor architecture. This architecture is popular in many embedded and “Internet of Things” products.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.