Post

Exploring the cpio-newcx format

Understanding a non-mainlined archive format

Posted Updated
By Tom Ice
23 min read
Exploring the cpio-newcx format

I come across and work with many different formats and file types throughout my career. Some of them I deep dive because they’re unique and require further understanding in order to work with them. Others are simply interesting because you rarely see them outside specific applications. cpio’s newcx happens to be both of those combined. It was recently brought up in a handful of discussions at work, so I figured I’d do a bit of a deep dive on the format, explain what it is, and why someone might even want to use a non-standard archive format for their initramfs.

Booting Linux

Before we can begin talking about the cpio format, or the newcx variant of it, we need to start around the beginning to level set our understanding of what is going on under the hood of the system during boot.

When a system powers on, the firmware (BIOS or UEFI) initializes the hardware and loads our bootloader (GRUB, U-Boot, SYSLINUX). From there, the bootloader then loads the Linux kernel and, when configured, an initial RAM disk into memory before transferring control to the kernel.

After the bootloader transfers control, the kernel begins its own initialization. If an initial RAM image was provided, the kernel either mounts an older initrd as a RAM-backed root filesystem, or it extracts an initramfs archive into rootfs. Once the kernel executes the image’s initial userspace process, we enter the early userspace stage of the boot process.

That process performs the setup necessary to locate and mount the system’s real root filesystem, if one exists. I say if one exists because on most distributions, /init uses a temporary environment to discover and mount another root filesystem before handing control over to the normal init system from that filesystem. However, it is possible to package all of this together and continue running directly from the in-memory root filesystem, which will become more relevant later on.

Initrd vs Initramfs

There are two main approaches that you will come across - an older style initrd and a newer style initramfs. An old Linux initrd really was a ramdisk-oriented model. You could have something like an ext2 filesystem image, put that into memory, and have the kernel interact with it as a block-backed filesystem.

An initramfs, however, takes a bit of a different approach. Instead of it being a block device, it’s more of a serialized filesystem. Linux receives an archive and unpacks it into rootfs which is backed by ramfs or tmpfs. If the resulting root filesystem contains /init, the kernel executes it as PID 1. There is no ext-style superblock inside the normal initramfs, and there is no block device that needs to be mounted just to get at the files.

The kernel documentation goes into more detail on the exact differences between an initrd, initramfs, the concept of pivot_root(), and so on. Definitely check that out for a better understanding of what is going on.

Archive Format

As mentioned before, an initramfs is more like a serialized filesystem. Because of that, we need some way to package up everything, so an archive format was necessary to handle this. Long ago, deep within the kernel documentation, there was once a discussion of possibly using tar. In fact, you can read more about that in the previous section’s link. However, that was rejected in favor of a format called cpio, named for the phrase “copy in and out,” because it is extremely simple to generate and parse. This was created back in the 1970s by Bell Labs as a tape archiving program. Despite its main usage being tape archival, it eventually found massive adoption across a multitude of technologies such as the Linux kernel and RPM Package Manager.

Given how simple and straightforward it is to use, Linux was able to create its own usr/gen_init_cpio.c on the creation side and init/initramfs.c on the extraction side, so the kernel itself never needed a full general-purpose archive implementation in early boot.

newc

There have been a handful of cpio formats over the years given how long it has been around. The one relevant to an initramfs is usually called “new ASCII, SVR4 newc,” or simply newc. It’s actually a bit of a confusing name if you never looked up the history. The “new ASCII” name distinguishes it from an older ASCII-based cpio format commonly called odc. The older format represented numeric fields as fixed-width octal strings, while this “new ASCII” newc format redesigned the header around fixed-width hexadecimal ASCII fields and expanded some of the metadata it could represent. SVR4 refers to Unix’s System V Release 4.0 which is when this version first appeared.

Definitely check out the cpio format documentation. Those manpages, combined with the kernel docs on buffer formats are what we will be referencing this from here on our to help build our parser.

The archive contains a header record with metadata followed by the path and the file’s data. The archive stores a fixed-size header containing filesystem metadata such as inode number, mode, UID, GID, link count, timestamps, file size, and device numbers, followed by the pathname and file data. The end of the archive in a conventional cpio is terminated by a special entry named TRAILER!!!. This also has some semantics around tracking hard-links which we will talk about a little later.

The newc format uses 8-byte hexadecimal fields for all numbers and separates device numbers into separate fields for major and minor numbers. It looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

struct cpio_newc_header {
    char c_magic[6];
    char c_ino[8];
    char c_mode[8];
    char c_uid[8];
    char c_gid[8];
    char c_nlink[8];
    char c_mtime[8];
    char c_filesize[8];
    char c_devmajor[8];
    char c_devminor[8];
    char c_rdevmajor[8];
    char c_rdevminor[8];
    char c_namesize[8];
    char c_check[8];
};

You can always tell what version of newc you are dealing with by parsing the c_magic field in the header. It should result in the string 070701 for a standard newc format without the new CRC format and 070702 for a newc with the new CRC format. The CRC format changes the c_check field to be the sum of all the bytes in the file data which is computed by treating all bytes as unsigned values and using unsigned arithmetic. Other than that, the two are nearly identical formats, so it’s very easy to parse both versions.

The current kernel documentation describes the basic idea as such:

1
2
3
4
5
6
7

header
filename
NUL
padding
data
padding
next header

Or, more visually:

1
2
3
4
5
6
7
8
9
10
11
12
13

+--------------------------+
| 110-byte newc header     |
+--------------------------+
| filename                 |
| '\0'                     |
+--------------------------+
| 0-3 bytes of padding     |
+--------------------------+  <- 4-byte aligned
| file data                |
| ...                      |
+--------------------------+
| 0-3 bytes of padding     |
+--------------------------+  <- next cpio header

Everything is a flat, linear sequence of data. A directory is just an entry whose c_mode says it is a directory, a symlink is an entry whose data contains its target, a file is another record with some bytes after its header and so on. This typically continues until we hit the TRAILER!!! end of archive entry.

newc Shortcomings

The simplicity came with limitations, however. The obvious one for the newcx story is that newc has nowhere to store extended attributes that security systems such as SELinux use to more strictly enforce operations within the system.

Extended attributes can contain things such as:

1
2
3
4

security.selinux
security.ima
security.capability
user.*

The filesystem Linux is unpacking the initramfs into can support xattrs, but the archive has to somehow transport those xattrs to the filesystem in the first place. That gap was the original motivation for exploring a new format.

Birth of newcx

In January 2015, Mimi Zohar posted an RFC patch series titled:

[RFC][PATCH 0/9] extend initramfs archive format to support xattrs

The motivation was Linux integrity/security functionality. The root filesystem could be backed by tmpfs and therefore support extended attributes, but standard cpio had no representation for transporting them. The RFC explicitly considered three approaches:

  1. Put the xattrs into a separate manifest inside the initramfs
  2. Extend cpio itself
  3. Add tar support

The proposed patches took the second route and modified both usr/gen_init_cpio.c and init/initramfs.c. The RFC noted that security.ima had to be applied after the file data had been written, requiring the initramfs extractor’s state machine to distinguish between receiving the xattrs and actually setting them, so the problem was indeed a legitimate one.

The original discussion also immediately raised a second question - if we are assigning a new cpio magic value anyway, should we fix some of newc’s other limitations?

There are two other aging problems in newc that were brought up in the peer review. First, c_filesize is eight hexadecimal characters. Eight hex digits give us 32 bits to work with. Earlier, I pointed out usr/gen_init_cpio.c. If we look at the code, we can see that it still explicitly rejects files larger than 0xffffffff. So there is a clear filesize limit we will have to deal with across the system.

Second, c_mtime is also eight hexadecimal characters. The current kernel extractor still parses it as a 32-bit value and even contains this comment:

mtime = be32_to_cpu(header[5]); /* breaks in y2106 */

None of these limitations normally prevent a tiny embedded initramfs from booting today, but if you are already going to introduce an incompatible format to add xattrs, some may think it may be reasonable to ask whether you should fix the other known limitations at the same time.

Rob Landley pointed out the 32-bit timestamp problem and the lack of a strong external specification controlling future cpio evolution. That led to discussion of widening timestamps and file sizes as part of the same format change. Eventually, it settled on an entirely new magic of 070703, and we saw the birth of the first version of newcx.

Some time passed, and in 2018, the work resurfaced as a much larger 15-patch series. The January 2018 v2 proposal formally described newcx as an extended version of newc with:

  • Extended attributes
  • File sizes larger than 4 GiB
  • Larger modification-time field
  • Removal of the old checksum field

In v2, the timestamp was represented as a 16-character hexadecimal value containing microseconds.

The header was therefore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

struct cpio_newc_header {
    char c_magic[6];
    char c_ino[8];
    char c_mode[8];
    char c_uid[8];
    char c_gid[8];
    char c_nlink[8];
    char c_mtime[16];
    char c_filesize[16];
    char c_devmajor[8];
    char c_devminor[8];
    char c_rdevmajor[8];
    char c_rdevminor[8];
    char c_namesize[8];
    char c_xattrs_size[8];
};

Then, only a few weeks later, v3 changed it.

The February 2018 v3 proposal retained magic 070703, but changed the timestamp representation to:

1
2

c_mtime       16     seconds
c_mtime_nsec   8     nanoseconds

This resulted in a 134 byte header.

That gives us this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

struct cpio_newc_header {
    char c_magic[6];
    char c_ino[8];
    char c_mode[8];
    char c_uid[8];
    char c_gid[8];
    char c_nlink[8];
    char c_mtime[16];
    char c_mtime_nsec[8];
    char c_filesize[16];
    char c_devmajor[8];
    char c_devminor[8];
    char c_rdevmajor[8];
    char c_rdevminor[8];
    char c_namesize[8];
    char c_xattrs_size[8];
};

The history is actually important for anybody writing a parser for newcx. 070703 by itself does not tell us whether we have the v1, v2, or v3 layout. All of the proposals used the same magic because they were all basically iterations of the same overall patch set over time. You will need to verify the size of the layout in order to determine which version we are dealing with.

Parsing: The Plan

Since this is effectively a stream of bytes in an archive format, it is surprisingly straightforward to parse this with basically any programming language. So let’s try it! It’s honestly not that bad once you start to get into it.

So let’s say we want to inspect a cpio-newcx archive. Let’s say we want to be able to list files that are inside of it, display statistics about the files, view file contents, see the extended attributes, and write out the hex value of the file. So that means we basically want the standard Linux operations:

1
2
3
4
5

ls
stat
cat
xattrs
hex

I want to do all of that without ever mounting it, unpacking it, or doing anything with it outside of simply reading the file and displaying it to stdout. So for example, operations like:

1
2
3
4
5
6

./newcx-inspect ls initramfs.cpio-newcx
./newcx-inspect ls initramfs.cpio-newcx etc
./newcx-inspect stat initramfs.cpio-newcx /etc/shadow
./newcx-inspect cat initramfs.cpio-newcx /etc/shadow
./newcx-inspect xattrs initramfs.cpio-newcx /etc/shadow
./newcx-inspect hex initramfs.cpio-newcx /init 256

It’s easy to support all versions of newc/newcx, but for the purpose of this, we will keep it to just the latest version 3 from the 15 patch series.

I also want it to be relatively flexible so you can type things such as:

1
2
3

/etc/shadow
etc/shadow
./etc/shadow

Any of those should display a match, so we will need to do some normalizing of paths that the user might enter.

cpio can represent multiple hard-linked filenames using the same tuple. Meaning multiple files can contain the device major, device minor, and inode. For our purposes, only one member needs to actually carry the data. The kernel docs explicitly describes zero-sized members referring to a data-carrying hard-link member elsewhere in the archive. It is that same TRAILER!!! that resets that hard-link namespace.

For newcx v3, our entry layout becomes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

header offset
      |
      v
+----------------------+ 134 bytes
| newcx header         |
+----------------------+
| filename             | namesize bytes
| NUL                  |
+----------------------+
| alignment            |
+----------------------+ <- xattr_offset
| xattr entries        | xattrs_size bytes
+----------------------+
| alignment            |
+----------------------+ <- data_offset
| file data            | filesize bytes
+----------------------+
| alignment            |
+----------------------+ <- next_offset

So once the header fields are decoded, parsing an entry is mostly offset calculations and arithmetic.

Parsing: The Code

We got a rough plan of attack and an idea, so let’s see if we can use Python to try to parse this. We’ll start by filling out the skeleton functions first and expand from there. We want something that looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

load_archive()
      |
      v
iter_entries()
      |
      v
parse_entry()
      |
      +--> parse_hex()
      |
      +--> align4()
      |
      +--> pathname()
      |
      v
find_entry("/etc/foo")
      |
      v
data_entry()
      |
      v
[data_offset:data_offset + filesize]
      |
      v
stdout

Before getting into each function, there are two globals that define the format we’re working with:

1
2

MAGIC = b"070703"
HEADER_SIZE = 134

In this program, we are only supporting version 3 of newcx, so anything that does not have that magic and header size (remember, that’s what makes it v3) is rejected. You can easily expand on this later on if you wanted to add additional formats and expand this parser into something a bit more polished.

One more thing is that the parsed metadata for each file is stored in an Entry dataclass. This mirrors the fields that come directly from the newcx header, and it allows us to keep calculated offsets such as xattr_offset, data_offset, and next_offset to save us from recalculating them later when implementing things like cat.

load_archive()

The first step is simply getting the archive into memory:

1
2
3
4
5
6
7
8

def load_archive(path: str) -> bytes:
    if path == "-":
        return sys.stdin.buffer.read()

    try:
        return Path(path).read_bytes()
    except OSError as exc:
        raise ArchiveError(f"{path}: {exc}") from exc

We are assuming no compression in this logic. If the archive is compressed, you will need to use external tools to handle that and pipe the result in:

1
2

zstd -dc initramfs.cpio-newcx.zst |
    ./newcx-inspect stat - /etc/foo

iter_entries()

iter_entries() walks through the archive one record at a time:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def iter_entries(data: bytes) -> Iterator[Entry]:
    offset = 0
    segment = 0

    while offset < len(data):
        while offset < len(data) and data[offset] == 0:
            offset += 1

        if offset == len(data):
            return

        entry = parse_entry(data, offset, segment)

        if entry.name == "TRAILER!!!":
            segment += 1
        else:
            yield entry

        offset = entry.next_offset

Here, we skip any zero padding, parse the record at the current offset, and then jump directly to the next record. TRAILER!!! is handled separately because it marks the end of a cpio segment and resets the namespace used for hard links. The actual work is done in the next function by parse_entry().

parse_entry()

We know every newcx v3 header is 134 bytes, so the first thing we do is grab that header and verify its magic:

1
2
3
4
5
6
7
8
9
10
11

header_end = offset + HEADER_SIZE

if header_end > len(data):
    raise ArchiveError("truncated newcx header")

header = data[offset:header_end]

if header[: len(MAGIC)] != MAGIC:
    raise ArchiveError(
        f"expected 070703 at archive offset {offset}"
    )

The fields inside the header are sequential, so rather than manually calculating an absolute offset for every field, we keep our current position in the header and advance it as each field is read:

1
2
3
4
5
6
7
8
9

position = len(MAGIC)

def read_hex(width: int) -> int:
    nonlocal position

    field = header[position : position + width]
    position += width

    return parse_hex(field)

The header then maps almost directly to the newcx structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

ino = read_hex(8)
mode = read_hex(8)
uid = read_hex(8)
gid = read_hex(8)
nlink = read_hex(8)

mtime_sec = read_hex(16)
mtime_nsec = read_hex(8)

filesize = read_hex(16)

dev_major = read_hex(8)
dev_minor = read_hex(8)
rdev_major = read_hex(8)
rdev_minor = read_hex(8)

namesize = read_hex(8)
xattrs_size = read_hex(8)

The nature fo the format makes it very easy to compare our impl to the header we looked at earlier.

parse_hex()

There’s a small parse_hex() helper function that handles the conversion from ASCII hex directly to an integer:

1
2
3
4
5
6
7

def parse_hex(field: bytes) -> int:
    try:
        return int(field, 16)
    except ValueError as exc:
        raise ArchiveError(
            f"invalid hexadecimal field {field!r}"
        ) from exc

Pathname and Alignment

Once the fixed header has been parsed, we can find the variable-length portions of the entry. The pathname immediately follows the header:

1
2

name_offset = header_end
name_end = name_offset + namesize

After some basic validation, we remove the terminating NUL and decode it:

1
2
3
4
5
6

raw_name = data[name_offset : name_end - 1]

name = raw_name.decode(
    "utf-8",
    errors="surrogateescape",
)

From there, the rest is mostly alignment:

1
2
3

xattr_offset = align4(name_end)
data_offset  = align4(xattr_offset + xattrs_size)
next_offset  = align4(data_offset  + filesize)

align4() just rounds an offset up to the next four-byte boundary:

1
2

def align4(offset: int) -> int:
    return (offset + 3) & ~3

Those three calculated offsets are really the important result of parsing the entry. Once we know them, we know where its xattrs are, where its file contents are, and where the next newcx record begins.

iter_xattrs()

Extended attributes are stored as another small sequence of records inside the entry’s xattr region:

1
2
3
4
5
6
7
8
9
10

def iter_xattrs(
    data: bytes,
    entry: Entry,
) -> Iterator[tuple[str, bytes]]:
    offset = entry.xattr_offset
    end = offset + entry.xattrs_size

    while offset < end:
        size = parse_hex(data[offset : offset + 8])
        record_end = offset + size

Each record begins with its eight-byte hex size. We then find the NUL separating the xattr name from its value:

1
2
3
4
5
6
7
8
9

name_start = offset + 8
nul = data.find(b"\0", name_start, record_end)

name = data[name_start:nul].decode(
    "utf-8",
    errors="surrogateescape",
)

yield name, data[nul + 1 : record_end]

The value remains raw bytes because xattrs are not guaranteed to contain text. This lets us later display something like security.selinux = 'system_u:object_r:shadow_t:s0\x00' while still being able to handle binary attributes.

find_entry()

Now that we can walk the archive, finding a specific file is just a linear search:

1
2
3
4
5
6
7
8

def find_entry(data: bytes, path: str) -> Entry:
    wanted = normalize_path(path)

    for entry in iter_entries(data):
        if normalize_path(entry.name) == wanted:
            return entry

    raise ArchiveError(f"{path}: not found")

normalize_path() lets all of these resolve to the same entry:

1
2
3

/etc/shadow
etc/shadow
./etc/shadow

Fairly straight-forward and simple:

1
2
3
4
5

def normalize_path(path: str) -> str:
    path = path.lstrip("/")
    while path.startswith("./"):
        path = path[2:]
    return path

data_entry()

Hard links require one additional check before reading file contents. A cpio entry can describe a regular file with multiple links but contain no file data itself. Another entry with the same device and inode tuple can carry the actual contents. data_entry() handles that for us:

1
2
3
4
5
6
7

def data_entry(data: bytes, entry: Entry) -> Entry:
    if (
        not stat.S_ISREG(entry.mode)
        or entry.nlink <= 1
        or entry.filesize != 0
    ):
        return entry

If it looks like a possible hard-link entry, we search the same cpio segment for the data-bearing member:

1
2
3
4
5
6
7
8
9
10

for candidate in iter_entries(data):
    if (
        candidate.segment == entry.segment
        and candidate.ino == entry.ino
        and candidate.dev_major == entry.dev_major
        and candidate.dev_minor == entry.dev_minor
        and stat.S_ISREG(candidate.mode)
        and candidate.filesize > 0
    ):
        return candidate

For normal files, this function simply returns the entry we already had.

Execution

And that’s it! Admittedly, it was a lot of explaining for every function, but it honestly isn’t that terrible of a parser to write. You can now use this to inspect a cpio-newcx format like we talked about earlier.

1
2
3
4
5
6

./newcx-inspect ls initramfs.cpio-newcx # lists everything in the archive
./newcx-inspect ls initramfs.cpio-newcx etc # only lists what's in etc
./newcx-inspect stat initramfs.cpio-newcx /etc/foo # runs "stat" on /etc/foo
./newcx-inspect cat initramfs.cpio-newcx /etc/foo # runs "cat" on /etc/foo
./newcx-inspect xattrs initramfs.cpio-newcx /etc/foo # runs "xattrs" on /etc/foo
./newcx-inspect hex initramfs.cpio-newcx /init 256 # display hex val of a given file/folder

Use Cases

So why would you ever want to use the non-standard cpio-newcx format in the first place? The biggest reason is because of the ability for it to carry extended file attributes with it and preserve them. Outside of that, there is very little reason to actually use this format. If you do not need to preserve xattrs or any of the widened fields, you are better off using the cpio-newc format instead to maintain more general compatibility with different parsers out there. If you do care about those sorts of things, however, then these patches are one way to achieve that. Alternatively, you can look into implementing tar/pax, or using an xattr manifest.

Upstreaming

One might be wondering why this has not been upstreamed yet. The patches exist, and it clearly works. It should be as simple as merging it in. Instead, the format kinda quietly faded away and was never really revived.

The discussion exposed a disagreement about what problem should actually be solved. H. Peter Anvin’s objection was essentially that if Linux was going to create a new, incompatible format anyway, it should not blindly preserve several awkward properties inherited from cpio. He specifically objected to fixed-width ASCII numbers, alignment-sensitive parsing, and the magic TRAILER!!! filename being used as an in-band end marker. He suggested reconsidering POSIX tar/pax, which already had a broader metadata model and existing userspace support.

There is a reasonable argument there. newcx is compatible with the design of newc, but it is not wire-compatible with a newc parser. If you have to teach things like the kernel extractor, kernel generator, cpio, dracut, etc. about this new magic value anyway, then maybe that is an opportunity to stop perpetuating the things nobody particularly likes about old cpio.

The counterargument was almost the exact opposite. Mimi Zohar explained that tar had already been considered and that, during discussion at the 2014 Kernel Summit, Al Viro had recommended extending cpio because tar was unnecessarily complicated for this kernel use case. That was consistent with the original reason cpio had been chosen for initramfs in the first place - keeping the kernel-side implementation tiny and easy to understand.

With no real consensus, the patch set stalled and was simply left with no real course of action.

Wrapping Up

As of August 2026, current mainline init/initramfs.c still accepts:

1
2

070701
070702

It rejects an unknown magic, so there is no 070703 magic available for it to try to parse. Current usr/gen_init_cpio.c likewise continues to only generate 070701 or 070702.

It is a bit of a unique use case to ever need to use it, but there are legitimate, security-oriented reasons for wanting to pull in this patch set and work with a non-standard format that deviates from the mainline kernel.

If you ever happen to run across one of these, give my parser a try. It is very basic and can easily be extended for extra use cases, but it might just get you out of a pinch when trying to inspect a filesystem and verifying what is inside.

Further Reading

Some further reading that may be useful:

Happy parsing!

Programming, Linux, Embedded Linux, Python
linux embedded linux python
This post is licensed under CC BY 4.0 by the author.