[Toybox] [PATCH] cpio: support reading concatenated cpio files.

[enh]

On Fri, Apr 16, 2021 at 11:44 AM Yi-yo Chiang <yochiang at google.com> wrote:

> I'm not sure what Elliot's goal is? I assume he's trying to extract a
> concatenated ramdisk, and I still see a problem in the current solution.
>

oops, i was going to say "i added you to the bug", but it looks like i
added you to the *other* cpio bug (which appears to be resolving itself as
"user error, we'll fix our script"). i've added you to
https://issuetracker.google.com/184732694 now, but non-googlers won't be
able to follow that link.

all i want to do is get my rug back^W^W^W^Wkeep someone's existing workflow
working.


> The buffer-format (
> https://www.kernel.org/doc/Documentation/early-userspace/buffer-format.txt)
> says:
>
>   initramfs  := ("\0" | cpio_archive | cpio_gzip_archive)*
>
> In other words, both `cat a.cpio b.cpio >merged.cpio` and `(cat a.cpio &&
> echo -n -e '\0\0\0' && cat b.cpio) >merged.cpio` are valid initramfs.
>

weird. do we know _why_ that's supported?


> btw gen_init_cpio.c also pads initramfs to 512-byte boundary (
> https://github.com/torvalds/linux/blob/6fbd6cf85a3be127454a1ad58525a3adcf8612ab/usr/gen_init_cpio.c#L97
> )
>
> If we're viewing buffer-format.txt as the "right" cpio spec, then I think
> we should implement this too. We should skip arbitrary extra NUL-bytes
> padded between cpio file frames
>

not that it really matters (because i think we're all agreed that "do what
the kernel does" makes more sense than "do what GNU/BSD think") but does
GNU cpio cope with that? (see "strace" below for why i wonder.)


> On Fri, Apr 16, 2021 at 2:27 PM Rob Landley <rob at landley.net> wrote:
>
>> On 4/16/21 12:18 AM, Rob Landley wrote:
>> > On 4/15/21 6:57 PM, Rob Landley wrote:
>> >> On 4/15/21 12:06 PM, enh wrote:
>> >>> do you already have the "do the right thing" patch ready, or should i
>> send that
>> >>> today?
>> >>
>> >> I'm good, I'll try to get it in tonight. The hard part is the test
>> suite, not
>> >> the implementation.
>> >
>> > Correction: the hard part is I don't seem to have actually implemented
>> hardlink
>> > support yet. It just records/extracts them as separate files.
>> >
>> > Right. I need to test this against the kernel initramfs plumbing, in
>> both
>> > directions...
>>
>>      if (!strcmp("TRAILER!!!", name)) {
>> +      readall(afd, toybuf, 348);
>>
>> Where did you get 348 from? (What is this change doing?)
>>
>> I did what seems like a minimal change to continue past TRAILER!!! but
>> error on
>> an empty archive, but I apparently don't understand what your change did?
>> (I'm
>> assuming it fixed the use case you were seeing...?)
>>
>
yeah, from strace i think GNU cpio avoids this misalignment by always
reading 512-byte blocks (rather than toybox's "let me just read the header
and then worry about anything else later").

all i was trying to do was "get back in sync after a TRAILER!!! entry", and
348 (by measurement) got me back to 512 after what toybox cpio had read,
for the examples i had. (i had expected "is this guaranteed to be right for
all possible valid cpio files?" to be the discussion we'd be having rather
than the "shouldn't we just do what the kernel does?" :-) )

(if i haven't said so already, assume i know nothing about cpio and have
never knowingly used it myself.)

interestingly, the current patch seems to break the AOSP build:

2021-04-16 18:56:56 - common.py - WARNING : Unable to get boot image build
props: Failed to run command '['toybox', 'cpio', '-F',
'/mnt/disks/build-disk/src/googleplex-android/sc-dev/out/soong/.temp/boot_xwvxvq26.img/uncompressed_ramdisk',
'-i']' (exit code 1):
cpio: bad header

i'll chase up the person using `toybox cpio` and move them over to the
hermetic prebuilt build toybox later. for now i'll try to reproduce this
locally and see what the trouble is...


> Rob
>> _______________________________________________
>> Toybox mailing list
>> Toybox at lists.landley.net
>> http://lists.landley.net/listinfo.cgi/toybox-landley.net
>>
>
>
> --
>
> Yi-yo Chiang
> Software Engineer
> yochiang at google.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20210416/2474b6ba/attachment.htm>