<div dir="ltr">funnily enough (or perhaps not, because yochiang knows a lot more about cpio than i do...) it looks like the AOSP build is broken because of trailing NULs...<div><br></div><div>here's the previous version of toybox at the end of the ramdisk in question:</div><div><br></div><div>utimensat(AT_FDCWD, "system/lib64/libstdc++.so", [{tv_sec=0, tv_nsec=0}, {tv_sec=0, tv_nsec=0}], AT_SYMLINK_NOFOLLOW) = 0<br>read(3, "070701000493fe000001ed0000000000"..., 110) = 110<br>read(3, "TRAILER!!!\0\0\0\0", 14) = 14<br>close(3) = 0<br>exit_group(1) = ?<br></div><div><br></div><div>and here's the new toybox (ignore the lseek and the write from my debugging printf):</div><div><br></div><div>utimensat(AT_FDCWD, "system/lib64/libstdc++.so", [{tv_sec=0, tv_nsec=0}, {tv_sec=0, tv_nsec=0}], AT_SYMLINK_NOFOLLOW) = 0<br>lseek(3, 0, SEEK_CUR) = 5551408<br>read(3, "070701000493fe000001ed0000000000"..., 110) = 110<br>read(3, "TRAILER!!!\0\0\0\0", 14) = 14<br>lseek(3, 0, SEEK_CUR) = 5551532<br>read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 110) = 84<br>read(3, "", 26) = 0<br>write(2, "cpio: ", 6cpio: ) = 6<br>write(2, "unexpected EOF where=54b5ac size"..., 35unexpected EOF where=54b5ac size=84) = 35<br>write(2, "\n", 1<br>) = 1<br>exit_group(1) = ?<br></div><div><br></div><div>so we have < sizeof(header) NULs at the end, and the new toybox chokes on that.</div><div><br></div><div>here's xxd for the end of the file:</div><div><br></div><div>0054b530: 3037 3037 3031 3030 3034 3933 6665 3030 070701000493fe00<br>0054b540: 3030 3031 6564 3030 3030 3030 3030 3030 0001ed0000000000<br>0054b550: 3030 3030 3030 3030 3030 3030 3031 3030 0000000000000100<br>0054b560: 3030 3030 3030 3030 3030 3030 3030 3030 0000000000000000<br>0054b570: 3030 3030 3030 3030 3030 3030 3030 3030 0000000000000000<br>0054b580: 3030 3030 3030 3030 3030 3030 3030 3030 0000000000000000<br>0054b590: 3030 3030 3062 3030 3030 3030 3030 5452 00000b00000000TR<br>0054b5a0: 4149 4c45 5221 2121 0000 0000 0000 0000 AILER!!!........<br>0054b5b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>0054b5c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>0054b5d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>0054b5e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>0054b5f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br></div><div><br></div><div>for some reason that's 192 bytes? i can dig further, but since it seems like yochiang predicted this, i assume that you two already know where these NULs are coming from?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 1:48 PM enh <<a href="mailto:enh@google.com">enh@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 11:44 AM Yi-yo Chiang <<a href="mailto:yochiang@google.com" target="_blank">yochiang@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm not sure what Elliot's goal is? I assume he's trying to extract a concatenated ramdisk, and I still see a problem in the current solution. </div></blockquote><div><br></div><div>oops, i was going to say "i added you to the bug", but it looks like i added you to the *other* cpio bug (which appears to be resolving itself as "user error, we'll fix our script"). i've added you to <a href="https://issuetracker.google.com/184732694" target="_blank">https://issuetracker.google.com/184732694</a> now, but non-googlers won't be able to follow that link.</div><div><br></div><div>all i want to do is get my rug back^W^W^W^Wkeep someone's existing workflow working.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The buffer-format (<a href="https://www.kernel.org/doc/Documentation/early-userspace/buffer-format.txt" target="_blank">https://www.kernel.org/doc/Documentation/early-userspace/buffer-format.txt</a>) says:</div><div><br></div><div> initramfs := ("\0" | cpio_archive | cpio_gzip_archive)*<br><div><div><br></div><div>In other words, both `cat a.cpio b.cpio >merged.cpio` and `(cat a.cpio && echo -n -e '\0\0\0' && cat b.cpio) >merged.cpio` are valid initramfs.</div></div></div></div></blockquote><div><br></div><div>weird. do we know _why_ that's supported?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>btw gen_init_cpio.c also pads initramfs to 512-byte boundary (<a href="https://github.com/torvalds/linux/blob/6fbd6cf85a3be127454a1ad58525a3adcf8612ab/usr/gen_init_cpio.c#L97" target="_blank">https://github.com/torvalds/linux/blob/6fbd6cf85a3be127454a1ad58525a3adcf8612ab/usr/gen_init_cpio.c#L97</a>)<br></div><div><br></div><div>If we're viewing buffer-format.txt as the "right" cpio spec, then I think we should implement this too. We should skip arbitrary extra NUL-bytes padded between cpio file frames</div></div></div></div></blockquote><div><br></div><div>not that it really matters (because i think we're all agreed that "do what the kernel does" makes more sense than "do what GNU/BSD think") but does GNU cpio cope with that? (see "strace" below for why i wonder.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 2:27 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 4/16/21 12:18 AM, Rob Landley wrote:<br>
> On 4/15/21 6:57 PM, Rob Landley wrote:<br>
>> On 4/15/21 12:06 PM, enh wrote:<br>
>>> do you already have the "do the right thing" patch ready, or should i send that<br>
>>> today?<br>
>><br>
>> I'm good, I'll try to get it in tonight. The hard part is the test suite, not<br>
>> the implementation.<br>
> <br>
> Correction: the hard part is I don't seem to have actually implemented hardlink<br>
> support yet. It just records/extracts them as separate files.<br>
> <br>
> Right. I need to test this against the kernel initramfs plumbing, in both<br>
> directions...<br>
<br>
if (!strcmp("TRAILER!!!", name)) {<br>
+ readall(afd, toybuf, 348);<br>
<br>
Where did you get 348 from? (What is this change doing?)<br>
<br>
I did what seems like a minimal change to continue past TRAILER!!! but error on<br>
an empty archive, but I apparently don't understand what your change did? (I'm<br>
assuming it fixed the use case you were seeing...?)<br></blockquote></div></blockquote><div><br></div><div>yeah, from strace i think GNU cpio avoids this misalignment by always reading 512-byte blocks (rather than toybox's "let me just read the header and then worry about anything else later").</div><div><br></div><div>all i was trying to do was "get back in sync after a TRAILER!!! entry", and 348 (by measurement) got me back to 512 after what toybox cpio had read, for the examples i had. (i had expected "is this guaranteed to be right for all possible valid cpio files?" to be the discussion we'd be having rather than the "shouldn't we just do what the kernel does?" :-) )</div><div><br></div><div>(if i haven't said so already, assume i know nothing about cpio and have never knowingly used it myself.)</div><div><br></div><div>interestingly, the current patch seems to break the AOSP build:</div><div><br></div><div>2021-04-16 18:56:56 - common.py - WARNING : Unable to get boot image build props: Failed to run command '['toybox', 'cpio', '-F', '/mnt/disks/build-disk/src/googleplex-android/sc-dev/out/soong/.temp/boot_xwvxvq26.img/uncompressed_ramdisk', '-i']' (exit code 1):<br>cpio: bad header<br></div><div><br></div><div>i'll chase up the person using `toybox cpio` and move them over to the hermetic prebuilt build toybox later. for now i'll try to reproduce this locally and see what the trouble is...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Rob<br>
_______________________________________________<br>
Toybox mailing list<br>
<a href="mailto:Toybox@lists.landley.net" target="_blank">Toybox@lists.landley.net</a><br>
<a href="http://lists.landley.net/listinfo.cgi/toybox-landley.net" rel="noreferrer" target="_blank">http://lists.landley.net/listinfo.cgi/toybox-landley.net</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><table width="90%" border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px;font-family:"Times New Roman";max-width:348px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td style="padding:0px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://i.imgur.com/eGpkLls.png" width="200" height="64"><br></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="font-family:Arial,Helvetica,Verdana,sans-serif;padding:1px 0px 5px;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Yi-yo Chiang</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="font-family:Arial,Helvetica,Verdana,sans-serif;padding:0px 0px 5px;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="font-family:Arial,Helvetica,Verdana,sans-serif;padding:0px 0px 5px;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:yochiang@google.com" target="_blank">yochiang@google.com</a></td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="font-family:Arial,Helvetica,Verdana,sans-serif;padding:0px 0px 3px;font-size:11px;line-height:13px;color:rgb(3,112,248)"></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></div></div>
</blockquote></div></div>
</blockquote></div>