<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 14, 2021 at 11:49 PM Rob Landley <<a href="mailto:rob@landley.net">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 4/14/21 1:26 PM, enh wrote:<br>
> Could you read the linux doc thing and confirm that the behavior you want is<br>
> still to stop at TRAILER instead of flushing hardlink context but otherwise<br>
> continuing to extract like the kernel guys documented for initramfs? (Or am I<br>
> misremembering? It's been a while...)<br>
> <br>
> in the thread you linked to, they say "I wonder how existing GNU or BSD cpio ...<br>
> would deal with reading such a file". all i'm saying is "GNU cpio exits on the<br>
> next record boundary, and people have scripts that rely on this".<br>
<br>
My question was really "would continuing until you run out of cpio records, and<br>
exiting with an error if there were no cpio records, also satisfy those scripts"?<br>
<br>
> the Linux docs say things like<br>
> <br>
> The cpio "TRAILER!!!" entry (cpio end-of-archive) is optional, but is<br>
> not ignored; see "handling of hard links" below.<br>
> <br>
> but that doesn't match what actual implementations of cpio do. (assuming you<br>
> don't interpret optional as meaning "you don't have to have one, but if you<br>
> don't, the tool will exit with an error complaining that you don't have one" :-) )<br>
<br>
A year or three back I had it not adding the TRAILER!!! entry, then added a<br>
--trailer option, and you submitted a commit removing that option so it always<br>
adds TRAILER now.<br>
<br>
But not having one isn't unprecedented...<br>
<br>
> i think the most interesting thing for me in the docs was:<br>
> <br>
> When a "TRAILER!!!" end-of-archive marker is seen, the tuple buffer is<br>
> reset. This permits archives which are generated independently to be<br>
> concatenated.<br>
> <br>
> because -- even if i haven't really understood _why_ people are concatenating<br>
> cpio files -- at least this shows that the main consumers/producers agree that<br>
> this is an expected use case.<br>
<br>
They're incrementally generating filesystems, using a base cpio and then adding<br>
more entries.<br>
<br>
If your base has /dev/console style nodes in it with special ownership and<br>
permissions which you can't create locally as a normal user, you have to use an<br>
awkward tool like gen-initramfs-cpio from the kernel source to generate<br>
synthetic cpio entries. But you then often want to append a directory full of<br>
files that live in your local filesystem using normal "find | cpio".<br>
<br>
It's also a poor man's form of initramfs package management: select this and<br>
this and this without extracting them all into a temporary directory and then<br>
packaging up the directory (and potentially having<br>
permissions/ownership/timestamps change). This trick can even drop start files<br>
next to each other in etc/rc for sysvinit to pick up and run on boot.<br>
<br>
> i'm assuming the "exit when you see TRAILER!!! and let the next cpio instance<br>
> worry about the rest" behavior is just the least-effort implementation of the<br>
> hard-link flush stuff:<br>
> <br>
> To combine file data from different sources (without having to<br>
> regenerate the (c_maj,c_min,c_ino) fields), therefore, either one of<br>
> the following techniques can be used:<br>
> <br>
> a) Separate the different file data sources with a "TRAILER!!!"<br>
> end-of-archive marker, or<br>
> <br>
> exiting when you see TRAILER!!! implicitly loses any cpio state, and reporting<br>
> an error if you hit EOF without seeing TRAILER!!! lets you know when to stop<br>
> running a new cpio?<br>
<br>
Least-effort implementation of flush is what I'm assuming too. I prefer to put<br>
in more effort and doing it right.<br>
<br>
Extracting the whole archive seems like the correct behavior because it's what<br>
the kernel initramfs plumbing does, and given that posix yanked cpio back in<br>
susv2 and Jorg "Solaris Solaris Uber Alles" Schilling got outright indignant<br>
when I suggested putting it back because it's actually _used_, that means the<br>
only modern spec we have is the kernel spec (that I am aware of).<br>
<br>
> (i think the doc is trying to distinguish between a cpio file [where TRAILER!!!<br>
> marks the end] and an "initramfs buffer" which can contain multiple concatenated<br>
> cpio files [and hence more than one TRAILER!!!]. so things processing initramfs<br>
> buffers need to be cleverer than cpio when it comes to TRAILER!!!, but cpio<br>
> doesn't. [and in practice, isn't.])<br>
<br>
I agree gnu cpio doesn't, but that's because gnu.<br>
<br>
> i think that answers your question, but perhaps in excessive detail, so i'll<br>
> re-quote you and try again:<br>
<br>
I don't mind excessive detail when I'm trying to figure out the correct course<br>
of action for a design issue.<br>
<br>
>> confirm that the behavior you want is<br>
>> still to stop at TRAILER instead of flushing hardlink context but otherwise<br>
>> continuing to extract<br>
> <br>
> i agree that based on the Linux docs it would be more sensible to flush but<br>
> continue, but that's demonstrably not what GNU cpio does, so it doesn't seem<br>
> particularly helpful for us to do it. callers already have to have the bash<br>
> while loop nonsense,<br>
<br>
Some callers do, and I agree we can't _break_ them. But rendering the loop a NOP<br>
doesn't break it.<br>
<br>
> and implementing the better behavior in toybox would still<br>
> be "broken" from that perspective because they'd loop forever --- toybox would<br>
> at least have to consider the empty input as an error,<br>
<br>
Empty input should be an error, yes. That's consistent with tar:<br>
<br>
$ toybox cpio -i < /dev/null<br>
$ echo $?<br>
0<br>
$ toybox tar x < /dev/null<br>
tar: Not tar<br>
$ echo $?<br>
1<br>
<br>
Whatever else we decide to do here, making empty input be an error sounds<br>
correct to me. Of course the gnu/dammit version goes:<br>
<br>
$ cpio -i < /dev/null<br>
Found end of tape. To continue, type device/file name when ready.<br>
<br>
Which... no? Just no.<br>
<br>
> at which point we haven't<br>
> really reduced the ugliness much? (i'm also scared to suggest anything beyond<br>
> "do what GNU does" because i don't personally know anything about cpio, and have<br>
> never used it except to generate minimal repro cases for stuff that kernel folks<br>
> bring up.<br>
<br>
I have, sadly, had to learn rather a lot about it although I don't claim to be<br>
an expert.<br>
<br>
Still, if we're changing the behavior, eating all the input seems more correct,<br>
and erroring on empty input seems like it would satisfy the loops people are<br>
using to work around the limitations in the gnu/dammit implementation (a<br>
limitation which is already not present in the kernel's implementation used by<br>
initramfs).<br>
<br>
> i haven't looked at BSD, but they seem to interpret TRAILER!!! as end<br>
> of archive too: <a href="https://www.freebsd.org/cgi/man.cgi?query=cpio&sektion=5" rel="noreferrer" target="_blank">https://www.freebsd.org/cgi/man.cgi?query=cpio&sektion=5</a> ... and<br>
<br>
Have they changed the behavior of their tool in the past 25 years? (It's not<br>
like 64 bit processors or large file support means much when your file format is<br>
8 hex digits for all the metadata fields...)<br></blockquote><div><br></div><div>i had assumed that macOS used GNU cpio, but i just checked and it's actually the BSD cpio (3.3.2). so i can report that they behave differently again: they do stop at TRAILER!!! but putting them in the loop doesn't seem to help, but they also don't error so you end up in the infinite loop. so superficially similar to toybox's currently behavior, except without any error message?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> eighthly, carrying on past TRAILER!!! when no-one else does sounds like one of<br>
> those security issues Android had back in the "zip master key" days; even if the<br>
<br>
I didn't hear this story, but it sounds unpleasant.<br>
<br>
> format is stupid, it's safer when everyone interprets the format the same way...<br>
> who knows what crap people are accidentally/deliberately ignoring past a<br>
> TRAILER!!! that isn't actually at the end [because they _don't_ have the bash<br>
> while loop]? i'd prefer not to find out :-) )<br>
<br>
Ok, valid point. But if they feed such an initramfs into the kernel it will<br>
process all those records now, so the behavior _isn't_ currently consistent.<br></blockquote><div><br></div><div>/me checks kernel source to confirm that init/initramfs.c actually does what buffer-format.rst claims. seems to be true.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Who are the users of this you're seeing? (And the other major user of this (that<br>
I'm aware of) is RPM package format. I don't know what do they do, because I<br>
don't know where the source to the rpm tools lives. I lost track circa<br>
<a href="https://lwn.net/Articles/196523/" rel="noreferrer" target="_blank">https://lwn.net/Articles/196523/</a> and moved to .deb based systems anyway...)<br>
<br>
Hmmm. If you're really concerned about more capable default behavior being<br>
nebulously unsafe in a way that I can't prove a negative (grumble grumble),<br>
maybe it needs an --all option? The man page doesn't mention -A but of course:<br>
<br>
$ cpio -iA < /dev/null<br>
cpio: --append is meaningless with --extract<br>
$ cpio -ia < /dev/null<br>
cpio: --reset is meaningless with --extract<br>
<br>
This is gnu we're talking about: they only actually document stuff in "info"<br>
pages. Sigh. (The man page mentions --append but has no short option for it.<br>
The only reset it mentions is --reset-access-time and it doesn't say what that<br>
DOES...)<br>
<br>
Needing a for loop around the tool seems broken to me. Not breaking people's<br>
workarounds is important, but implementing behavior that WON'T while rendering<br>
the workaround unnecessary seems easy enough?<br>
<br>
People depending on a limitation of the tool for "security" is hard for me to<br>
say anything coherent about. I _want_ it to be the default behavior, but if it<br>
needs to be an option...<br></blockquote><div><br></div><div>no, the fact that the kernel does interpret these files the other i think actually flips this argument around... it would make a lot more sense if cpio agreed.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> hmm. my second attempt seems to have more words than my first. i'll stop here.<br>
<br>
Another reason the for loop creeps me out is programs read more data than they<br>
actually need from input ALL THE TIME. It's how ansi FILE * buffers work, and an<br>
input pipe isn't seekable so you can't put the data _back_ if you find yourself<br>
with extra and are about to exit.<br>
<br>
This is an implicit dependence on an implementation detail, that you can<br>
continue from where the previous program left off reading the same pipe without<br>
having lost anything to buffers reading ahead. (Yes, I wrote my cpio with fd<br>
rather than FILE for that reason, but DEPENDING on it? Ew.)<br></blockquote><div><br></div><div>afaict this is the stackoverflow workaround for the GNU cpio behavior.</div><div><br></div><div>but, yeah, you've persuaded me that "behave like initramfs.c" is the way to go. i've asked the original submitter and they got back really quickly saying basically "that sounds great; i didn't ask for that because i assumed you'd want to behave the same as GNU".</div><div><br></div><div>do you already have the "do the right thing" patch ready, or should i send that today?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> (i noticed as well that everyone seems to actually deal in _compressed_ cpio<br>
> files, so in an ideal world i suspect cpio should be as intelligent as tar when<br>
> it comes to such things --- but i think cpio'ing is too niche to warrant doing<br>
> anything better than GNU.)<br>
<br>
The linux kernel already does better than gnu. That's why they wrote their own<br>
cpio create and extract plumbing. Create is in:<br>
<br>
<a href="https://github.com/torvalds/linux/blob/master/usr/gen_init_cpio.c" rel="noreferrer" target="_blank">https://github.com/torvalds/linux/blob/master/usr/gen_init_cpio.c</a><br>
<a href="https://github.com/torvalds/linux/blob/master/usr/gen_initramfs.sh" rel="noreferrer" target="_blank">https://github.com/torvalds/linux/blob/master/usr/gen_initramfs.sh</a><br>
<br>
And extract is:<br>
<br>
<a href="https://github.com/torvalds/linux/blob/master/init/initramfs.c#L256" rel="noreferrer" target="_blank">https://github.com/torvalds/linux/blob/master/init/initramfs.c#L256</a><br>
<br>
Dunno what rpm is doing behind the scenes, but the kernel guys have talked about<br>
xattr support (and sparse files, and 64 bit timestamps, and...) on more than one<br>
occasion. Hence my todo list section on that, albeit in the probably post-1.0<br>
"teach patch.c about the git file rename syntax" sort of way...<br>
<br>
Linux cpio outgrowing gnu is probably inevitable. Richard Stallman is not<br>
steering anything, hasn't been for decades. (He's sitting in a big chair making<br>
vroom-vroom noises with his mouth, but the wheel and pedals aren't connected to<br>
anything.)<br>
<br>
Rob<br>
<br>
P.S. sparse files are also a potential way to handle files > 4 gigs, by breaking<br>
them into segments, but this is initramfs we're talking about so people<br>
generally make pained noises when it comes up.<br>
<br>
P.P.S. RPM also addressed large file support, but as usual is profoundly<br>
unhelpful in saying exactly HOW ala <a href="https://rpm.org/devel_doc/large_files.html" rel="noreferrer" target="_blank">https://rpm.org/devel_doc/large_files.html</a><br>
because their business model is to obfuscate stuff until you pay them thousands<br>
of dollars to be experts and not ask questions. I'm under the impression they<br>
named this business model "enterprise" after the way the holodeck keeps<br>
malfunctioning and trying to kill people. See also systemd.<br>
</blockquote></div></div>