From rob at landley.net Tue Jun 13 22:06:45 2017 From: rob at landley.net (Rob Landley) Date: Tue, 13 Jun 2017 17:06:45 -0500 Subject: [mkroot] [landley/mkroot] Smoke test the correct compiler (#6) In-Reply-To: References: Message-ID: <974497fa-4dda-12ee-0989-77dc2b0f5349@landley.net> On 06/12/2017 09:32 AM, Sam Thursfield wrote: > Maybe I'm missing something, but I doubt the intention here was to call > the host's native compiler. No, the check is intended to be on the native compiler, even back in Aboriginal Linux (https://github.com/landley/aboriginal/commit/ab2d8cd098c3). If you supply a broken cross compiler that's pilot error. Back in the aboriginal linux days, populating the airlock directory was done with static binaries (same as on the target) and that build failed on Fedora where Ulrich Drepper's insanity poisoned the distro to the point where libc.a was a separate package not installed by default. (These days they've switched insanity providers to the systemd guys instead.) Mkroot checked the host compiler for a few other reasons: 1) You may not have a host compiler installed at all. (It's not part of the default install on some distros.) 2) The test is right after the airlock install, so if we screwed up the compiler doing that it figures it out early. Distros have had all SORTS of funky wrappers, from distcc and ccache to Ubuntu making "gcc" a perl script that called "gcc.real" (YES REALLY, see https://github.com/landley/aboriginal/commit/988d9dc7b501f) so _that_ had to be in the wrapped $PATH... 3) If you build without cross compiling, it's going to try to use the host compiler to produce static binaries anyway. That said, the toybox airlock build we do is dynamic linking (or at least leaving it to the host toolchain to determine how to link the binaries), so I don't strictly have to _require_ static linking. I can either move this check into the "native compiling" if/else, change it to just check that it can build a runnable hello world program, or remove it entirely. Rob From rob at landley.net Wed Jun 14 17:04:05 2017 From: rob at landley.net (Rob Landley) Date: Wed, 14 Jun 2017 12:04:05 -0500 Subject: [mkroot] [landley/mkroot] Smoke test the correct compiler (#6) In-Reply-To: <974497fa-4dda-12ee-0989-77dc2b0f5349@landley.net> References: <974497fa-4dda-12ee-0989-77dc2b0f5349@landley.net> Message-ID: <5efc72b9-6ff4-1213-4b31-0c456179e426@landley.net> On 06/13/2017 05:06 PM, Rob Landley wrote: > That said, the toybox airlock build we do is dynamic linking (or at > least leaving it to the host toolchain to determine how to link the > binaries), so I don't strictly have to _require_ static linking. I can > either move this check into the "native compiling" if/else, change it to > just check that it can build a runnable hello world program, or remove > it entirely. The problem with moving it is it's currently run right after the paths are set (including the output directory to drop the hello world binary in), which have to be after we check whether or not we're cross compiling. I'd rather not split the path variable setting into two blocks, so I'll just make the check a warning in the cross compiling case, and only be an error in the native compiling case. I'll also clarify the error message that the HOST compiler can't produce static binaries (which is a common issue because Red Hat broke theirs). Providing a broken target toolchain is still pilot error. There are way too many things that could go wrong there for me to check them all. Rob From rob at landley.net Wed Jun 21 20:46:38 2017 From: rob at landley.net (Rob Landley) Date: Wed, 21 Jun 2017 15:46:38 -0500 Subject: [mkroot] Design issue: uploading binaries. Message-ID: I built mkroot binaries for a bunch of targets, but am not quite sure where (or if) to upload them. This remains an unsolved problem and I'm open to suggestions. I can stick them on landley.net/mkroot pretty easily, but the project doesn't have the same kind of website the old one did. It's a self-contained github project with a README. In theory github has a "releases" mechanism that lets you attach binaries to a tag, but the web-based mechanism is very manual and https://developer.github.com/v3/repos/releases/ is not making a lot of sense to me. (Googling for this sort of thing finds stuff for the older https://developer.github.com/v3/repos/downloads/ mechanism that was deprecated and removed in 2012.) (I'm also leaning towards not posting binaries until I get busybox cleaned out of the build. At which point everything would be bsd/pd licensed.) As for toolchain binaries, I've been poking Rich Felker about uploading musl-cross-make binaries for ages (although he's now waited long enough that https://lwn.net/Articles/726023/ happened and kinda took the thunder away from that), and _he_ wants to use https://docs.travis-ci.com/user/deployment/releases/ to produce and automatically upload them for each new tag. Which is nice, but neither of us has sat down to work out how to actually do it, and github has a 1gb quota per account that could get eaten pretty quickly by that sort of thing. Meanwhile, the kernel guys are using https://www.kernel.org/pub/tools/crosstool/ which doesn't have a C library (so you can pretty much _only_ build the kernel with them, no userspace). And I have a todo item to try to build stuff with Android's NDK when the next version ships. Lack of native compiler binary tarballs (which only musl-cross-make builds, and it's still not providing downloadable binaries) is semi-blocking the next major design stage, which is reproducing aboriginal linux's build control images under mkroot. What it's building right now isn't a development environment. I have to compile make from source (and/or implement one in toybox), but without a toolchain it's not much use. That said I do have an mcm-buildall.sh script checked in, and I can just have this _also_ depend on the mcm symlink like cross.sh does now... Rob From rob at landley.net Wed Jun 21 20:54:42 2017 From: rob at landley.net (Rob Landley) Date: Wed, 21 Jun 2017 15:54:42 -0500 Subject: [mkroot] Design issue: web page? Message-ID: Aboriginal Linux had a _lot_ of documentation associated with it, most prominently: 1) The about page https://landley.net/aboriginal/about.html which has a LOT of info that still applies to mkroot. 2) The FAQ http://landley.net/aboriginal/FAQ.html explains stuff like miniconfig that's still relevant. 3) I did a big presentation way back when about native compiling uner emulation instead of cross compiling https://speakerdeck.com/landley/developing-for-non-x86-targets-using-qemu and even though I was terrible ab 4) The http://landley.net/aboriginal/screenshots/ page has a brief description of each architecture. There was also a comment in sources/targets about that sort of thing. Right now, all mkroot has is the README file, and I'm not sure what I should add to that. Maybe I should put stuff in the github wiki? Except then it's not downloaded as part of the project and versioned independently from the repo. (Then again, so would a normal web page be...) Again, open to suggestions. :) Rob From rob at landley.net Wed Jun 21 20:59:23 2017 From: rob at landley.net (Rob Landley) Date: Wed, 21 Jun 2017 15:59:23 -0500 Subject: [mkroot] I tagged a mkroot release. Message-ID: Here it is: https://github.com/landley/mkroot/releases/tag/0.9.0 You can grab it and build it right now, and it should work. (The big blocker was it needed a new toybox, which is now released and it's using it by default.) I just posted about a couple pending architectural issues on the mailing list: http://lists.landley.net/pipermail/mkroot-landley.net/2017-June/000018.html http://lists.landley.net/pipermail/mkroot-landley.net/2017-June/000019.html I'm probably going to move the aboriginal linux mailing list subscriptions over to the new mkroot list this weekend, now that there's something to try out and complain about. :) Rob From atoussaint1976 at gmail.com Wed Jun 21 21:08:45 2017 From: atoussaint1976 at gmail.com (Alain Toussaint) Date: Wed, 21 Jun 2017 17:08:45 -0400 Subject: [mkroot] Design issue: uploading binaries. In-Reply-To: References: Message-ID: Would it be useful to have a linode server for that. I've been mulling for one myself and would likely rent one soon which I can make available to mkroot and install travis-ci on it. Alain 2017-06-21 16:46 GMT-04:00 Rob Landley : > I built mkroot binaries for a bunch of targets, but am not quite sure > where (or if) to upload them. This remains an unsolved problem and I'm > open to suggestions. > > I can stick them on landley.net/mkroot pretty easily, but the project > doesn't have the same kind of website the old one did. It's a > self-contained github project with a README. In theory github has a > "releases" mechanism that lets you attach binaries to a tag, but the > web-based mechanism is very manual and > https://developer.github.com/v3/repos/releases/ is not making a lot of > sense to me. (Googling for this sort of thing finds stuff for the older > https://developer.github.com/v3/repos/downloads/ mechanism that was > deprecated and removed in 2012.) > > (I'm also leaning towards not posting binaries until I get busybox > cleaned out of the build. At which point everything would be bsd/pd > licensed.) > > As for toolchain binaries, I've been poking Rich Felker about uploading > musl-cross-make binaries for ages (although he's now waited long enough > that https://lwn.net/Articles/726023/ happened and kinda took the > thunder away from that), and _he_ wants to use > https://docs.travis-ci.com/user/deployment/releases/ to produce and > automatically upload them for each new tag. Which is nice, but neither > of us has sat down to work out how to actually do it, and github has a > 1gb quota per account that could get eaten pretty quickly by that sort > of thing. > > Meanwhile, the kernel guys are using > https://www.kernel.org/pub/tools/crosstool/ which doesn't have a C > library (so you can pretty much _only_ build the kernel with them, no > userspace). And I have a todo item to try to build stuff with Android's > NDK when the next version ships. > > Lack of native compiler binary tarballs (which only musl-cross-make > builds, and it's still not providing downloadable binaries) is > semi-blocking the next major design stage, which is reproducing > aboriginal linux's build control images under mkroot. What it's building > right now isn't a development environment. I have to compile make from > source (and/or implement one in toybox), but without a toolchain it's > not much use. > > That said I do have an mcm-buildall.sh script checked in, and I can just > have this _also_ depend on the mcm symlink like cross.sh does now... > > Rob > _______________________________________________ > mkroot mailing list > mkroot at lists.landley.net > http://lists.landley.net/listinfo.cgi/mkroot-landley.net From rob at landley.net Wed Jun 21 21:41:15 2017 From: rob at landley.net (Rob Landley) Date: Wed, 21 Jun 2017 16:41:15 -0500 Subject: [mkroot] Design issue: uploading binaries. In-Reply-To: References: Message-ID: On 06/21/2017 04:08 PM, Alain Toussaint wrote: > Would it be useful to have a linode server for that. I've been mulling > for one myself and would likely rent one soon which I can make > available to mkroot and install travis-ci on it. > > Alain For mkroot, or for musl-cross-make? For mkroot I'm ok building binaries as part of a release. (I already have to do that to test everything before shipping it.) For musl-cross-make, I'm fuzzy on the details of Rich's plans. Rob > 2017-06-21 16:46 GMT-04:00 Rob Landley : >> I built mkroot binaries for a bunch of targets, but am not quite sure >> where (or if) to upload them. This remains an unsolved problem and I'm >> open to suggestions. >> >> I can stick them on landley.net/mkroot pretty easily, but the project >> doesn't have the same kind of website the old one did. It's a >> self-contained github project with a README. In theory github has a >> "releases" mechanism that lets you attach binaries to a tag, but the >> web-based mechanism is very manual and >> https://developer.github.com/v3/repos/releases/ is not making a lot of >> sense to me. (Googling for this sort of thing finds stuff for the older >> https://developer.github.com/v3/repos/downloads/ mechanism that was >> deprecated and removed in 2012.) >> >> (I'm also leaning towards not posting binaries until I get busybox >> cleaned out of the build. At which point everything would be bsd/pd >> licensed.) >> >> As for toolchain binaries, I've been poking Rich Felker about uploading >> musl-cross-make binaries for ages (although he's now waited long enough >> that https://lwn.net/Articles/726023/ happened and kinda took the >> thunder away from that), and _he_ wants to use >> https://docs.travis-ci.com/user/deployment/releases/ to produce and >> automatically upload them for each new tag. Which is nice, but neither >> of us has sat down to work out how to actually do it, and github has a >> 1gb quota per account that could get eaten pretty quickly by that sort >> of thing. >> >> Meanwhile, the kernel guys are using >> https://www.kernel.org/pub/tools/crosstool/ which doesn't have a C >> library (so you can pretty much _only_ build the kernel with them, no >> userspace). And I have a todo item to try to build stuff with Android's >> NDK when the next version ships. >> >> Lack of native compiler binary tarballs (which only musl-cross-make >> builds, and it's still not providing downloadable binaries) is >> semi-blocking the next major design stage, which is reproducing >> aboriginal linux's build control images under mkroot. What it's building >> right now isn't a development environment. I have to compile make from >> source (and/or implement one in toybox), but without a toolchain it's >> not much use. >> >> That said I do have an mcm-buildall.sh script checked in, and I can just >> have this _also_ depend on the mcm symlink like cross.sh does now... >> >> Rob >> _______________________________________________ >> mkroot mailing list >> mkroot at lists.landley.net >> http://lists.landley.net/listinfo.cgi/mkroot-landley.net > From atoussaint1976 at gmail.com Wed Jun 21 21:53:36 2017 From: atoussaint1976 at gmail.com (Alain Toussaint) Date: Wed, 21 Jun 2017 17:53:36 -0400 Subject: [mkroot] Design issue: uploading binaries. In-Reply-To: References: Message-ID: Could be both but at least mkroot. Basically, it's been slightly over a year I've been building LFS svn on a bi-monthly basis and would like to automate that. Automating mkroot & musl-cross-make would be a useful addition to the linode server. Alain 2017-06-21 17:41 GMT-04:00 Rob Landley : > On 06/21/2017 04:08 PM, Alain Toussaint wrote: >> Would it be useful to have a linode server for that. I've been mulling >> for one myself and would likely rent one soon which I can make >> available to mkroot and install travis-ci on it. >> >> Alain > > For mkroot, or for musl-cross-make? > > For mkroot I'm ok building binaries as part of a release. (I already > have to do that to test everything before shipping it.) > > For musl-cross-make, I'm fuzzy on the details of Rich's plans. > > Rob > >> 2017-06-21 16:46 GMT-04:00 Rob Landley : >>> I built mkroot binaries for a bunch of targets, but am not quite sure >>> where (or if) to upload them. This remains an unsolved problem and I'm >>> open to suggestions. >>> >>> I can stick them on landley.net/mkroot pretty easily, but the project >>> doesn't have the same kind of website the old one did. It's a >>> self-contained github project with a README. In theory github has a >>> "releases" mechanism that lets you attach binaries to a tag, but the >>> web-based mechanism is very manual and >>> https://developer.github.com/v3/repos/releases/ is not making a lot of >>> sense to me. (Googling for this sort of thing finds stuff for the older >>> https://developer.github.com/v3/repos/downloads/ mechanism that was >>> deprecated and removed in 2012.) >>> >>> (I'm also leaning towards not posting binaries until I get busybox >>> cleaned out of the build. At which point everything would be bsd/pd >>> licensed.) >>> >>> As for toolchain binaries, I've been poking Rich Felker about uploading >>> musl-cross-make binaries for ages (although he's now waited long enough >>> that https://lwn.net/Articles/726023/ happened and kinda took the >>> thunder away from that), and _he_ wants to use >>> https://docs.travis-ci.com/user/deployment/releases/ to produce and >>> automatically upload them for each new tag. Which is nice, but neither >>> of us has sat down to work out how to actually do it, and github has a >>> 1gb quota per account that could get eaten pretty quickly by that sort >>> of thing. >>> >>> Meanwhile, the kernel guys are using >>> https://www.kernel.org/pub/tools/crosstool/ which doesn't have a C >>> library (so you can pretty much _only_ build the kernel with them, no >>> userspace). And I have a todo item to try to build stuff with Android's >>> NDK when the next version ships. >>> >>> Lack of native compiler binary tarballs (which only musl-cross-make >>> builds, and it's still not providing downloadable binaries) is >>> semi-blocking the next major design stage, which is reproducing >>> aboriginal linux's build control images under mkroot. What it's building >>> right now isn't a development environment. I have to compile make from >>> source (and/or implement one in toybox), but without a toolchain it's >>> not much use. >>> >>> That said I do have an mcm-buildall.sh script checked in, and I can just >>> have this _also_ depend on the mcm symlink like cross.sh does now... >>> >>> Rob >>> _______________________________________________ >>> mkroot mailing list >>> mkroot at lists.landley.net >>> http://lists.landley.net/listinfo.cgi/mkroot-landley.net >>