[Toybox] GitHub Action Example

[enh]

On Mon, Jun 15, 2020 at 11:05 AM Rob Landley <rob at landley.net> wrote:
>
> On 6/15/20 11:07 AM, enh via Toybox wrote:
> > On Sat, Jun 13, 2020 at 12:30 AM Eric Molitor <emolitor at molitor.org> wrote:
> >>
> >> Would aarch64 builds on Ubuntu and Alpine be useful? That will be slightly more complex to setup but is possible.
> >
> > i don't actually remember us ever having an aarch64-specific issue.
> > (funnily enough, a 32-bit x86 build would probably find more bugs,
> > since i don't think anyone regularly tests any 32-bit arch locally. i
> > certainly only find 32-bit issues when i try to run on an Android
> > emulator!)
>
> I test it before releases, and I test the j-core stuff which is still only 32
> bit. But it's not tested nearly as regularly as the 64 bit stuff is.

(to be clear, i meant "at the time of commit". thanks to the 32-bit
x86 "cuttlefish" emulator testing, we do get testing every time i try
to sync AOSP. but .)

> I'm trying to get the mkroot plumbing to run the test suite under qemu. I'm
> about 3/4 of the way there. That should get more variants tested in a more
> easily automatable fashion, but it's all musl (and glibc) unless bionic stops
> segfaulting "hello world" when run in a chroot that doesn't have /dev/null in it
> yet.
>
> But right now the test suite hasn't got nearly as much coverage as I'd like, and
> until I fix that running it regularly doesn't really _prove_ anything. (And
> finishing the shell is eating my cycles...)

in the Dijkstra sense (page 16 of
http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1969.PDF), sure.
but we definitely have introduced test failures. and as we've seen
from Android, just running them a lot helps shake out issues too ---
mostly in the tests, but also in the toys. (and anyone running tests
downstream is going to hit test flake, so that still needs fixing.)

and note also that this checks you can _build_ both glibc and musl
(which has been problematic in the past), both gcc and clang (which
has been problematic in the past), and that there are no ASan issues
that will prevent running the tests/toys on a HWASan/MTE aarch64
device.

> And 32 bit argument processing has a known structural limitation (the "truncate
> -s 8G" thing) which I've mentioned here before. I know how to fix it, but the
> fix is intrusive enough I'm not sure it's worth doing?

(i'm much more interested in getting to where we have 64-bit-only,
both to replace the current 64/32 high end and the 32-only low end.)

> >>> it seems like your setup is running on a cron-like timer? is there a way to say "on every push" instead?
> >>>
> >> There are three build triggers in the configuration, cron, push on master and pull request on master. As I forked the repo you only see changes to my repo (emolitor/toybox) trigger builds and not landley/toybox.
> >
> > ah, i see. hopefully rob will look at
> > https://github.com/emolitor/toybox/runs/767753680?check_suite_focus=true
> > and turn this on for the main repo then :-)
>
> I'm uncomfortable putting Microsoft Github dependencies directly into toybox,
> especially now Microsoft seems to be back on its "embrace and extend" kick:
>
>   https://www.youtube.com/watch?v=pRMb51rx2Gg#t=38s

it's not a dependency though. just a convenience. right now, we have
humans doing this, and we can always go back to that if we have to.
but if MS is going to give you free CPU cycles to save a little bit of
human time...

> >>  I just added a parallel ASAN build on Ubuntu using clang to test.
> >
> > awesome, thanks!
> >
> > and rob just needs to add this file to the repository to get this?
> > there's no extra setup in the github ui?
>
> I've been using github as a mirror. It's a distribution server I never pull from
> or use the web infrastructure for anything (except as a mutant proprietary
> bugzilla because people keep filing stuff there, but back in the busybox days
> distros would often have their own bugzillas with busybox bugs and patches in
> them, nothing new there).
>
> Adding a .github subdirectory to the source would be a policy change. I'm happy
> with a fork doing it, but am uncomfortable putting it in the main repo. (Not
> fatally uncomfortable, but... ergh?)

but if it's in a fork, we don't get the benefit. that's basically back
to humans doing something that's a job for a computer...

> >> [https://github.com/emolitor/toybox/actions/runs/134098865]
> >> --- a/.github/workflows/toybox.yml
> >> +++ b/.github/workflows/toybox.yml
> >> @@ -37,6 +37,20 @@ jobs:
> >>      - name: Test
> >>        run: make tests
> >>
> >> +  Ubuntu-20_04-Clang-ASAN:
> >> +    runs-on: ubuntu-20.04
> >> +
> >> +    steps:
> >> +    - uses: actions/checkout at v2
> >> +    - name: Setup
> >> +      run: sudo apt-get install build-essential clang
> >> +    - name: Configure
> >> +      run: make defconfig
> >> +    - name: Build
> >> +      run: CC=clang ASAN=1 make
> >> +    - name: Test
> >> +      run: make tests
> >> +
>
> Given that you aren't setting VERBOSE=fail I assume you want me to add a patch like:
>
> diff --git a/scripts/test.sh b/scripts/test.sh
> index 20f76d09..cdfe3bdb 100755
> --- a/scripts/test.sh
> +++ b/scripts/test.sh
> @@ -60,3 +60,5 @@ else
>      do_test "$i"
>    done
>  fi
> +
> +[ $FAILCOUNT -eq 0 ]
>
> So make can _tell_ it failed?

(even ignoring the slightly different "run it on Android" wrapper, i
always run the regular toybox tests with VERBOSE=1. i don't understand
why i'd want to _not_ see the detail that might help me fix it.
especially if it's a flaky failure rather than trivially reproducible,
but even in the latter case it's still annoying to have to run again
with VERBOSE=1.)

> Also, you can "CC=clang ASAN=1 make defconfig toybox tests" all on one line. And
> you probably want defconfig to have CC=clang anyway because some of the config
> probes run there (creating generated/*).
>
> Rob