[Toybox] FreeBSD issues

[Rob Landley]

On 1/16/19 10:25 AM, Ed Maste wrote:
> I've taken a first look at the tests on FreeBSD. Although I expect any fixes
> will come from someone in the FreeBSD community (other than if we find general
> issues) I wonder what is the best way to track these. I'd find it convenient to
> track them in GitHub issues in landley/toybox, but understand if you'd rather
> not have them there.

If somebody _else_ finds github useful, go for it.

> In that case I can track issues in a GitHub fork for FreeBSD.

That would work too, although I'd prefer not to have long-term divergence in the
codebase. (If android's avoided that, modulo the build system, I hope FreeBSD
can. :)

> I'm looking at grep now, and find that it always returns error:
> 
> % echo abcdef | ./toybox grep abc; echo $?
> abcdef
> 2
> % echo abcdef | ./toybox grep xyz; echo $?
> 2

You were going to send a procedure for setting up a qemu build/test environment?

> and grep -q is not quiet:
> 
> % echo abcdef | ./toybox grep -q abc; echo $?
> abcdef
> 2

It's working here for me on Linux...

Could you do:

  grep FOR_grep -A 33 generated/flags.h

Mine's:

#ifdef FOR_grep
#ifndef TT
#define TT this.grep
#endif
#define FLAG_x (1<<0)
#define FLAG_m (1<<1)
#define FLAG_A (1<<2)
#define FLAG_B (1<<3)
#define FLAG_C (1<<4)
#define FLAG_f (1<<5)
#define FLAG_e (1<<6)
#define FLAG_q (1<<7)
#define FLAG_l (1<<8)
#define FLAG_c (1<<9)
#define FLAG_w (1<<10)
#define FLAG_v (1<<11)
#define FLAG_s (1<<12)
#define FLAG_r (1<<13)
#define FLAG_o (1<<14)
#define FLAG_n (1<<15)
#define FLAG_i (1<<16)
#define FLAG_h (1<<17)
#define FLAG_b (1<<18)
#define FLAG_a (1<<19)
#define FLAG_I (1<<20)
#define FLAG_H (1<<21)
#define FLAG_F (1<<22)
#define FLAG_E (1<<23)
#define FLAG_z (1<<24)
#define FLAG_Z (1<<25)
#define FLAG_M (1<<26)
#define FLAG_S (1<<27)
#define FLAG_color (1<<28)
#endif

The plumbing that builds that starts at
https://github.com/landley/toybox/blob/master/scripts/make.sh#L161 which
compiles a C program
(https://github.com/landley/toybox/blob/master/scripts/mkflags.c), defines a
bash function which feeds data through the C program, and then calls the bash
function if the file needs to be updated.

The C program should work on BSD (it should be all posix+c99+LP64 standard code
and it worked on macos), but one of the commands the bash function calls might
behave differently? Note that the pipeline on line 205 saves the input to the C
program in generated/flags.raw for debugging purposes. I attached one I have
lying around.

The shell function is extracting the NEWTOY() optstring argument from each
toys/*/*.c file and resolving the USE_BLAH() macros (see the description of
generated/config.h under http://landley.net/toybox/code.html#generated and yes I
need to do a youtube video series walking people through all this)...

The C program is producing the header file: its output is saved directly with no
further processing.

The C program's input (what the shell function is producing and saving in
flags.raw) is a sorted series o lines with three arguments on each: the command
name, the option string for the current config (with any unneeded arguments
removed), and the option string for allyesconfig.

This lets the C program work out what the flag bit values should be for
allyesconfig, and make sure the bit positions in the header don't change when
the config changes. (The macros resolve to 0 when the config switches them off
via USE_CONFIGOPTION("abcde") macros, so if (FLAG(x)) can resolve to zero and
dead code elimination can remove the unneeded stuff. But you can also #define
FORCE_FLAGS to make them NOT resolve to zero, see
http://landley.net/toybox/code.html#lib_args for more on that...)

I tried to explain it all in http://landley.net/toybox/code.html but it's a
constant struggle to keep docs updated/complete. See "I need to do a youtube
video series", above...

Most likely the shell stuff that's generating the three argument lines is
borked, which means most likely one of the FreeBSD commands it's calling isn't
working the way Linux's does. (Toybox builds under toybox, for the commands it's
implemented which aren't in toys/pending anyway. That I regression test with
https://github.com/landley/mkroot semi-regularly.)

Rob
-------------- next part --------------
acpi "abctV" "abctV"
arp " " "vi:nDsdap:A:H:[+Ap][!sd]"
arping " " "<1>1s:I:w#<0c#<0AUDbqf[+AU][+Df]"
ascii  " "  " "
base64 "diw#<0=76[!dw]" "diw#<0=76[!dw]"
basename "<1as:" "<1as:"
bc " " "i(interactive)l(mathlib)q(quiet)s(standard)w(warn)"
blkid  " "  " "
blockdev "<1>1(setro)(setrw)(getro)(getss)(getbsz)(setbsz)#<0(getsz)(getsize)(getsize64)(getra)(setra)#<0(flushbufs)(rereadpt)" "<1>1(setro)(setrw)(getro)(getss)(getbsz)(setbsz)#<0(getsz)(getsize)(getsize64)(getra)(setra)#<0(flushbufs)(rereadpt)"
bootchartd " "  " "
brctl " " "<1"
bunzip2 "cftkv" "cftkv"
bzcat  " "  " "
cal ">2" ">2"
cat "u" "uvte"
catv "vte" "vte"
cd " "  " "
chattr  " "  " "
chcon " " "<2hvR"
chgrp "<2hPLHRfv[-HLP]" "<2hPLHRfv[-HLP]"
chmod "<2?vRf[-vf]" "<2?vRf[-vf]"
chroot "^<1" "^<1"
chrt "^mp#<0iRbrfo[!ibrfo]" "^mp#<0iRbrfo[!ibrfo]"
chvt "<1" "<1"
cksum "HIPLN" "HIPLN"
clear  " "  " "
cmp "<2>2ls(silent)(quiet)[!ls]" "<2>2ls(silent)(quiet)[!ls]"
comm "<2>2321" "<2>2321"
count  " "  " "
cp "<2(preserve):;D(parents)RHLPprdaslvnF(remove-destination)fi[-HLPd][-ni]" "<2(preserve):;D(parents)RHLPprdaslvnF(remove-destination)fi[-HLPd][-ni]"
cpio "(no-preserve-owner)(trailer)mduH:p:|i|t|F:v(verbose)o|[!pio][!pot][!pF]" "(no-preserve-owner)(trailer)mduH:p:|i|t|F:v(verbose)o|[!pio][!pot][!pF]"
crc32  " "  " "
crond " " "fbSl#<0=8d#<0L:c:[-bf][-LS][-ld]"
crontab " " "c:u:elr[!elr]"
cut "b*|c*|f*|F*|C*|O(output-delimiter):d:sDn[!cbf]" "b*|c*|f*|F*|C*|O(output-delimiter):d:sDn[!cbf]"
date "d:D:r:u[!dr]" "d:D:r:u[!dr]"
dd " "  " "
deallocvt " " ">1"
demo_many_options " " "ZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjihgfedcba"
demo_number " " "hdbs"
demo_scankey " "  " "
demo_utf8towc " "  " "
df "HPkhit*a[-HPkh]" "HPkhit*a[-HPkh]"
dhcp " " "V:H:F:x*r:O*A#<0=20T#<0=3t#<0=3s:p:i:SBRCaovqnbf"
dhcp6 " " "r:A#<0T#<0t#<0s:p:i:SRvqnbf"
dhcpd " " ">1P#<0>65535fi:S46[!46]"
diff " " "<2>2(color)B(ignore-blank-lines)d(minimal)b(ignore-space-change)ut(expand-tabs)w(ignore-all-space)i(ignore-case)T(initial-tab)s(report-identical-files)q(brief)a(text)L(label)*S(starting-file):N(new-file)r(recursive)U(unified)#<0=3"
dirname "<1" "<1"
dmesg "w(follow)CSTtrs#<1n#c[!Ttr][!Cc][!Sw]" "w(follow)CSTtrs#<1n#c[!Ttr][!Cc][!Sw]"
dos2unix  " "  " "
du "d#<0=-1hmlcaHkKLsx[-HL][-kKmh]" "d#<0=-1hmlcaHkKLsx[-HL][-kKmh]"
dumpleases " " ">0arf:[!ar]"
echo "^?en" "^?en"
eject ">1stT[!tT]" ">1stT[!tT]"
env "^0iu*" "^0iu*"
exit " "  " "
expand "t*" "t*"
expr " "  " "
factor  " "  " "
fallocate ">1l#|o#" ">1l#|o#"
false  " "  " "
fdisk " " "C#<0H#<0S#<0b#<512ul"
file "<1hL[!hL]" "<1hL[!hL]"
find "?^HL[-HL]" "?^HL[-HL]"
flock "<1>1nsux[-sux]" "<1>1nsux[-sux]"
fmt "w#<0=75" "w#<0=75"
fold " " "bsuw#<1"
free "htgmkb[!htgmkb]" "htgmkb[!htgmkb]"
freeramdisk "<1>1" "<1>1"
fsck " " "?t:ANPRTVsC#"
fsfreeze "<1>1f|u|[!fu]" "<1>1f|u|[!fu]"
fstype "<1" "<1"
fsync "<1d" "<1d"
ftpget "<2>3P:cp:u:vgslLmMdD[-gs][!gslLmMdD][!clL]" "<2>3P:cp:u:vgslLmMdD[-gs][!gslLmMdD][!clL]"
getconf ">2al" ">2al"
getenforce " " ">0"
getfattr " " "dhn:"
getprop " " ">2Z"
getty " " "<2t#<0H:I:l:f:iwnmLh"
grep "(color):;S(exclude)*M(include)*ZzEFHIabhinorsvwclqe*f*C#B#A#m#x[!wx][!EFw]" "(color):;S(exclude)*M(include)*ZzEFHIabhinorsvwclqe*f*C#B#A#m#x[!wx][!EFw]"
groupadd " " "<1>2g#<0S"
groupdel " " "<1>2"
groups  " "  " "
gunzip " " "cdfk123456789[-123456789]"
gzip " " "cdfk123456789[-123456789]"
head "?n(lines)#<0=10c(bytes)#<0qv[-nc]" "?n(lines)#<0=10c(bytes)#<0qv[-nc]"
hello " "  " "
help "ah" "ah"
hexedit "<1>1r" "<1>1r"
host " " "<1>2avt:"
hostid " " ">0"
hostname ">1bdsfF:[!bdsf]" ">1bdsfF:[!bdsf]"
hwclock ">0(fast)f(rtc):u(utc)l(localtime)t(systz)s(hctosys)r(show)w(systohc)[-ul][!rtsw]" ">0(fast)f(rtc):u(utc)l(localtime)t(systz)s(hctosys)r(show)w(systohc)[-ul][!rtsw]"
i2cdetect ">3aFly" ">3aFly"
i2cdump "<2>2fy" "<2>2fy"
i2cget "<3>3fy" "<3>3fy"
i2cset "<4fy" "<4fy"
iconv "cst:f:" "cst:f:"
id ">1nGgru[!Ggu]" ">1ZnGgru[!ZGgu]"
ifconfig "^?aS" "^?aS"
init " "  " "
inotifyd "<2" "<2"
insmod "<1" "<1"
install "<1cdDpsvm:o:g:" "<1cdDpsvm:o:g:"
ionice "^tc#<0>3=2n#<0>7=5p#" "^tc#<0>3=2n#<0>7=5p#"
iorenice "?<1>3" "?<1>3"
iotop ">0AaKOHk*o*p*u*s#<1=7d%<100=3000m#n#<1bq" ">0AaKOHk*o*p*u*s#<1=7d%<100=3000m#n#<1bq"
ip " "  " "
ipcrm " " "m*M*s*S*q*Q*"
ipcs " " "acptulsqmi#"
kill "?ls: " "?ls: "
killall "?s:lqvi" "?s:lqvi"
killall5 "?o*ls: [!lo][!ls]" "?o*ls: [!lo][!ls]"
klogd " " "c#<1>8n"
last " " "f:W"
link "<2>2" "<2>2"
ln "<1vnfs" "<1vnfs"
load_policy " " "<1>1"
log " " "<1p:t:"
logger "st:p:" "st:p:"
login ">1f:ph:" ">1f:ph:"
logname ">0" ">0"
losetup ">2S(sizelimit)#s(show)ro#j:fdca[!afj]" ">2S(sizelimit)#s(show)ro#j:fdca[!afj]"
ls "(color):;(full-time)(show-control-chars)ZgoACFHLRSabcdfhikl at mnpqrstux1[-Cxm1][-Cxml][-Cxmo][-Cxmg][-cu][-ftS][-HL][!qb]" "(color):;(full-time)(show-control-chars)ZgoACFHLRSabcdfhikl at mnpqrstux1[-Cxm1][-Cxml][-Cxmo][-Cxmg][-cu][-ftS][-HL][!qb]"
lsattr "vldaR" "vldaR"
lsmod  " "  " "
lsof " " "lp*t"
lspci "emkn at i:" "emkn at i:"
lsusb  " "  " "
makedevs "<1>1d:" "<1>1d:"
md5sum "bc(check)s(status)[!bc]" "bc(check)s(status)[!bc]"
mdev " " "s"
microcom "<1>1s:X" "<1>1s:X"
mix "c:d:l#r#" "c:d:l#r#"
mkdir "<1vp(parent)(parents)m:" "<1Z:vp(parent)(parents)m:"
mke2fs " " "<1>2g:Fnqm#N#i#b#"
mkfifo "<1m:" "<1Z:m:"
mknod "<2>4m(mode):" "<2>4m(mode):Z:"
mkpasswd ">2S:m:P#=0<0" ">2S:m:P#=0<0"
mkswap "<1>1L:" "<1>1L:"
mktemp ">1uqd(directory)p(tmpdir):t" ">1uqd(directory)p(tmpdir):t"
modinfo "<1b:k:F:0" "<1b:k:F:0"
modprobe " " "alrqvsDbd*"
more " "  " "
mount "?O:afnrvwt:o*[-rw]" "?O:afnrvwt:o*[-rw]"
mountpoint "<1qdx[-dx]" "<1qdx[-dx]"
mv "<2vnF(remove-destination)fi[-ni]" "<2vnF(remove-destination)fi[-ni]"
nbd_client "<3>3ns" "<3>3ns"
netcat "^tlLw#<1W#<1p#<1>65535q#<1s:f:46u[!tlL][!Lw][!46]" "^tlLw#<1W#<1p#<1>65535q#<1s:f:46u[!tlL][!Lw][!46]"
netstat "pWrxwutneal" "pWrxwutneal"
nice "^<1n#" "^<1n#"
nl "v#<1=1l#w#<0=6Eb:n:s:" "v#<1=1l#w#<0=6Eb:n:s:"
nohup "<1^" "<1^"
nproc "(all)" "(all)"
nsenter " " "<1F(no-fork)t#<1(target)i:(ipc);m:(mount);n:(net);p:(pid);u:(uts);U:(user);"
od "j#vw#<1=16N#xsodcbA:t*" "j#vw#<1=16N#xsodcbA:t*"
oneit "^<1nc:p3[!pn]" "^<1nc:p3[!pn]"
openvt " " "c#<1>63sw"
partprobe "<1" "<1"
passwd ">1a:dlu" ">1a:dlu"
paste "d:s" "d:s"
patch "(dry-run)ulp#d:i:R" "(dry-run)xulp#d:i:R"
pgrep "?cld:u*U*t*s*P*g*G*fnovxL:[-no]" "?cld:u*U*t*s*P*g*G*fnovxL:[-no]"
pidof "<1so:" "<1so:"
ping "<1>1m#t#<0>255=64c#<0=3s#<0>4088=56i%W#<0=3w#<0qf46I:[-46]" "<1>1m#t#<0>255=64c#<0=3s#<0>4088=56i%W#<0=3w#<0qf46I:[-46]"
pivot_root "<2>2" "<2>2"
pkill "?Vu*U*t*s*P*g*G*fnovxl:[-no]" "?Vu*U*t*s*P*g*G*fnovxl:[-no]"
pmap "<1xq" "<1xq"
printenv "0(null)" "0(null)"
printf "<1?^" "<1?^"
ps "k(sort)*P(ppid)*aAdeflMno*O*p(pid)*s*t*Tu*U*g*G*wZ[!ol][+Ae][!oO]" "k(sort)*P(ppid)*aAdeflMno*O*p(pid)*s*t*Tu*U*g*G*wZ[!ol][+Ae][!oO]"
pwd ">0LP[-LP]" ">0LP[-LP]"
pwdx "<1a" "<1a"
readahead  " "  " "
readlink "<1>1nqmef(canonicalize)[-mef]" "<1>1nqmef(canonicalize)[-mef]"
realpath "<1" "<1"
reboot "fn" "fn"
renice "<1gpun#|" "<1gpun#|"
reset  " "  " "
restorecon " " "<1DFnRrv"
rev  " "  " "
rfkill "<1>2" "<1>2"
rm "fiRr[-fi]" "fiRr[-fi]"
rmdir "<1p" "<1p"
rmmod "<1wf" "<1wf"
route " " "?neA:"
runcon " " "<2"
sed "(help)(version)e*f*i:;nEr[+Er]" "(help)(version)e*f*i:;nEr[+Er]"
sendevent " " "<4>4"
seq "<1>3?f:s:w[!fw]" "<1>3?f:s:w[!fw]"
setenforce " " "<1>1"
setfattr "hn:|v:x:|[!xv]" "hn:|v:x:|[!xv]"
setprop " " "<2>2"
setsid "^<1t" "^<1t"
sh " " "c:i"
sha1sum "bc(check)s(status)[!bc]" "bc(check)s(status)[!bc]"
shred "<1zxus#<1n#<1o#<0f" "<1zxus#<1n#<1o#<0f"
skeleton " " "(walrus)(blubber):;(also):e at d*c#b:a"
skeleton_alias " " "b#dq"
sleep "<1" "<1"
sort "gS:T:mo:k*t:xVbMcszdfirun" "gS:T:mo:k*t:xVbMcszdfirun"
split ">2a#<1=2>9b#<1l#<1[!bl]" ">2a#<1=2>9b#<1l#<1[!bl]"
start " "  " "
stat "<1c:fLt" "<1c:fLt"
stop " "  " "
strings "t:an#=4<1fo" "t:an#=4<1fo"
stty " " "?aF:g[!ag]"
su "lmpc:s:" "lmpc:s:"
sulogin " " "t#<0=0"
swapoff "<1>1" "<1>1"
swapon "<1>1p#<0>32767d" "<1>1p#<0>32767d"
switch_root "<2c:h" "<2c:h"
sync  " "  " "
sysctl "^neNqwpaA[!ap][!aq][!aw][+aA]" "^neNqwpaA[!ap][!aq][!aw][+aA]"
syslogd " " ">0l#<1>8=8R:b#<0>99=1s#<0=200m#<0>71582787=20O:p:f:a:nSKLD"
tac  " "  " "
tail "?fc-n-[-cn]" "?fc-n-[-cn]"
tar " " "&(no-recursion)(numeric-owner)(no-same-permissions)(overwrite)(exclude)*(to-command):o(no-same-owner)p(same-permissions)k(keep-old)c(create)|h(dereference)x(extract)|t(list)|v(verbose)j(bzip2)z(gzip)O(to-stdout)m(touch)X(exclude-from)*T(files-from)*C(directory):f(file):[!txc][!jz]"
taskset "<1^pa" "<1^pa"
tcpsvd " " "^<3c#=30<1C:b#=20<0u:l:hEv"
tee "ia" "ia"
telnet " " "<1>2"
telnetd " " "w#<0b:p#<0>65535=23f:l:FSKi[!wi]"
test  " "  " "
tftp " " "<1b#<8>65464r:l:g|p|[!gp]"
tftpd " " "rcu:l"
time "<1^pv" "<1^pv"
timeout "<2^vk:s: " "<2^vk:s: "
top ">0O*Hk*o*p*u*s#<1d%<100=3000m#n#<1bq[!oO]" ">0O*Hk*o*p*u*s#<1d%<100=3000m#n#<1bq[!oO]"
touch "<1acd:fmr:t:h[!dtr]" "<1acd:fmr:t:h[!dtr]"
toybox  " "  " "
tr " " "^>2<1Ccsd[+cC]"
traceroute " " "<1>2i:f#<1>255=1z#<0>86400=0g*w#<0>86400=5t#<0>255=0s:q#<1>255=3p#<1>65535=33434m#<1>255=30rvndlIUF64"
true  " "  " "
truncate "<1s:|c" "<1s:|c"
tty "s" "s"
tunctl "<1>1t|d|u:T[!td]" "<1>1t|d|u:T[!td]"
ulimit ">1P#<1SHavutsrRqpnmlifedc[-SH][!apvutsrRqnmlifedc]" ">1P#<1SHavutsrRqpnmlifedc[-SH][!apvutsrRqnmlifedc]"
umount "cndDflrat*v[!na]" "cndDflrat*v[!na]"
uname "oamvrns[+os]" "oamvrns[+os]"
uniq "f#s#w#zicdu" "f#s#w#zicdu"
unix2dos  " "  " "
unlink "<1>1" "<1>1"
unshare " " "<1^f(fork);r(map-root-user);i:(ipc);m:(mount);n:(net);p:(pid);u:(uts);U:(user);"
uptime ">0ps" ">0ps"
useradd " " "<1>2u#<0G:s:g:h:SDH"
userdel " " "<1>1r"
usleep "<1" "<1"
uudecode ">1o:" ">1o:"
uuencode "<1>2m" "<1>2m"
uuidgen ">0r(random)" ">0r(random)"
vconfig "<2>4" "<2>4"
vi " " "<1>1"
vmstat ">2n" ">2n"
w  " "  " "
watch "^<1n%<100=2000tebx" "^<1n%<100=2000tebx"
wc "mcwl" "mcwl"
wget " " "f:"
which "<1a" "<1a"
who "a" "a"
xargs "^I:E:L#ptxrn#<1s#0[!0E]" "^I:E:L#ptxrn#<1s#0[!0E]"
xxd ">1c#l#o#g#<1=2iprs#[!rs]" ">1c#l#o#g#<1=2iprs#[!rs]"
xzcat " "  " "
yes  " "  " "
zcat " " "cdfk123456789[-123456789]"