[seqfan] Some observations of testing programs (was metadata for Mathematica code)

[Sean A. Irvine]

>Note that comparing terms to a program has a problem: some of the
>programs run much too slowly; the test will never finish. Even for
>sequences where a fast algorithm is known (and hopefully has a
>function implementing it), there may be other programs present that
>are not efficient. So you can't do any sort of automatic scan, nor
>unconditionally test new programs.


I have some numbers to back up Frank's comment about testing programs,
although not for Mathematica.  I have Java implementations for 10,192
sequences in the OEIS concentrated mainly on those with sequence ids
under A004100.  On a daily basis, I test all these programs against
the data in the stripped sequences file (which I download about once a
month).  Of these, in 396 cases, I test a subset of the available
terms due to the time or memory it would take to test all available
terms.  There is some bias here towards sequences more easily
implemented.  But by my reckoning this means that about 4-5% of
programs will not easily be testable.  Also, I have the advantage that
all my implementations have a consistent interface (I only support a
method next() which sequentially generates the next term of a
sequence, there is no support for asking for a specific term).

To my mind the biggest advantage to automated testing of programs is
that it is a good way of identifying errors in the actual sequence
data.  Many of my edits to the OEIS have been the result of
discovering that implementation does not produce the sequence.
Obviously care needs to be taken it working out if it is the program
or sequence that is wrong.  These observations are perhaps less true
of newer sequences, many of the sequences I have looked at are the
older ones where there have been typos in either the original papers
or less often in the copying of the sequence into the OEIS.

Aside from detecting problems when I first implement a sequence, the
next most common problem is when a change occurs to the initial terms
of a sequence.  That is, over time it seems fairly common that someone
might add an a(0) term to a sequence that didn't previously have one.
Without some form of automated testing, programs could become stale in
this way -- although the fix is usually trivial.  This problem can be
exacerbated when an implementation of a sequence depends on other
sequences.

Sean.