[xquery-talk] BumbleBee XQuery Test Harness

[Michael Kay]

I have finally got around to downloading this and trying it out. Thanks,
you've done a good job, and it's very nicely packaged.

The vast majority of Saxon failures on the NIST tests, as far as I can
see, fall into one of three categories:

(a) facilities documented as being unimplemented (e.g. integers > 2^63)
(b) tests where the spec is unclear or changeable on what the result
should be
    (e.g. "12.0" versus "12" as the output)
(c) tests where the reference results seem to be plain wrong, e.g.

Query: 
<results>
       {-(xs:decimal("-99999999999999999999"))}
      </results>

Actual Result: 
<results>99999999999999999999</results>

Expected Result: 
ERROR

I wonder where these expected results came from? And how do we move
forward now, to keep the tests and the expected results accurate against
the spec?

Michael Kay



> -----Original Message-----
> From: talk-bounces at xquery.com 
> [mailto:talk-bounces at xquery.com] On Behalf Of Jason Hunter
> Sent: 14 October 2003 23:35
> To: talk at xquery.com
> Subject: [xquery-talk] BumbleBee XQuery Test Harness
> 
> 
> Hi everyone,
> 
> I'm proud to announce here a product I've helped develop: the 
> BumbleBee 
> XQuery test harness.  It's an automated test harness that you can use 
> both for evaluating XQuery engines (seeing which vendor is more spec 
> compliant and scalable) and for validating your own queries 
> (making sure 
> your queries continue to work as you upgrade or change engines).
> 
> BumbleBee -- named because it buzzes around FLWORs -- has the 
> following 
> features:
> 
> * Includes over 70 automated tests representative of the W3C XQuery
>    Use Cases
> * Includes over 1000 automated tests representative of the NIST
>    XQuery Test Suite
> * Includes adapters to 6 different free and commercial XQuery engines
> * Includes the runtime libraries for 3 free XQuery engines
> * Allows you to write and run your own custom XQuery tests
> * Tests are automated, self checking, and report an unambiguous
>    pass or fail status
> * Tests can define multiple, possible expected results
> * Can run arbitrary suites of standard and/or custom tests
> * Can run tests against one or more XQuery engines
> * Generates a comprehensive report of passed and failed tests
>    per XQuery engine being tested
> * Supports the addition of new XQuery engines through a pluggable
>    adapter design
> * Allows disabling and enabling tests, globally or on a per-engine
>    basis
> * Allows customization of whitespace comparison behavior
> * Supports negative testing, where the engine must report an error
>    to pass
> * Written in Java for maximum portability
> 
> A free, no-login-required, fully-functioning 30-day 
> evaluation copy of 
> BumbleBee and the "Getting Started With BumbleBee" 
> quick-start guide is 
> available at:
> 
     http://xquery.com/bumblebee

Licensed copies of BumbleBee are available free of charge for developers

of open source XQuery implementations, and for qualified non-profit and 
educational use.  Licenses of BumbleBee are available for commercial use

by contacting "buzz" at xquery.com with your usage requirements.

I hope that this mailing list will host some discussion about the tool 
and what it discovers about engine quality, so if you have any thoughts 
please write in.

-jh-


_______________________________________________
talk at xquery.com
http://www.x-query.com/mailman/listinfo/talk