[xquery-talk] BumbleBee XQuery Test Harness
jhunter at servlets.com
Sun Oct 26 18:11:02 PST 2003
Michael Kay wrote:
> I have finally got around to downloading this and trying it out. Thanks,
> you've done a good job, and it's very nicely packaged.
Thanks, Michael. I'm glad you're finding it useful.
> The vast majority of Saxon failures on the NIST tests, as far as I can
> see, fall into one of three categories:
> (a) facilities documented as being unimplemented (e.g. integers > 2^63)
> (b) tests where the spec is unclear or changeable on what the result
> should be
> (e.g. "12.0" versus "12" as the output)
> (c) tests where the reference results seem to be plain wrong, e.g.
> Actual Result:
> Expected Result:
> I wonder where these expected results came from?
As you probably know, the tests from NIST contain queries but no
results. We used tools and human verification to generate complete
scripts. We added comments in the .bee files about known shortcomings.
> And how do we move
> forward now, to keep the tests and the expected results accurate against
> the spec?
We're working with Carmelo (who posted here earlier) so that his future
NIST tests will have results as well as queries.
>>From: talk-bounces at xquery.com
>>[mailto:talk-bounces at xquery.com] On Behalf Of Jason Hunter
>>Sent: 14 October 2003 23:35
>>To: talk at xquery.com
>>Subject: [xquery-talk] BumbleBee XQuery Test Harness
>>I'm proud to announce here a product I've helped develop: the
>>XQuery test harness. It's an automated test harness that you can use
>>both for evaluating XQuery engines (seeing which vendor is more spec
>>compliant and scalable) and for validating your own queries
>>your queries continue to work as you upgrade or change engines).
>>BumbleBee -- named because it buzzes around FLWORs -- has the
>>* Includes over 70 automated tests representative of the W3C XQuery
>> Use Cases
>>* Includes over 1000 automated tests representative of the NIST
>> XQuery Test Suite
>>* Includes adapters to 6 different free and commercial XQuery engines
>>* Includes the runtime libraries for 3 free XQuery engines
>>* Allows you to write and run your own custom XQuery tests
>>* Tests are automated, self checking, and report an unambiguous
>> pass or fail status
>>* Tests can define multiple, possible expected results
>>* Can run arbitrary suites of standard and/or custom tests
>>* Can run tests against one or more XQuery engines
>>* Generates a comprehensive report of passed and failed tests
>> per XQuery engine being tested
>>* Supports the addition of new XQuery engines through a pluggable
>> adapter design
>>* Allows disabling and enabling tests, globally or on a per-engine
>>* Allows customization of whitespace comparison behavior
>>* Supports negative testing, where the engine must report an error
>> to pass
>>* Written in Java for maximum portability
>>A free, no-login-required, fully-functioning 30-day
>>evaluation copy of
>>BumbleBee and the "Getting Started With BumbleBee"
>>quick-start guide is
> Licensed copies of BumbleBee are available free of charge for developers
> of open source XQuery implementations, and for qualified non-profit and
> educational use. Licenses of BumbleBee are available for commercial use
> by contacting "buzz" at xquery.com with your usage requirements.
> I hope that this mailing list will host some discussion about the tool
> and what it discovers about engine quality, so if you have any thoughts
> please write in.
> talk at xquery.com
More information about the talk