[xquery-talk] Re: The State of Native XML databases

[Ryan Grimm]

Frank, it sounds like you have some pretty high expectations for a 
fairly young community (compared to relational databases).  Don't get me 
wrong, I'm a fan of high expectations but I'm also a fan of putting 
things into perspective.

It sounds like you'd like the xml database world to have the maturity, 
history and community that relational databases have.  I'd love this as 
well but relational theory has been around for over 35 years and I 
believe Oracle itself is 30 years old.  When I look at how much progress 
the xml database world has made in the last five years I'm very proud of 
it and excited to see what it will be like five years from now.

That being said you do raise a few good points.  I agree that the 
community could use more tools and shared libraries.  But there are 
tools out there.  I know that the oXygen xml editor has an xQuery 
debugger and can directly connect to the MarkLogic database.  I believe 
some people are working on various Eclipse plugins as well.

I'm also a big fan of improving our community.  I've thought about 
starting up a XQuery user group or something in silicon valley on a 
number of occasions and if people are interested let me know.  As for 
conferences I heard last year's XML 2006 conference was fantastic and 
the call for participation for this years conference ends on the 31st so 
you've still got some time :).

I'm interested in your statement about the difficulties for developers 
to go from the XML data model to an XML database.  I see in some other 
posts that you talk about things like the lack of an update mechanism in 
XQuery.  I personally don't see this as a deal breaker.  The vendors 
that need such mechanisms have built them.  It would be nice to have a 
standard way of performing them (which looks like it's coming) but the 
notion that all xml databases will (or even should) be 100% 
interoperable is probably unrealistic.  Even with a simple language like 
SQL all the implementations differ.  Some things like being able to 
limit the number of results aren't even a part of the specification but 
are needed for many applications and vendors extend things in order to 
fill those needs.  So by that measure even relational databases aren't 
in an "excellent" state :).

As for thought leaders and practical applications, we can always use 
more of them.  There are a number of XQuery driven applications backed 
by xml databases out there used in production systems.  I think these 
applications qualify as being practical but many of them are not 
publicly visible.  I personally spent two years working at O'Reilly on 
XQuery applications using the MarkLogic database.  One of the 
applications was SafariU (www.safariu.com, pretty much closed to the 
general public).  But in my spare time I worked on little applications 
that ended up being thrown on their labs site (labs.oreilly.com).  The 
site is 100% XQuery driven and was my little playground to see what I 
could do with this new technology, needless to say I had fun :).  I 
would love to get a list of products and sites that use our various 
technologies to help people understand what this new world can offer them.

--Ryan


Frank Cohen wrote:
> For a topic titled "The State of Native XML Databases" the conversation 
> seems to be more about semantics. From my perspective the state of 
> native XML databases is pretty poor. There is not that much uptake on 
> native XML databases from enterprises, developers, and architects.
> 
> It seems to me that native XML databases would be in a much better state 
> when the following happen:
> 
> 1) The Implementations Provide The Full Benefits of The Data Model.
> 
> The XML data model is wonderfully rich and flexible. There is big divide 
> that developers have to overcome when going from the XML data model to 
> an XML database. It is not a given that XQuery is way to implement a 
> database for the XML data model. When I was at the W3C Plenary 2 years 
> ago there was plenty of energy in discussing alternatives to XQuery and 
> XSLT. And I would prefer to write code today, not wait another 7 years 
> for a standard like XQuery to emerge.
> 
> It seems to me that developer marketing from the native XML database 
> vendors and open-source projects is mostly disingenuous when they 
> position their products as a general-purpose database. In my experience 
> most of the native XML database projects come to the XML data model from 
> a different perspective and that impacts their implementation of the 
> data model as an XML database. For instance, there is still argument on 
> this list about the best way to create a collection with one perspective 
> using XML Schema and others arguing for an language semantics approach. 
> (By the way, neither work for me as a Java developer. I want to from 
> object definitions and maybe Java annotations to a collection.)
> 
> As a developer I would prefer to know which native XML database is 
> suited to my application. For instance, Ilya's application requires 
> node-level locking and my design for the next generation of TestMaker 
> requires roles-based securing at the XML document level. In the RDBMS 
> marketplace I have many resources to show me which databases are 
> appropriate for my applications. The native XML database marketing needs 
> something similar.
> 
> 2) The lack of thought leaders
> 
> Where are the Tim Bray's, the Adam Bosworth's, and Erik Meijer's, the 
> Sam Ramji's for the native XML database space? I had great hope for 
> Jason Hunter but even his star is taking a while to cross over from the 
> Java to the native XML db space.
> 
> The native XML database space needs someone of reputation and standing 
> in the Java and .NET developer community to stand-up and show some great 
> and practical applications.
> 
> 3) The lack of tools
> 
> I like some of the tools I have seen. The Progress tools set is nice. I 
> would love to see native XML database support with a flexible binding 
> system in NetBeans, in Visual Studio, Eclipse, IntelliJ.
> 
> 4) The lack of a clubhouse
> 
> I want a place to meet up with all of you! JavaOne, JavaPolis, JAOO, and 
> QCon are my place to meet up, share ideas, and exchange solutions in the 
> Java space. I would love to have XMLOne once a year!
> 
> I spent 2005-6 inside Raining Data and know first hand what it is like 
> to ask a quarter-by-quarter focused management team to invest in 
> developer programs.
> 
> A lot of work is still needed to move the state of native XML databases 
> to "excellent".
> 
> -Frank