[xquery-talk] Query 3.1 vs. JSONiq WAS Re: MarkLogic using JSONiq for processing JSON ?

[Michael Kay]

That’s the standardisation process, unfortunately. It’s very rare for an existing design to be simply rubber-stamped by a standards committee. When SQL was standardized, ANSI didn’t simply rubber-stamp what IBM had done in System R.

What happens when you take a spec like JSONiq to a standards committee is that it gets scrutiny from a wider audience, some of whom want to address requirements that were outside the original scope. For some people, adding maps and arrays to XDM was more about increasing the expressive power of the language than specifically about supporting JSON; and the requirements for maps and arrays came from use cases other than JSON.

> 
> 
> JSONiq is 3 years old, implemented by a variety of systems, and used in production in many places. And it was very well
> designed: 2 years of VERY CAREFUL design by very good engineers went into the design of JSONiq. 

I agree, it was an excellent piece of work, that did what it set out to do extremely well.
> 
> Plus JSONiq will CONTINUE to be used widely in the NOSQL community, simply for the following two reasons:
> (a) it has a JSON-only subset that makes sense for the JSON-only community (who in general doesn’t want to see any shadow of angle brackets) and
> (b) has the great advantage that doesn’t have the hated word “Xquery” in its name.

You may or may not be right, time will tell. I think it’s fairly inevitable that the user community addressed by the XQuery standards group will be the community that likes XQuery, not the community that hates it.
> 
> 
> By not following JSONiq, by taking “slightly different” choices, or by not trying to extend it, XQuery 3.1 just broke the interoperability
> in the NOSQL community, and this is what I consider a total lack of vision from the “leaders” of this community.

There were those on the committee who strongly defended the design choices that JSONiq had made. But no-one on the XQuery WG felt that the objective was to put a W3C stamp on work that had already been completed elsewhere.

Remember that JSONiq was not the only input to this work. The XSLT Working Group added maps to the XDM data model during 2010/2011 because they were needed for streaming use cases. During 2012 we become aware of the JSONiq work and attempted to bring the ideas into line; but we never considered dropping a feature (like using dates as keys in a map, or nodes as values) just because it wasn’t needed for JSON support.

> 
> Let’s look again of the differences you mentioned between JSONiq and XQuery 3.1. I quote:
> ***********************************************
> At the data model level:
> 
> * Maps can use any atomic value as the key, it does not have to be a string
> * The members of an array are sequences, not necessarily items
> * JSON’s null is represented as an empty sequence, not as a new atomic data type
> 
> At the syntax level:
> 
> * In XQ3.1, Map constructors use the syntax map{ a:b, c:d } rather than bare curlies
> * XQ 3.1 introduces a lookup operator for maps and arrays: employee?name, or book?author?1
> **********************************************
> 
> 
> Now, dear Michael, ask yourself the following questions:
> 
> 
> *********************************************
> 1. Was there any “mistake”, or “bug”, that was in JSONiq that those choices solved ?  
> I am not aware of any.

The differences were not motivated by fixing mistakes or bugs, but by addressing a wider range of use cases.
> 
> 2. Are any of those choices “fundamental’, aka introduce a much larger expressive power, make the language much easier to use, etc.?
> Anything fundamental in those “new” choices ? 
> Nope, they are just small, mostly insignificant, just enough itsy bitsy differences to confuse and irritate the entire NoSQL community.

I think the generalization of maps and arrays is rather fundamental, in that it increases the orthogonality of the data model. I cited the example of fn:apply(), which takes two arguments, a function and an array of arguments. Since each argument to a function is a sequence, we need a data structure which is an array (or sequence) of sequences, and limiting arrays to contain only items would have considerably reduced their power.

> 
> 3. In case where a larger expressive power was desired, wasn’t it better to extend JSONiq in an upper compatible way, rather to take a different route ?

Most of these differences were in fact compatible. Some were not, like using the map{} keyword. That was a carefully debated decision, but I don’t think an argument of compatibility with JSONiq would have carried much weight: for better or for worse, standardizing JSONiq wasn’t the group’s charter.
> 
> 4. Did XQuery 3.1 “committee” ever tried to contact the JSONiq community to try to find common solution that would eventual lead to a single, better language ?

I think that the JSONiq developers were well represented in the discussions and argued their case vigorously, and often successfully. The XSL WG made a lot of changes/concessions during this process: in fact, everyone did.

> ALL decisions we took in the design of JSONiq were definitely not random, but VERY carefully thought out. You were not even curious of what those reason where,

No, that’s not true. I don’t know whether the “you” here is referring to me personally, or to the group as a whole, but in both cases, it’s not true. Many arguments were aired in great detail and were heard with patience by all concerned.


> (and sometimes XQuery 3.1 got it totally wrong — I an write you a long email about THAT).
> *********************************************
> 
> So: no bugs to be fixed, no new major technical advantages, and no desire to cooperate.

Getting to the point where we got to was not easy. There were passionate disagreements. The fact that we got a spec out is proof that everyone was trying to cooperate. Inevitably, not everyone will be happy with every decision that was made. Those who took part in the process are probably happier than those who did not, because they can see what the arguments were.

> 
> What conclusion do you get from here ?
> 
> I get a single conclusion: the XQuery 3.1 committee has a total lack of vision and leadership in the NoSQL world. 

Well you might be right there. Standards groups don’t really do vision and leadership: they argue about braces and semicolons.

> 
> This can has two possible explanations: pure stupidity and lack of good will.

No, I think it’s simply the result of having a variety of perspectives. Standards groups bring people together who have different ideas about what it is important to achieve, and the result always involves compromises, which not everyone will be happy with.
> 

Regards,
Michael Kay
Saxonica