[xquery-talk] Is it possible to maintain a list of value in XQuery

[David Lee]

>>> Liam

I'm not sure I'm failing to follow you incorrectly here.
------  Someone is failing somethign :)_

Yes, declarative languages, especially with referential transparency,
are often massively easier to optimize than procedural ones. Or rather,
the computer science and mathematics might be harder in some cases, but
the optimizations can be more far-reaching. The down side is that they
tend to start out slower, so it doesn't always balance out. *BUT* they
are also often more maintainable.
-----------

This is the "Promised Land of Milk and Honey" that I have been told about ...
And I can believe it in some cases.  
What I boldly propose out of ignorance is that in reality its somewhat (mostly?) BS.
Like the class I took in collage about proving computer programs ... it only worked
if your program was about 3 lines or less beyond that it was *effectively* impossible.
I know many kinds of optimizations are possible (and yea have been achieved) with purely functional languages.
Query Optimization of SQL has some very deep proof of this and I suspect that may have been the rational
behind XQuery and XSLT.   But in reality ... at least the reality I have absorbed by attending conferences,
reading papers, talking with implementers, and delving deeply into the guts of real world XQuery and XSLT processors ....
that reality has come up short from the promise.   There are a huge range of *possible* optimizations that
simply never get achieved.    Its *HARD* ... I am not blaming the implementers.    There is Theory  and then there is Engineering.  (and lets not forget time and money).
One example is parallelism.   A purely functional language should be able to be highly paralyzed automatically.
But what we have seen, is that few vendors do this at all, or very little.  
 I know from Mr Kay's papers he has had some success,
but largely by introducing vendor (and now some standard) specific "hints" into the language to allow for points where parallism can be recognized  and he himself has stated that reality has not lived up to the ideal.
The comment about XQuery and cache hits I propose is theory not practice ... unless one implements XQuery in machine 
language *directly* (the implementations I have seen are interpretive or late compiled to bytecode ... ), 
then the illusion that xquery statements will actually fit snugly in cachelines is an illusion.   Now if someone does write a direct XQuery to machine code compiler that might be valid ... maybe there is one, would love to see the paper at Balisage.
So what are the languages people write XQuery compilers in *really* ? What I have seen is C, C++ and Java.  I am sure there are more.   And *those* implementations are highly optimized using non-functional techniques.   So in summary I just dont buy the "functional languages inherantly optimize better" angle.   Maybe in theory ... and maybe in some select cases, but not globally in practice.  
Global *mutable* variables are another example.   To avoid them 100% can be very difficult and results in code that in theory should be optimized but in practice is often not. And can be very "ugly" by some standards.
 Adding the ability to "open a crack" from the functional to procedural
allows users to achieve optimizations simply not practical by the optimizer.  Or atleast be able to write code that they can understand.    This is why I am seriously suggesting that  XQuery and XSLT (and MAYBE XPath ... but not so convinced on xpath) be opened up *philosophically* to non-functional paradigms.   Maybe not the whole floodgate, but bits.
The bits that vendors have found necessary to introduce ... Yes in theory this lets the cat out of the bag, and the camels nose in the tent and cats will sleep with dogs and the world will end .... But on the other hand maybe just a few carefully chosen idioms that "brake the shackles of pure functional programming"  could be a good thing.  For one maybe it would encourage vendors to switch over from their vendor specific holes and instead use the standard ones thus making XQuery and XSLT more portable do agree with a layered approach.   You dont have to sink the entire ship with this, you can *document* where use of such a bastard might unduly affect the optimizer and let the user decide.   
For Layered Approaches, XQuery scripting is one,  xproc another, xmlsh another.
But I suggest the layer is too high.   I suggest XQuery has grown in use to more complexity then possibly imagined at the beginning.  Whole apps are being written in XQuery alone.   So maybe the layer should be a little deeper *into* the language instead of above of.  ( This is why I am shy of suggesting xpath itself break the mold of purely functional,
it might actually be the layer where pure functional ideology works).

 I know from a standards committee perspective this is a nearly impossible task ... so instead I just rant on a mailing list.  Maybe someone will see the view ... or intelligently disagree, or just think about it for a second before abandoning it as the ramblings of an insane apostate  ... or maybe I will just be beat to a pulp.

All in all a good day :)

-David