<p dir="ltr">I seriously suggest that you email the XQuery Working Group with your use case. In particular you should contact Ghislain Fourny and Jonathan Robie, as both have been working on and gathering use cases for arrays in XQuery 3.1. So far efficient matrix operations is not one that I have seen.</p>
<p dir="ltr">Cheers Adam.</p>
<div class="gmail_quote">On 31 Dec 2013 17:03, "jean-marc Mercier" <<a href="mailto:jeanmarc.mercier@gmail.com">jeanmarc.mercier@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">@David pairs are also basically needed to write a linear algebra modulus, the topic of this thread. And XQUERY don't provide any efficient pair. You can't use Marklogic map, or any other vendor map to store vectors for performance issues (a map is really slow).<div>
<br></div><div>Note that there are a lot of workaround : I am using direct JAVA binding or C++ binding from XQUERY for linear algebra not to pay a too heavy tribute to these issue performances.</div><div><br></div><div>The point is simply to notice that XQUERY could be really good to write an efficient linear algebra modulus. But, due to these performance issues, nobody can write it. I just hope that the next XQUERY version will give the necessary container to write it. Meanwhile, nobody can contribute to XQUERY through external modulus using heavy algorithms, and that's just too bad.</div>
<div><br></div><div><br></div><div><br><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/12/31 David Lee <span dir="ltr"><<a href="mailto:dlee@calldei.com" target="_blank">dlee@calldei.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Indeed ... pair( (1,2,3) ) is just a function that returns a function as per my example.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">But to the point, you can either use vendor extensions (such as marklogic's json:array and map:map types) which have good support for efficient operations,
or look to another language (<sigh>) You may find Scala more amenable to this type of programming.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:talk-bounces@x-query.com" target="_blank">talk-bounces@x-query.com</a> [mailto:<a href="mailto:talk-bounces@x-query.com" target="_blank">talk-bounces@x-query.com</a>]
<b>On Behalf Of </b>jean-marc Mercier<br>
<b>Sent:</b> Tuesday, December 31, 2013 11:49 AM<br>
<b>To:</b> Michael Sokolov<br>
<b>Cc:</b> xquery-discuss; ihe onwuka; Andrew Welch</span></p><div><br>
<b>Subject:</b> Re: [xquery-talk] Matrix Multiplication<u></u><u></u></div><p></p>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">@David, there are tricks to overcome sequence concatenations now. See for instance definition of a pair by Leo, John Snelson, or me : you can write pair(<span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d"> (
1 , 2 , 3 ) , (4 5 , 6) ) to avoid sequence concatenation. I ve written also constant sized vectors using this trick : for instance NUPLE(1,(),<toto>,5), withh associated getters.</span><u></u><u></u></p><div><div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">The bad new is that these operations takes too much time with all the interpreters I have tried, and can't be used in heavy algorithms.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
</div></div></div><div><div>
<div>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">2013/12/31 jean-marc Mercier <<a href="mailto:jeanmarc.mercier@gmail.com" target="_blank">jeanmarc.mercier@gmail.com</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">@Michael Concerning your remark over the discussions I quoted, maps are the basic block for sparse linear algebra. Without performent "maps of nodes" (equivalent to std::map in C++) you will not be able to write
any performant linear algebra modulus for sparse matrix.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">However, before even thinking to sparse matrix, operations on sequences as concatenations are the first "show-stop" to write a linear algebra modulus, since sequences are vectors.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Another one is the lack of constant sized vectors (needed for basic dense matrix operations).<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">@Ryan thx for these links, they are very interesting.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Well, I am going to party ! Have a happy new year !<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">2013/12/31 Michael Sokolov <<a href="mailto:msokolov@safaribooksonline.com" target="_blank">msokolov@safaribooksonline.com</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I would love to see some tests of pure XQuery implementations of both sparse and dense operations. I suspect performance of matrix multiply, inversion, etc, will be poorer than in C++ or Java, but I would expect
performance comparable to Perl or Python (w/o its numpy extension) - just a wild guess. I'd also expect it might be easier to get good sparse performance. I don't know why, just a hunch.<br>
<br>
But the more interesting question for me is whether language changes are really needed to support this. I would have thought that proper optimization of operations on sequences would be enough? So for example, an extension module using sequences as matrix
datatypes could possibly optimize performance using a lower-level implementation. Does anyone see any reason why that wouldn't be possible?<br>
<br>
-Mike<br>
<br>
PS: <br>
I reviewed the discussion you referred to, jean-marc, but it seems to have more to do with using functions as map keys, and I don't see any direct connection to linear algebra?<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
On 12/31/2013 9:55 AM, jean-marc Mercier wrote:<u></u><u></u></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It is not due to the spec. It is rather due to the common usage of XQUERY, forcing vendor solutions (as BaseX) to focus primarily on XML Data Base requests more than algorithmic performances.
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">There are actually some threads that are discussing these performance issues in the context of maps (maps are for instance used for sparse matrix representations) : look for instance to ""map module for XQUERY ?"
that I initiated or "Higher-order XQuery Modules" by Leo from BaseX, on talk@<a href="http://x-query.com" target="_blank">x-query.com</a> mailing list. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Anyhow, to write a serious linear algebra modulus, the basic need is to have a vector containers. Unfortunately, XQUERY does not provide any performant vector containers at present time, and it is not possible to
code them in pure XQUERY : I have tried, and more experienced xquery developpers than me have also tried, without success.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">We have to wait for the XQUERY version that will give us these containers, a decision to be taken by the W3C.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">2013/12/31 Andrew Welch <<a href="mailto:andrew.j.welch@gmail.com" target="_blank">andrew.j.welch@gmail.com</a>><u></u><u></u></p>
<p style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
Are you saying the spec as it stands somehow forces all implementations to be 1000x slower, or just what you have observed in some particular implementation?<br>
<br>
<u></u><u></u></p>
<div>
<div>
<p style="margin-left:.5in">On 31 Dec 2013 14:27, "jean-marc Mercier" <<a href="mailto:jeanmarc.mercier@gmail.com" target="_blank">jeanmarc.mercier@gmail.com</a>> wrote:<br>
><br>
> As far as I understand, you want to write a linear algebra module using XQUERY ?<br>
> If so, I opened a thread some months ago about this idea. My opinion today is that this is a false good idea at present time.<br>
><br>
> 1) XQUERY would be really good for writing concise, efficient linear algebra modulus.<br>
> 2) However, I strongly recommend to wait a little bit for starting coding : the current version of XQUERY (3.0) suffers from performance issues. A linear algebra modulus written in XQUERY is expected to have performances performances 1000 X slower than its
corresponding C++ or JAVA (you can measure it precisely). Any mathematician linear algebra modulus would probably trashed your modulus after the first test.<br>
><br>
> Hope this helps<br>
><br>
><br>
><br>
> 2013/12/31 Ihe Onwuka <<a href="mailto:ihe.onwuka@gmail.com" target="_blank">ihe.onwuka@gmail.com</a>><br>
>><br>
>> Assuming a sparse representation it is about 4 lines of SQL. This is known not least because you can read enough articles and papers that discuss it and it optimises well. The obvious google search does not reveal any corresponding XQuery discussion, neither
does it appear to have surfaced on this or the eXist mailing list (allowing for my deficient search skills). For something so "trivial" I thought that was rather strange. Hence I thought it would be prudent to ask before naively embarking on a 600k X 40k
matrix multiplication.<br>
>><br>
>><br>
>><br>
>> On Tue, Dec 31, 2013 at 11:31 AM, Andrew Welch <<a href="mailto:andrew.j.welch@gmail.com" target="_blank">andrew.j.welch@gmail.com</a>> wrote:<br>
>>><br>
>>><br>
>>> It should be pretty trivial... <br>
>>><br>
>>> On 31 Dec 2013 11:07, "Ihe Onwuka" <<a href="mailto:ihe.onwuka@gmail.com" target="_blank">ihe.onwuka@gmail.com</a>> wrote:<br>
>>> ><br>
>>> > Has anybody tried this in XQuery or if I am so foolish (not yet but give me time) would I be the courageous <culturalReference><a href="http://www.youtube.com/watch?v=ik8JT2S-kBE" target="_blank">http://www.youtube.com/watch?v=ik8JT2S-kBE</a></culturalReference>
early adopter.<br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > <a href="mailto:talk@x-query.com" target="_blank">talk@x-query.com</a><br>
>>> > <a href="http://x-query.com/mailman/listinfo/talk" target="_blank">http://x-query.com/mailman/listinfo/talk</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> <a href="mailto:talk@x-query.com" target="_blank">talk@x-query.com</a><br>
>> <a href="http://x-query.com/mailman/listinfo/talk" target="_blank">http://x-query.com/mailman/listinfo/talk</a><br>
><br>
><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<u></u> <u></u></p>
</div>
</div>
<pre style="margin-left:.5in">_______________________________________________<u></u><u></u></pre>
<pre style="margin-left:.5in"><a href="mailto:talk@x-query.com" target="_blank">talk@x-query.com</a><u></u><u></u></pre>
<pre style="margin-left:.5in"><a href="http://x-query.com/mailman/listinfo/talk" target="_blank">http://x-query.com/mailman/listinfo/talk</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>
<br>_______________________________________________<br>
<a href="mailto:talk@x-query.com">talk@x-query.com</a><br>
<a href="http://x-query.com/mailman/listinfo/talk" target="_blank">http://x-query.com/mailman/listinfo/talk</a><br></blockquote></div>