From ihe.onwuka at gmail.com Tue Mar 4 07:13:37 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Tue, 4 Mar 2014 15:13:37 +0000 Subject: [xquery-talk] simple map at the moment is not so simple Message-ID: I have something like distinct-values(somepath/@x) ! http:send-request( which works fine I now want to find out the @x's for which the http request fails which entails checking the status attribute of the first sequential item returned from the send-request (thats not the problem I already know how to do that). Now i thought you could flowr the right hand side of the map operator which would allow me to variablize the output of the http request and interrogate that. However as soon as it sees the $ in let $ following the ! operator it objects (well eXist does anyway). I'm using or trying to use ! because the iteration of it's right hand side is not over nodes. Not clear on exactly what can and can't follow !. So to summarize I want to output the LHS (@x) of the simple map expression where the http request in the RHS has not worked (i.e http:send-request()[1]/@status ne 200). From mike at saxonica.com Tue Mar 4 07:28:27 2014 From: mike at saxonica.com (Michael Kay) Date: Tue, 4 Mar 2014 15:28:27 +0000 Subject: [xquery-talk] simple map at the moment is not so simple In-Reply-To: References: Message-ID: Show us the failing expression. I suspect the expression on the RHS of "!" should be in parens. Michael Kay Saxonica On 4 Mar 2014, at 15:13, Ihe Onwuka wrote: > I have something like > > distinct-values(somepath/@x) ! > http:send-request( method ="get"/> > > which works fine > > I now want to find out the @x's for which the http request fails which > entails checking the status attribute of the first sequential item > returned from the send-request (thats not the problem I already know > how to do that). > > Now i thought you could flowr the right hand side of the map operator > which would allow me to variablize the output of the http request and > interrogate that. However as soon as it sees the $ in let $ following > the ! operator it objects (well eXist does anyway). > > I'm using or trying to use ! because the iteration of it's right hand > side is not over nodes. Not clear on exactly what can and can't follow > !. > > So to summarize I want to output the LHS (@x) of the simple map > expression where the http request in the RHS has not worked (i.e > http:send-request()[1]/@status ne 200). > _______________________________________________ > talk at x-query.com > http://x-query.com/mailman/listinfo/talk From ihe.onwuka at gmail.com Tue Mar 4 08:32:14 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Tue, 4 Mar 2014 16:32:14 +0000 Subject: [xquery-talk] simple map at the moment is not so simple In-Reply-To: References:

Message-ID: Away from my machine at the mo but I can tell you it isn't. I'm sure I've fallen for that one before. On Tue, Mar 4, 2014 at 3:28 PM, Michael Kay wrote: > Show us the failing expression. I suspect the expression on the RHS of "!" should be in parens. > > Michael Kay > Saxonica > > On 4 Mar 2014, at 15:13, Ihe Onwuka wrote: > >> I have something like >> >> distinct-values(somepath/@x) ! >> http:send-request(> method ="get"/> >> >> which works fine >> >> I now want to find out the @x's for which the http request fails which >> entails checking the status attribute of the first sequential item >> returned from the send-request (thats not the problem I already know >> how to do that). >> >> Now i thought you could flowr the right hand side of the map operator >> which would allow me to variablize the output of the http request and >> interrogate that. However as soon as it sees the $ in let $ following >> the ! operator it objects (well eXist does anyway). >> >> I'm using or trying to use ! because the iteration of it's right hand >> side is not over nodes. Not clear on exactly what can and can't follow >> !. >> >> So to summarize I want to output the LHS (@x) of the simple map >> expression where the http request in the RHS has not worked (i.e >> http:send-request()[1]/@status ne 200). >> _______________________________________________ >> talk at x-query.com >> http://x-query.com/mailman/listinfo/talk > From ihe.onwuka at gmail.com Fri Mar 14 03:08:03 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Fri, 14 Mar 2014 11:08:03 +0000 Subject: [xquery-talk] Typeswitch - scope of variables Message-ID: I cannot discern from the specification what the scope of variables introduced in the operand expression should be. Yes I can to see if they are but such an experiment would not adjudicate specification conformance, would it. Fleshing it out for clarification (and possible misinterpretation on my part). Since an operand expression can be FLOWR, are any lets introduced therein within the scope of the case clauses of the typeswitch expression? From mike at saxonica.com Fri Mar 14 03:54:53 2014 From: mike at saxonica.com (Michael Kay) Date: Fri, 14 Mar 2014 11:54:53 +0000 Subject: [xquery-talk] Typeswitch - scope of variables In-Reply-To: References: Message-ID: <6F8A1ADA-04F5-4C33-B5AE-9DD5AABA7AE6@saxonica.com> On 14 Mar 2014, at 11:08, Ihe Onwuka wrote: > I cannot discern from the specification what the scope of variables > introduced in the operand expression should be. Same as in any other expression. E.g. if the operand expression is a FLWOR expression, you go to the rules for a FLWOR expression, e.g. the let clause, and the rules for the let clause say The scope of a bound variable includes all subexpressions of the containing FLWOR that appear after the variable binding. [I think we can read "includes" here as "comprises", i.e. it includes nothing other than these subexpressions] Clearly the case clauses of the typeswitch are not subexpressions of this FLWOR so they are not within the scope of a variable introduced by such a let clause. Michael Kay Saxonica > Yes I can to see if > they are but such an experiment would not adjudicate specification > conformance, would it. > > Fleshing it out for clarification (and possible misinterpretation on > my part). Since an operand expression can be FLOWR, are any lets > introduced therein within the scope of the case clauses of the > typeswitch expression? > _______________________________________________ > talk at x-query.com > http://x-query.com/mailman/listinfo/talk From ihe.onwuka at gmail.com Fri Mar 14 05:59:12 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Fri, 14 Mar 2014 13:59:12 +0000 Subject: [xquery-talk] lazy evaluation of sequences Message-ID: does (expr1, expr2,.......,exprN)[1] and variants thereof evaluate lazily. Is this implementation dependent. From ihe.onwuka at gmail.com Fri Mar 14 06:32:33 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Fri, 14 Mar 2014 14:32:33 +0000 Subject: [xquery-talk] lazy evaluation of sequences In-Reply-To: References: Message-ID: Well (4,5 div 0)[1] on Saxon gives 4. Hurray. (4,5 div 0)[1] in eXist gives divide by zero error. Boo. So the answer seems to be it depends. On Fri, Mar 14, 2014 at 1:59 PM, Ihe Onwuka wrote: > does (expr1, expr2,.......,exprN)[1] and variants thereof evaluate > lazily. Is this implementation dependent. From mike at saxonica.com Fri Mar 14 08:12:48 2014 From: mike at saxonica.com (Michael Kay) Date: Fri, 14 Mar 2014 16:12:48 +0000 Subject: [xquery-talk] lazy evaluation of sequences In-Reply-To: References:

Message-ID: <61855492-20B4-4706-991D-923058F04B87@saxonica.com> On 14 Mar 2014, at 14:32, Ihe Onwuka wrote: > Well (4,5 div 0)[1] on Saxon gives 4. Hurray. > > (4,5 div 0)[1] in eXist gives divide by zero error. Boo. > > So the answer seems to be it depends. > Correct, it's entirely implementation-dependent. Saxon would fail on (4, 5 div 0, 6)[3], but other products might return 6. The rule (see the "Errors and Optimization" section of the spec) is (my precis) that a processor doesn't have to evaluate an expression if it doesn't need to know the value of that expression in order to deliver the final result, and in particular, it never needs to evaluate an expression merely in order to discover whether evaluating it throws an error. Michael Kay Saxonica From wcandillon at gmail.com Mon Mar 24 06:00:34 2014 From: wcandillon at gmail.com (William Candillon) Date: Mon, 24 Mar 2014 15:00:34 +0100 Subject: [xquery-talk] [ANN] XQLint - XQuery & JSONiq Code Quality Tool Message-ID: Hello, We are very happy to announce the release of xqlint, a tool that helps to detect errors and potential problems in your queries. We hope that you will enjoy it: https://github.com/wcandillon/xqlint Kind regards, William From benito at benibela.de Tue Mar 25 05:18:36 2014 From: benito at benibela.de (Benito van der Zander) Date: Tue, 25 Mar 2014 14:18:36 +0100 Subject: [xquery-talk] [ANN] CLI XQuery engine Xidel 0.8 released Message-ID: <5331822C.3040202@benibela.de> Hi, Xidel is a dependency-free cli tool to apply (X-)queries to webpages. The 0.8 version improves the JSONiq support (updated to JSONiq 1.0.3) and our own JSON extensions (more compatible with XQuery, assignable), can use arbitrary precision arithmetic, adds a trivial subset of XPath/XQuery 3 (!, ||, and switch) and some utility functions. Best, Benito From james.fuller.2007 at gmail.com Tue Mar 25 05:31:11 2014 From: james.fuller.2007 at gmail.com (James Fuller) Date: Tue, 25 Mar 2014 14:31:11 +0100 Subject: [xquery-talk] [ANN] CLI XQuery engine Xidel 0.8 released In-Reply-To: <5331822C.3040202@benibela.de> References: <5331822C.3040202@benibela.de> Message-ID: interesting stuff url ? thx, Jim Fuller On Tue, Mar 25, 2014 at 2:18 PM, Benito van der Zander wrote: > Hi, > Xidel is a dependency-free cli tool to apply (X-)queries to webpages. > > The 0.8 version improves the JSONiq support (updated to JSONiq 1.0.3) and > our own JSON extensions (more compatible with XQuery, assignable), can use > arbitrary precision arithmetic, adds a trivial subset of XPath/XQuery 3 (!, > ||, and switch) and some utility functions. > > Best, > Benito > _______________________________________________ > talk at x-query.com > http://x-query.com/mailman/listinfo/talk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ihe.onwuka at gmail.com Sun Mar 30 17:02:00 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Mon, 31 Mar 2014 02:02:00 +0100 Subject: [xquery-talk] Getting more out of Freebase with XQuery Message-ID: This is a follow on from http://en.wikibooks.org/wiki/XQuery/Freebase which originated from a problem Michael Westbay assisted me with. Again it illustrates how to obtain information from Freebase via it's MQL language (it predated Sparql). The previous query was taken from https://developers.google.com/freebase/v1/mql-overview and it limits the data that results from the call to the Freebase API. You can see a limit parameter being set to 3 below in the API call. https://www.googleapis.com/freebase/v1/mqlread?query=[{"type":"/music/album","name":null,"artist":{"id":"/en/bob_dylan"},"limit":3}]&cursor If you do not specify a limit with your API call, Freebase will impose a limit of 100 records on your query. This message addresses the question of how to get everything. The key to doing this is dangling at the end of the above API call, it's the cursor parameter and it's usage is discussed with an example here https://developers.google.com/freebase/v1/mql-overview#querying-with-cursor-paging-results To summarise you ask for a cursor (see the example API call above for the form of the initial request) to be returned with your query results which acts as a link to the next set of query results. You obtain that next set by supplying the value of the cursor returned from the previous invocation. Along with that next set you get another cursor that points to the set after that. When the final set of results are retrieved the cursor returns a string value of false (the Freebase overview has this in upper case but my code used lower case 'false' and that works). The overview has sample Python code which I have not tried or parsed in anger but which I believe invokes libraries that take care of all the cursor handling for you. https://developers.google.com/freebase/v1/mql-overview#looping-through-cursor-results However the same thing can easily be achieved from XQuery with a little bit of tail recursion. We will use as an example a MQL query that returns all films with their netflix_id's. [{ "type": "/film/film", "name": null, "netflix_id": [] }] A few brief comments about MQL. You ask for something by giving the field name and a value null. Null gets replaced by the actual value. However if the field can have multiple values MQL will return an array and cause your null query to error. This may happen even when you are expecting a singular value so you can avoid this problem by using the symbol for an empty array instead of null as in the query above. You can paste the query above into http://www.freebase.com/query to see the results (we will take care of the cursor in the code example). Now to the code, which assumes XQuery 3.0 xquery version "3.0"; import module namespace xqjson="http://xqilla.sourceforge.net/lib/xqjson"; Freebase returns JSON but we want to store this in an xml db so we use the above package for json to XML conversion. From eXist you can install the package by just clicking it on the eXist Package Manager which you can get to from the eXist Dashboard. We declare a variable for our query. declare variable $mqlQuery {'[{ "type": "/film/film", "name": null, "netflix_id": [] }]'}; declare variable $freebase {'https://www.googleapis.com/freebase/v1/mqlread'}; declare variable $key {obtain an API key from freebase and puts it's value here'}; Here is a link to a blog that describes the process of obtaining a Freebase API key http://anchetawern.github.io/blog/2013/02/11/getting-started-with-freebase-api/ Since we are going to be doing tail recursion we need to put the API call in a function. Lets start with the function signature. declare function local:freebaseCall($cursor as xs:string,$i as xs:integer) 2 parameters the first is the cursor and the second an integer which I use to provision an auto-incremented unique file name an to tell me how many records were loaded at the end - since there are a 100 records per API call it's ( $i - 1) * 100 + the number of records returned by the final cursor. This function will make the API call and store the results in the db { if ($cursor eq 'false') termination condition then $i || ' pages loaded' else let $params := ('query=' || encode-for-uri($mqlQuery), 'key=' || $key, 'cursor=' || encode-for-uri($cursor)) Above uri encodes the parameters to the API call - we have three the MQL query, the API key and the cursor let $href := $freebase || '?' || string-join($params, '&') This constructs the API call - again thanks to Michael Westbay for showing the correct way to do this by string joining the parameters with a separator of & let $responses := http:send-request() Make the API call. return if ($responses[1]/@status ne '200') then {$responses[1]} else let $jsonResponse:= util:base64-decode($responses[2]) Standard EXPATH http error checking - don't forget to base64 decode the body of the response. let $freebaseXML:= xqjson:parse-json($jsonResponse) Convert the returned JSON to XML because we are going to construct an http PUT to store it in our xml db. let $movieData := http:send-request( , (), {transform:transform($freebaseXML,doc(identity.xsl'),())}) Standard EXPATH PUT request. On the last line we are wrapping the returned XML with an element that carries the value of the cursor that was used to obtain the page. Identity.xsl is of course the standard XSLT identity transform, you can use it as a placeholder for the insertion of your own custom transform. return local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1) Finally the tail recursive call. We extract the cursor from the returned JSON for parameter 1 and increment $i to give us a unique document name for the next page to store. }; Don't forget your closing curly brace and the attendant semi colon One last thing to kick it all off pass the null string as the initial cursor value and initialise your counter local:freebaseCall('',1) One last thing. I experienced repeated eXist crashes when running this. Note that you can prime the function call with a cursor to pick up from where you left off. That way you'll get to the end. return local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1) }; From joewiz at gmail.com Mon Mar 31 12:26:44 2014 From: joewiz at gmail.com (Joe Wicentowski) Date: Mon, 31 Mar 2014 16:26:44 -0400 Subject: [xquery-talk] Getting more out of Freebase with XQuery In-Reply-To: References: Message-ID: Hi Ihe, > http:send-request() ... > One last thing. I experienced repeated eXist crashes when running > this. Check to make sure you're closing your HTTP connection. See http://markmail.org/message/3opzcgyzv6auevkt for the solution; the whole thread is edifying though. Joe From ihe.onwuka at gmail.com Mon Mar 31 22:44:37 2014 From: ihe.onwuka at gmail.com (Ihe Onwuka) Date: Tue, 1 Apr 2014 07:44:37 +0100 Subject: [xquery-talk] Getting more out of Freebase with XQuery In-Reply-To: References: Message-ID: On Mon, Mar 31, 2014 at 9:26 PM, Joe Wicentowski wrote: > Hi Ihe, > >> http:send-request() > ... >> One last thing. I experienced repeated eXist crashes when running >> this. > > Check to make sure you're closing your HTTP connection. See > http://markmail.org/message/3opzcgyzv6auevkt for the solution; the > whole thread is edifying though. > Will investigate - thanks. A couple of dilemmas both related. The return type of the function freebaseCall is string..... would it be helpful to specify that given that it has a side-effect (HTTP Put) One for the RESTFUL purists that might object to a GET that has side-effects. What HTTP method should be used to invoke this XQuery? On reflection the function could probably use a better name.