M (PowerQuery), set the value of a non-primitive variable in a let statement - excel

I'm writing a custom M Language (PowerQuery in Excel) function to query a RESTful interface. This interface has a large number of optional parameters.
Starting with a simple case- I handle an optional limit passed as a simple (primitive) value as follows-
/*
* RESTful API Get from the named endpoint
*/
(endpoint as text, optional limit) =>
let
// query limit
// If limit is supplied as a number, it will be converted to text
// If limit is not supplied it will be set to the value "1000"
limit = if limit <> null then Text.From(limit) else "1000",
As the full API has many paramaters I wanted to use a Record to pass them to the function, but then I realised I don't know how to persuade M to write the default values into the parameter record.
I tried a couple of options.
Direct access-
(endpoint as text, optional params as record) =>
let
params[limit] = if (params[limit] = null) then "1000",
the result is a syntax error-'Token equal expected'
Merging the new value of limit as a Record with "&"
(endpoint as text, optional params as record) =>
let
params = params & if params[limit] = null then [limit = "1000"] else [],
result syntax error-'Token Literal expected'
I'm clearly missing something about the syntax rules for let statements, I know I need a variable = value assignment, and it looks as if putting anything other than a plain variable name on the LHS to write elements inside a structured value is not allowed, but i'm not sure how to acieve this otherwise?

Not sure exactly what you want here, but to create a List of Records where some Records have a default parameter and others do not, you could try something like:
(newParams as record) =>
let
default = [limit=1000, param2=2, param3=3],
final = Record.Combine({default, newParams})
in
final
With regard to Record.Combine, the beauty is that the right hand record will override the left hand record if both are present; and it will just add to it if nothing is present.
So something like:
let
Source = [limit=400, param3="x", param7=246],
conv = fnParams(Source)
in
conv
=>
Depending on the required format of your output string, you can build it using List.Accumulate. eg:
let
Source = [limit=400, param3="x", param7=246],
conv = fnParams(Source),
list = List.Accumulate(List.Zip({Record.FieldNames(conv), Record.ToList(conv)}), "",
(state,current) =>state & "&" & current{0} & "=" & Text.From(current{1}) )
in
list
=> &limit=400&param2=2&param3=x&param7=246

Related

WHERE IN with Azure DocumentDB (CosmosDB) .Net SDK

Having seen this question I'm not sure why the following code for my DocDb instance isn't working:
var userApps = _docs.CreateDocumentQuery(UriFactory.CreateDocumentCollectionUri(Constants.Databases.Applications.ID, Constants.Databases.Applications.Collections.USER_APPS),
new SqlQuerySpec(#"SELECT r.appId FROM ROOT r WHERE r.userId = #userId", (#"#userId", userId).ToSqlParameters()))
.ToList()
.Select(s => (string)s.appId);
var query = _docs.CreateDocumentQuery<Document>(UriFactory.CreateDocumentCollectionUri(Constants.Databases.Applications.ID, Constants.Databases.Applications.Collections.APP_DEFINITIONS),
new SqlQuerySpec(#"SELECT r.id, r.appName FROM ROOT r WHERE r.appId IN (#userApps)", (#"#userApps", userApps.ToArray()).ToSqlParameters()),
new FeedOptions { EnableCrossPartitionQuery = true })
.AsDocumentQuery();
When I execute this, though I know the data should be returning me back a result set, it comes back empty every time.
Troubleshooting so far
Variants of .Select()
Return strings that I string.Join in to a comma-separated list.
Eg:
var userApps = string.Join(#",", _docs.CreateDocumentQuery(UriFactory.CreateDocumentCollectionUri(Constants.Databases.Applications.ID, Constants.Databases.Applications.Collections.USER_APPS),
new SqlQuerySpec(#"SELECT r.appId FROM ROOT r WHERE r.userId = #userId", (#"#userId", userId).ToSqlParameters()))
.ToList()
.Select(s => $#"'{s.appId}'");
Don't encapsulate IN parameter
Removing the () around the parameter spec in the query thinking maybe the SqlParameter spec was doing the array specification?
Eg: #"SELECT r.id, r.appName FROM ROOT r WHERE r.appId IN #userApps")
Ends up throwing "Syntax error, incorrect syntax near '#userApps'."
Validate via Azure Portal queries
Ran the (expected) SQL that this code should be running.
I get back my expected results without issue (ie: I know there is a result set for these queries as-written).
Debugger output for Query 1
AppIds are coming back from query 1.
Unsatisfactory workaround
Change query 2 to not be parameterized. Rather, inject the comma-separated list of IDs from query 1 in to it:
var userApps = string.Join(#",", _docs.CreateDocumentQuery(UriFactory.CreateDocumentCollectionUri(Constants.Databases.Applications.ID, Constants.Databases.Applications.Collections.USER_APPS),
new SqlQuerySpec(#"SELECT r.appId FROM ROOT r WHERE r.userId = #userId", (#"#userId", userId).ToSqlParameter()))
.ToList()
.Select(s => $#"'{s.appId}'"));
var query = _docs.CreateDocumentQuery<Document>(UriFactory.CreateDocumentCollectionUri(Constants.Databases.Applications.ID, Constants.Databases.Applications.Collections.APP_DEFINITIONS),
new SqlQuerySpec($#"SELECT r.id, r.appName FROM ROOT r WHERE r.appId IN ({userApps})"),
new FeedOptions { EnableCrossPartitionQuery = true })
.AsDocumentQuery();
Works perfectly but I'm not going to accept it as an answer to this problem as it goes against a couple decades-worth of SQL best practices and, frankly, shouldn't be a solution.
Here's my ToSqlParameters() extension method in case it's the culprit (this works everywhere else I've used it, though. Maybe something special is needed for arrays?):
public static SqlParameterCollection ToSqlParameters(this (string, object) parmDef) => new SqlParameterCollection(new[] { new SqlParameter(parmDef.Item1, parmDef.Item2) });
Thanks!
If you use a parameterized IN list, then it will be considered as a single value when the parameter is expanded.
For instance, for this query:
SELECT * FROM r WHERE r.item IN (#items) And #items is defined as "['val1', 'val2', 'val3']" will be interpreted as such:
SELECT * FROM r WHERE r.item IN (['val1', 'val2', 'val3'])
which basically means that you're comparing r.item to a single value that is an array of three elements (i.e. equivalent to r.item = ['val1', 'val2', 'val3']).
To compare to multiple items, you need to use a parameter for each value. Something like this: SELECT * FROM r WHERE r.item IN (#val1, #val2, #val3])
A more convenient way to write this query is to use ARRAY_CONTAINS instead and pass the array of items as a single parameter. So the above query will be written like this:
SELECT * FROM r WHERE ARRAY_CONTAINS(#items, r.item)

Arangodb LET variable from AQL for use it in FILTER

I have collection 'Notifier'.
Given the following example document of this collection:
{
"timestamp": 1413543986,
"message": "message",
"readed": {
"user_8": 0,
"user_9": 0,
"user_22": 0
},
"type": "1014574149174"
}
I try find this document with the following AQL:
FOR n IN Notifier
LET usersFound = TO_BOOL(
FOR user IN n.readed
FILTER user == 'user_8'
LIMIT 1
RETURN user
)
FILTER usersFound==true
RETURN n
I get the following error:
[1501] syntax error, unexpected FOR declaration, expecting ) near 'OR
user IN n.readed
FILT...' at position 3:10
How can I write this AQL correctly using LET?
Your query is almost correct, there are 2 minor issues.
1) TO_BOOL() are chars to trigger the correspoding function, you now want to insert a subquery, that is triggered by wrapping an AQL statement in additional ().
So instead of TO_BOOL(AQL) you have to use: TO_BOOL((AQL));
2) n.readed is a JSON object, FOR x IN expectes a list. As you are iterating over the Attributes of n.readed you can use ATTRIBUTES(n.readed) here.
Here is the correct solution for your example:
FOR n IN Notifier
LET usersFound = TO_BOOL(
(FOR user IN ATTRIBUTES(n.readed)
FILTER user == 'user_8'
LIMIT 1
RETURN user)
)
FILTER usersFound==true
RETURN n
PS:
If you are only looking for the exisence of an attribute and do not want to do further filtering you can get that a bit easier using HAS:
FOR n IN Notifier
LET usersFound = HAS(n.readed, 'user_8')
FILTER usersFound==true
RETURN n
If you are just looking for a presence of some field which value can be converted to boolean value you don't need to bother with LET and subqueries. Following query uses HAS function to determine presence of specified field inside document and BOOL to convert it's value to boolean representation:
LET search = 'user_8'
FOR n IN Notifier
FILTER HAS(n.readed, search) == true && TO_BOOL(n.readed[search])
RETURN n

Dapper and varchars

I found the following comment on the Dapper .NET project home page.
Dapper supports varchar params, if you are executing a where clause on a varchar column using a param be sure to pass it in this way:
Query<Thing>("select * from Thing where Name = #Name", new {Name =
new DbString { Value = "abcde", IsFixedLength = true, Length = 10, IsAnsi = true });
On Sql Server it is crucial to use the unicode when querying unicode and ansi when querying non unicode
I'm evaluating Dapper for use with a legacy database (SQL Server 2008), with lots of stored procedures with varchar parameters, and I'm a little confused by this restriction.
With hand-crafted ADO.NET code, I'd use the following for the above query:
new SqlParameter("#Name", "abcde")
without specifying whether it's unicode or not, nor the length.
Why do I need this verbose DbString syntax with Dapper, specifying the column length, IsFixedLength and IsAnsi?
Why IsFixedLength = true for a varchar column (I'd expect it to be true for a char or nchar column)?
Do I have to use DbString like this for stored procedure parameters?
I was expecting Dapper to make my DAL code more concise, but this seems to be making it more verbose for varchar parameters.
UPDATE
I've researched a bit further, to try to understand why Dapper would have this varchar restriction, which I don't appear to have in my hand-crafted code, where I would normally create an input parameter as follows:
var parameter = factory.CreateParameter(); // Factory is a DbProviderFactory
parameter.Name = ...;
parameter.Value = ...;
and usually leave the provider to infer the DbType using its own rules, unless I specifically want to coerce it.
Looking at Dapper's DynamicParameters class, it has a method AddParameters which creates parameters as follows:
var dbType = param.DbType; // Get dbType and value
var val = param.Value; // from
...
// Coerce dbType to a non-null value if val is not null !!!!!
if (dbType == null && val != null) dbType = SqlMapper.LookupDbType(val.GetType(),name);
...
var p = command.CreateParameter();
...
if (dbType != null)
{
p.DbType = dbType.Value;
}
I.e. it explicitly coerces IDataParameter.DbType to a value it looks up with its own algorithm, rather than leaving the provider to use its own rules.
Is there a good reason for this? It seems wrong for me, particularly in the light of the comment about Dapper's support for varchar parameters.
You need this syntax when working with ODBC.
You would need to define a CHAR(30) field as a DbString in c# for Dapper and also set the length (30) and ansi (true) values to prevent Dapper from assuming the string was a text/blob type. Otherwise you will likely receive the error: "Illegal attempt to convert Text/Byte blob type".
I was getting this error using ODBC to connect to Informix until I defined my param as a DbString() and set the length and ansi values.
More info here.
var param = new { Varchar1 = "", Varchar2 = "" };
db.Query("SP", param, commandType:CommandType.StoredProcedure);

How to maintain counters with LinqToObjects?

I have the following c# code:
private XElement BuildXmlBlob(string id, Part part, out int counter)
{
// return some unique xml particular to the parameters passed
// remember to increment the counter also before returning.
}
Which is called by:
var counter = 0;
result.AddRange(from rec in listOfRecordings
from par in rec.Parts
let id = GetId("mods", rec.CKey + par.UniqueId)
select BuildXmlBlob(id, par, counter));
Above code samples are symbolic of what I am trying to achieve.
According to the Eric Lippert, the out keyword and linq does not mix. OK fair enough but can someone help me refactor the above so it does work? A colleague at work mentioned accumulator and aggregate functions but I am novice to Linq and my google searches were bearing any real fruit so I thought I would ask here :).
To Clarify:
I am counting the number of parts I might have which could be any number of them each time the code is called. So every time the BuildXmlBlob() method is called, the resulting xml produced will have a unique element in there denoting the 'partNumber'.
So if the counter is currently on 7, that means we are processing 7th part so far!! That means XML returned from BuildXmlBlob() will have the counter value embedded in there somewhere. That's why I need it somehow to be passed and incremented every time the BuildXmlBlob() is called per run through.
If you want to keep this purely in LINQ and you need to maintain a running count for use within your queries, the cleanest way to do so would be to make use of the Select() overloads that includes the index in the query to get the current index.
In this case, it would be cleaner to do a query which collects the inputs first, then use the overload to do the projection.
var inputs =
from recording in listOfRecordings
from part in recording.Parts
select new
{
Id = GetId("mods", recording.CKey + part.UniqueId),
Part = part,
};
result.AddRange(inputs.Select((x, i) => BuildXmlBlob(x.Id, x.Part, i)));
Then you wouldn't need to use the out/ref parameter.
XElement BuildXmlBlob(string id, Part part, int counter)
{
// implementation
}
Below is what I managed to figure out on my own:.
result.AddRange(listOfRecordings.SelectMany(rec => rec.Parts, (rec, par) => new {rec, par})
.Select(#t => new
{
#t,
Id = GetStructMapItemId("mods", #t.rec.CKey + #t.par.UniqueId)
})
.Select((#t, i) => BuildPartsDmdSec(#t.Id, #t.#t.par, i)));
I used resharper to convert it into a method chain which constructed the basics for what I needed and then i simply tacked on the select statement right at the end.

What is wrong in this LINQ Query, getting compile error

I have a list AllIDs:
List<IAddress> AllIDs = new List<IAddress>();
I want to do substring operation on a member field AddressId based on a character "_".
I am using below LINQ query but getting compilation error:
AllIDs= AllIDs.Where(s => s.AddressId.Length >= s.AddressId.IndexOf("_"))
.Select(s => s.AddressId.Substring(s.AddressId.IndexOf("_")))
.ToList();
Error:
Cannot implicitly convert type 'System.Collections.Generic.List<string>' to 'System.Collections.Generic.List<MyCompany.Common.Users.IAddress>'
AllIDs is a list of IAddress but you are selecting a string. The compiler is complaining it cannot convert a List<string> to a List<IAddress>. Did you mean the following instead?
var substrings = AllIDs.Where(...).Select(...).ToList();
If you want to put them back into Address objects (assuming you have an Address class in addition to your IAddress interface), you can do something like this (assuming the constructor for Address is in place):
AllIDs = AllIDs.Where(...).Select(new Address(s.AddressID.Substring(s.AddressID.IndexOf("_")))).ToList();
You should also look at using query syntax for LINQ instead of method syntax, it can clean up and improve the readability of a lot of queries like this. Your original (unmodified) query is roughly equivalent to this:
var substrings = from a in AllIDs
let id = a.AddressId
let idx = id.IndexOf("_")
where id.Length >= idx
select id.Substring(idx);
Though this is really just a style thing, and this compiles to the same thing as the original. One slight difference is that you only have to call String.IndexOf() one per entry, instead of twice per entry. let is your friend.
Maybe this?
var boundable =
from s id in AllIDs
where s.AddressId.Length >= s.AddressId.IndexOf("_")
select new { AddressId = s.AddressId.Substring(s.AddressId.IndexOf("_")) };
boundable = boundable.ToList();

Resources