YamlDotNet !!binary type - yamldotnet

I am trying to send binary data, i.e. byte arrays, using yaml. According to the yaml documentation, Yaml Binary Type, this is supported. On the Java side I use SnakeYaml and if a value of byte[] is passed, then the yaml correctly gives !!binary.
This functionality does not seem to be supported "out of the box" in YamlDotNet. The below code snippet creates a sequence of integer values:
IDictionary<string, object> data = new Dictionary<string, object>();
const string value = ":| value: <XML> /n\n C:\\cat";
byte[] bytes = Encoding.UTF8.GetBytes(value);
data.Add(ValueKey, bytes);
// Turn the object representation into text
using (var output = new StringWriter())
{
var serializer = new Serializer();
serializer.Serialize(output, data);
return output.ToString();
}
Output like:
val:\r- 58\r- 124\r- 32\r- 118\r- 97\r- 108\r- 117\r- 101\r- 58\r- 32\r- 60\r- 88\r- 77\r- 76\r- 62\r- 32\r- 47\r- 110\r- 10\r- 32\r- 67\r- 58\r- 92\r- 99\r- 97\r- 116\r
But I would like something more like:
val: !!binary |-
OnwgdmFsdWU6IDxYTUw+IC9uCiBDOlxjYXQ=
Can anyone recommend a workaround?

The preferred way to add support for custom types is to use a custom IYamlTypeConverter. A possible implementation for the !!binary type would be:
public class ByteArrayConverter : IYamlTypeConverter
{
public bool Accepts(Type type)
{
// Return true to indicate that this converter is able to handle the byte[] type
return type == typeof(byte[]);
}
public object ReadYaml(IParser parser, Type type)
{
var scalar = (YamlDotNet.Core.Events.Scalar)parser.Current;
var bytes = Convert.FromBase64String(scalar.Value);
parser.MoveNext();
return bytes;
}
public void WriteYaml(IEmitter emitter, object value, Type type)
{
var bytes = (byte[])value;
emitter.Emit(new YamlDotNet.Core.Events.Scalar(
null,
"tag:yaml.org,2002:binary",
Convert.ToBase64String(bytes),
ScalarStyle.Plain,
false,
false
));
}
}
To use the converter in the Serializer, you simply need to register it using the following code:
var serializer = new Serializer();
serializer.RegisterTypeConverter(new ByteArrayConverter());
For the Deserializer, you also need to register the converter, but you also need to add a tag mapping to resolve the !!binary tag to the byte[] type:
var deserializer = new Deserializer();
deserializer.RegisterTagMapping("tag:yaml.org,2002:binary", typeof(byte[]));
deserializer.RegisterTypeConverter(new ByteArrayConverter());
A fully working example can be tried here

For anyone that's interested.... I fixed this by creating the string myself and adding the !!binary tag, and also doing some clean up. Below is the code.
ToYaml:
IDictionary<string, string> data = new Dictionary<string, string>();
string byteAsBase64Fromat = Convert.ToBase64String("The string to convert");
byteAsBase64Fromat = "!!binary |-\n" + byteAsBase64Fromat + "\n";
data.Add(ValueKey, byteAsBase64Fromat);
string yaml;
using (var output = new StringWriter())
{
var serializer = new Serializer();
serializer.Serialize(output, data);
yaml = output.ToString();
}
string yaml = yaml.Replace(">", "");
return yaml.Replace(Environment.NewLine + Environment.NewLine, Environment.NewLine);
And then back by:
string binaryText = ((YamlScalarNode)data.Children[new YamlScalarNode(ValueKey)]).Value
String value = Convert.FromBase64String(binaryText);

Related

Generic Template String like in Python in Dart

In python, I often use strings as templates, e.g.
templateUrl = '{host}/api/v3/{container}/{resourceid}'
params = {'host': 'www.api.com', 'container': 'books', 'resourceid': 10}
api.get(templateUrl.format(**params))
This allows for easy base class setup and the like. How can I do the same in dart?
I'm assuming I will need to create a utility function to parse the template and substitute manually but really hoping there is something ready to use.
Perhaps a TemplateString class with a format method that takes a Map of name/value pairs to substitute into the string.
Note: the objective is to have a generic "format" or "interpolation" function that doesn't need to know in advance what tags or names will exist in the template.
Further clarification: the templates themselves are not resolved when they are set up. Specifically, the template is defined in one place in the code and then used in many other places.
Dart does not have a generic template string functionality that would allow you to insert values into your template at runtime.
Dart only allows you to interpolate strings with variables using the $ syntax in strings, e.g. var string = '$domain/api/v3/${actions.get}'. You would need to have all the variables defined in your code beforehand.
However, you can easily create your own implementation.
Implementation
You pretty much explained how to do it in your question yourself: you pass a map and use it to have generic access to the parameters using the [] operator.
To convert the template string into something that is easy to access, I would simply create another List containing fixed components, like /api/v3/ and another Map that holds generic components with their name and their position in the template string.
class TemplateString {
final List<String> fixedComponents;
final Map<int, String> genericComponents;
int totalComponents;
TemplateString(String template)
: fixedComponents = <String>[],
genericComponents = <int, String>{},
totalComponents = 0 {
final List<String> components = template.split('{');
for (String component in components) {
if (component == '') continue; // If the template starts with "{", skip the first element.
final split = component.split('}');
if (split.length != 1) {
// The condition allows for template strings without parameters.
genericComponents[totalComponents] = split.first;
totalComponents++;
}
if (split.last != '') {
fixedComponents.add(split.last);
totalComponents++;
}
}
}
String format(Map<String, dynamic> params) {
String result = '';
int fixedComponent = 0;
for (int i = 0; i < totalComponents; i++) {
if (genericComponents.containsKey(i)) {
result += '${params[genericComponents[i]]}';
continue;
}
result += fixedComponents[fixedComponent++];
}
return result;
}
}
Here would be an example usage, I hope that the result is what you expected:
main() {
final templateUrl = TemplateString('{host}/api/v3/{container}/{resourceid}');
final params = <String, dynamic>{'host': 'www.api.com', 'container': 'books', 'resourceid': 10};
print(templateUrl.format(params)); // www.api.com/api/v3/books/10
}
Here it is as a Gist.
Here is my solution:
extension StringFormating on String {
String format(List<String> values) {
int index = 0;
return replaceAllMapped(new RegExp(r'{.*?}'), (_) {
final value = values[index];
index++;
return value;
});
}
String formatWithMap(Map<String, String> mappedValues) {
return replaceAllMapped(new RegExp(r'{(.*?)}'), (match) {
final mapped = mappedValues[match[1]];
if (mapped == null)
throw ArgumentError(
'$mappedValues does not contain the key "${match[1]}"');
return mapped;
});
}
}
This gives you a very similar functionality to what python offers:
"Test {} with {}!".format(["it", "foo"]);
"Test {a} with {b}!".formatWithMap({"a": "it", "b": "foo"})
both return "Test it with foo!"
It's even more easy in Dart. Sample code below :
String host = "www.api.com"
String container = "books"
int resourceId = 10
String templateUrl = "$host/api/v3/$container/${resourceId.toString()}"
With the map, you can do as follows :
Map<String, String> params = {'host': 'www.api.com', 'container': 'books', 'resourceid': 10}
String templateUrl = "${params['host']}/api/v3/${params['container']}/${params['resourceId']}"
Note : The above code defines Map as <String, String>. You might want <String, Dynamic> (and use .toString())
Wouldn't it be simplest to just make it a function with named arguments? You could add some input validation if you wanted to.
String templateUrl({String host = "", String container = "", int resourceid = 0 }) {
return "$host/api/v3/$container/$resourceId";
}
void main() {
api.get(templateUrl(host:"www.api.com", container:"books", resourceid:10));
}

Lucene - Sorting Date as NumericField

While trying to sort datetime (long) numeric fields I always get a FormatException.
When converting a string to DateTime, parse the string to take the
date before putting each variable into the DateTime object.
Adding the numeric field:
doc.Add(new NumericField("creationDate", Field.Store.YES, true)
.SetLongValue(DateTime.UtcNow.Ticks);
Add sorting:
// boolean query
var sortField = new SortField("creationDate", SortField.LONG, true);
var inverseSort = new Sort(sortField);
var results = searcher.Search(query, null, 100, inverseSort); // exception thrown here
Inspecting the index, I can verify that 'creationDate' field is storing "long" values. What could be causing this exception?
EDIT:
Query
var query = new BooleanQuery();
foreach (var termQuery in incomingProps.Select(p => new TermQuery(new Term(kvp.Key, kvp.Value.ToLowerInvariant()))
{
query.Add(new BooleanClause(termQuery , Occur.Must));
}
return query;
Version: Lucene.Net 3.0.3
UPDATE:
This issue is occurring again, now with INT values.
I downloaded Lucene.Net source code and debugged the issue.
So it's somewhere in the FieldCache, when trying to parse the value "`\b\0\0\0" to Integer, which seems a bit odd.
I'm adding these values as numeric fields:
doc.Add(new NumericField(VersionNum, int.MaxValue, Field.Store.YES,
true).SetIntValue(VersionValue));
I get the exception when I'm supposed to get at least 1 hit back.
After inspecting the Index I see that the field's term is as following:
And the field text is:
EDIT:
I've hardcoded an int value and added a few segments:
doc.Add(new Field(VersionNum, NumericUtils.IntToPrefixCoded(1), Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS));
Which resulted on storing the version field as:
And still, when I try to sort I get the parsing error:
var sortVersion = new SortField(VersionNum, SortField.INT, true);
For every exception, Lucene is trying to parse " \b\0\0\0 ".
Looking at the prefixed coded stored as string, 1 would translate to " \b\0\0\0\1 " I'm guessing?
Is Lucene probably leaving some garbage behind in the FieldCache ?
Here's a unit test that tries to capture what you're asking. The test passes. Can you explain what the difference with your code is? (posting a full failing test would help us understand what you're doing :-) )
using System;
using System.Linq;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Lucene.Net.Search;
using Lucene.Net.Index;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.QueryParsers;
using Lucene.Net.Documents;
using Lucene.Net.Store;
namespace SO_answers
{
[TestClass]
public class UnitTest1
{
[TestMethod]
public void TestShopping()
{
var item = new Dictionary<string, string>
{
{"field1", "value1" },
{"field2", "value2" },
{"field3", "value3" }
};
var writer = CreateIndex();
Add(writer, item);
writer.Flush(true, true, true);
var searcher = new IndexSearcher(writer.GetReader());
var result = Search(searcher, item);
Assert.AreEqual(1, result.Count);
writer.Dispose();
}
private List<string> Search(IndexSearcher searcher, Dictionary<string, string> values)
{
var query = new BooleanQuery();
foreach (var termQuery in values.Select(kvp => new TermQuery(new Term(kvp.Key, kvp.Value.ToLowerInvariant()))))
query.Add(new BooleanClause(termQuery, Occur.MUST));
return Search(searcher, query);
}
private List<string> Search(IndexSearcher searcher, Query query)
{
var sortField = new SortField("creationDate", SortField.LONG, true);
var inverseSort = new Sort(sortField);
var results = searcher.Search(query, null, 100, inverseSort); // exception thrown here
var result = new List<string>();
var matches = results.ScoreDocs;
foreach (var item in matches)
{
var id = item.Doc;
var doc = searcher.Doc(id);
result.Add(doc.GetField("creationDate").StringValue);
}
return result;
}
IndexWriter CreateIndex()
{
var directory = new RAMDirectory();
var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
var writer = new IndexWriter(directory, analyzer, new IndexWriter.MaxFieldLength(1000));
return writer;
}
void Add(IndexWriter writer, IDictionary<string, string> values)
{
var document = new Document();
foreach (var kvp in values)
document.Add(new Field(kvp.Key, kvp.Value.ToLowerInvariant(), Field.Store.YES, Field.Index.ANALYZED));
document.Add(new NumericField("creationDate", Field.Store.YES, true).SetLongValue(DateTime.UtcNow.Ticks));
writer.AddDocument(document);
}
}
}

System.OutOfMemoryException: Exception of Type 'System.OutOfMemory' was thrown

private List<T> ReadCurrentFile(string currentExtractedFile, PurgingDetails purgingParams)
{
List<T> thinLogDoList = new List<T>();
using (StreamReader sr = new StreamReader(currentExtractedFile))
{
string currentLine = string.Empty;
Dictionary<string, string> ColumnNamesDictionary = null;
while ((currentLine = sr.ReadLine()) != null)
{
if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
{
string[] columnNames = currentLine.Split(new char[] { 'Æ' });
ColumnNamesDictionary = FillColumnNameDictionary(columnNames);
if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
{
thinLogDoList.Add(FillThinLogDO(ColumnNamesDictionary));
}
}
}
}
return thinLogDoList;
}
(Above code is for Reading a File and adding data to the List by filling the object.)
The function is reading file of size 10 MB which is inside a zip file, first I am extracting the zip files, then reading the data, using this function and storing it into List and then deleting the extracted zip files. It is working for approximately 6L(6,00,000) Data but above that data it throws exception.
I want to read More data 10L(10,00,000) how should I do that ?
Do not return a list. Instead, use yield return to just run through the data:
private IEnumerable<i1LogThinDO> ReadCurrentFile(string currentExtractedFile,
PurgingDetails purgingParams)
{
using (StreamReader sr = new StreamReader(currentExtractedFile))
{
string currentLine = string.Empty;
Dictionary<string, string> ColumnNamesDictionary = null;
while ((currentLine = sr.ReadLine()) != null)
{
if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
{
string[] columnNames = currentLine.Split(new char[] { 'Æ' });
ColumnNamesDictionary = FillColumnNameDictionary(columnNames);
if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
{
yield return FillThinLogDO(ColumnNamesDictionary);
}
}
}
}
}
This way, the ball is in the caller's yard. The caller must be able to process the data returned from this method without keeping them all in memory. This could mean that you have to redesign the calling methods as well, but it would bring a huge cut down in memory footprint of the application if you could do all the processing without keeping the data in memory.

How do I create an OrderBy statement using a reflected value?

I would like to create a method that orders an IEnumerable List by a given property where the property is passed into the method by a string i.e. (Mind you the first code example does not work, but the second does and is what I am trying to emulate dynamically).
string sortName = "SerialNumber";
IEnumerable<PartSummary> partList = FunctionToCreateList();
partOrderedList = partList.OrderBy(what do I stick in here);
that would be equivalent to
IEnumerable<PartSummary> partList = FunctionToCreateList();
partOrderedList = partList.OrderBy(p => p.SerialNumber);
How can I accomplish this?
Are you saying you want to pass the order by in to your method? If so, you can use this:
Expression<Func<PartSummary, bool>> orderByClause
Then you can do this:
partOrderedList = partList.OrderBy(orderByClause);
Then you can handle your order by in your business layer or wherever you wish.
Okay, update: If you want to pass in the column name as a string you can do something like as follows:
Create a static class for an extension method (reference: http://social.msdn.microsoft.com/Forums/en-US/linqprojectgeneral/thread/39028ad2-452e-409f-bc9e-d1b263e921f6/):
static class LinqExtensions
{
public static IQueryable<T> OrderBy<T>(this IQueryable<T> source, string sortingColumn, bool isAscending)
{
if (String.IsNullOrEmpty(sortingColumn))
{
return source;
}
ParameterExpression parameter = Expression.Parameter(source.ElementType, String.Empty);
MemberExpression property = Expression.Property(parameter, sortingColumn);
LambdaExpression lambda = Expression.Lambda(property, parameter);
string methodName = isAscending ? "OrderBy" : "OrderByDescending";
Expression methodCallExpression = Expression.Call(typeof(Queryable), methodName,
new Type[] { source.ElementType, property.Type },
source.Expression, Expression.Quote(lambda));
return source.Provider.CreateQuery<T>(methodCallExpression);
}
}
Then you can create your method:
static IQueryable<PartSummary> FunctionToCreateList()
{
IList<PartSummary> list = new List<PartSummary>();
list.Add(new PartSummary
{
Id = 1,
SerialNumber = "A",
});
list.Add(new PartSummary
{
Id = 2,
SerialNumber = "B",
});
return list.AsQueryable();
}
And then call your method:
static void Main(string[] args)
{
IQueryable<PartSummary> partOrderedList = FunctionToCreateList();
PartSummary partSummary = new PartSummary();
string sortBy = "Id";
partOrderedList = partOrderedList.OrderBy(sortBy, false);
foreach (PartSummary summary in partOrderedList)
{
Console.WriteLine(summary.Id + ", " + summary.SerialNumber);
}
Console.ReadLine();
}
Now you can pass in the column name as a string and sort.
Hope this helps!
You can also avoid extending and just use a compiled expression tree to accomplish this:
public Func<T, object> ResolveToProperty<T>(String propertyName)
{
Type t = typeof(T);
var paramExpression = Expression.Parameter(t, "element");
var propertyExpression = Expression.Property(paramExpression, propertyName);
return Expression.Lambda<Func<T, object>>(propertyExpression, paramExpression).Compile();
}
string sortName = "SerialNumber";
IEnumerable<PartSummary> partList = FunctionToCreateList();
var partOrderedList = partList.OrderBy(ResolveToProperty<PartSummary>(sortName));

Using binary type of a UserProperty

For some reason I need to save some big strings into user profiles. Because a property with type string has a limit to 400 caracters I decited to try with binary type (PropertyDataType.Binary) that allow a length of 7500. My ideea is to convert the string that I have into binary and save to property.
I create the property using the code :
context = ServerContext.GetContext(elevatedSite);
profileManager = new UserProfileManager(context);
profile = profileManager.GetUserProfile(userLoginName);
Property newProperty = profileManager.Properties.Create(false);
newProperty.Name = "aaa";
newProperty.DisplayName = "aaa";
newProperty.Type = PropertyDataType.Binary;
newProperty.Length = 7500;
newProperty.PrivacyPolicy = PrivacyPolicy.OptIn;
newProperty.DefaultPrivacy = Privacy.Organization;
profileManager.Properties.Add(newProperty);
myProperty = profile["aaa"];
profile.Commit();
The problem is that when I try to provide the value of byte[] type to the property I receive the error "Unable to cast object of type 'System.Byte' to type 'System.String'.". If I try to provide a string value I receive "Invalid Binary Value: Input must match binary byte[] data type."
Then my question is how to use this binary type ?
The code that I have :
SPUser user = elevatedWeb.CurrentUser;
ServerContext context = ServerContext.GetContext(HttpContext.Current);
UserProfileManager profileManager = new UserProfileManager(context);
UserProfile profile = GetUserProfile(elevatedSite, currentUserLoginName);
UserProfileValueCollection myProperty= profile[PropertyName];
myProperty.Value = StringToBinary(GenerateBigString());
and the functions for test :
private static string GenerateBigString()
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 750; i++) sb.Append("0123456789");
return sb.ToString();
}
private static byte[] StringToBinary(string theSource)
{
byte[] thebytes = new byte[7500];
thebytes = System.Text.Encoding.ASCII.GetBytes(theSource);
return thebytes;
}
Have you tried with smaller strings? Going max on the first test might hide other behaviors. When you inspect the generated string in the debugger, it fits the requirements? (7500 byte[])
For those, who are looking for answer. You must use Add method instead:
var context = ServerContext.GetContext(elevatedSite);
var profileManager = new UserProfileManager(context);
var profile = profileManager.GetUserProfile(userLoginName);
profile["MyPropertyName"].Add(StringToBinary("your cool string"));
profile.Commit();

Resources