I have created a lexer using antlr 4 for tokenizing Turkish natural language texts, what I need to do is to have a token stream which I can fetch tokens one by one. CommonTokenStream returns a List if I use it like this:
ANTLRInputStream inputStream = new ANTLRInputStream(input);
TurkishLexer lexer = new TurkishLexer(inputStream);
CommonTokenStream tokenStream = new CommonTokenStream(lexer);
List<Token> tokens = tokenStream.fill();
for (Token token : token) ...
However I don't want to construct a list of tokens as my input could be huge, I just want something like:
for (Token token: tokenStream.next()) ...
Which I would iterate until getting an EOF token.
Is there a Token Stream that allows me to iterate over tokens?
Rather than use a CommonTokenStream, you could simply use Lexer.nextToken.
for (Token token = lexer.nextToken();
token.getType() != Token.EOF;
token = lexer.nextToken())
{
...
Related
I am using gremlin with Azure Cosmos DB. I'm using this code to get a list of files from a graph database.
public async Task<List<string>> GetFilesWithMoreThanOneFilename()
{
List<string> list = new List<string>();
using (var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(), GremlinClient.GraphSON2MimeType))
{
var resultSet = await gremlinClient.SubmitAsync<dynamic>("g.V().hasLabel('file').where(out().count().is(gt(1)))");
if (resultSet.Count > 0)
{
foreach (var result in resultSet)
{
string output = JsonConvert.SerializeObject(result);
list.Add(output);
}
}
}
return list;
}
The output string looks like this:
{"id":"0a37e4896b6310b6d152f6cf89336173ffb89b819f7955494322e0f0bec017b4","label":"file","type":"vertex","properties":{"fileSize":[{"id":"456b087c-7cf3-43ea-a482-0f31219bc520","value":"41096"}],"mimeType":[{"id":"d849b065-16f8-465b-986c-f8e0fdda9ac7","value":"text/plain"}]}}
My Question is how I can get a single value from the result. For example just the ID or the mimeType or is the only possiblity to work with the output and string manipulation?
Due to your output data is in json format, so you could use Newtonsoft.Json to read the data.
I create a json file with your data, you could just parse the json data without file. And just read the id and properties.fileSize
static void Main(string[] args)
{
JObject jsonData = JObject.Parse(File.ReadAllText( "test.json"));
Console.WriteLine("id:"+jsonData["id"].ToString());
Console.WriteLine("properties:"+jsonData["properties"]["fileSize"].ToString());
Console.ReadLine();
}
And here is the result:
Hope this could help you, if yo ustill have other questions, please let me know.
Update: if you want to get the value in the array, you could use this to get the value:
Console.WriteLine("mimeType.value:" + jsonData["properties"]["mimeType"][0]["value"].ToString());
In general, you should only retrieve the values that you actually need and not vertices with all their properties and then filter locally. This is equivalent to relational databases where you wouldn't usually do SELECT * and instead do something like SELECT name.
Additionally, you should specify the return type you're expecting which allows Gremlin.NET to deserialize the result for you so you don't have to do that yourself.
These two suggestions together give you something like this:
var names = await client.SubmitAsync<string>(
"g.V().hasLabel('person').where(out().count().is(gt(1))).values('name')");
Console.WriteLine($"First name: {names.First()}");
names is then just a ResultSet<string> which implements IReadOnlyCollection<string>.
Recently I noticed some sensitive data was being written out to a log file in an app I'm working on. I looked into it, and our app has a list of keys that it will remove from our logs before writing them out, but the list feels to me like it could be expanded.
Is there a list somewhere of common keys that should be removed from log files when found?
For example the following keys and their variations should probably never be logged anywhere in a log file
access_token
auth_token
client_id
client_secret
oauthSecret
oauthToken
password
refresh_token
connection_string
Note that I've seen the OWASP Logging cheat sheet, but it didn't seem to have any specifics in it, just generalizations.
So far the best I've been able to come up with is the following, and I found my best results by searching for "redact secret password token" in Google.
Redact keys that have the following anywhere in their name
remembering that keys can show up in any of the following ways: x=y, x:y, 'x':y, "x":y <x>y</x>
token
secret
password
session
connection
jwt
consider removing all username variations as well, like:
username
user_name
user
email
credential
Redact values that look like
credit card numbers
social insurance numbers
possibly email addresses
Cloudera has a page detailing regular expressions for redacting some data values, not keys:
Credit Card numbers (with separator)
\d{4}[^\w]\d{4}[^\w]\d{4}[^\w]\d{4}
Social Security numbers (with separator) \d{3}[^\w]\d{2}[^\w]\d{4}
Email addresses \b([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9-._] \
[A-Za-z0-9])#(([A-Za-z0-9]|[A-Za-z] \ [A-Za-z0-9-][A-Za-z0-9]).)+([A-Za-z0-9] \
|[A-Za-z0-9][A-Za-z0-9-]*[A-Za-z0-9])\b
Hostnames \b(([A-Za-z]|[A-Za-z][A-Za-z0-9-] \
[A-Za-z0-9]).)+([A-Za-z0-9] \ |[A-Za-z0-9][A-Za-z0-9-][A-Za-z0-9])\b
Node module for redacting data
https://github.com/watson/redact-secrets
Note: A coworker of mine did a great job of supplying me with a java sample of key value redacting in Java, using a regex to eliminate key values that contain `session,secret,token,password,passwd,connection' in xml and json, and quoted variants.
import java.security.GeneralSecurityException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class Redact {
private static final String prefix = "[\"']?(session|(auth_)?token|password|passwd|secret|connection)[\"']?";
private static final String capture = "([\"']?)([\\w\\d!#$%()*+,-./:<=>?#[\\\\]^_`{|}~]+)([\"']?)";
private static Pattern pattern = Pattern.compile(prefix + "\\s*(=|:)\\s*" + capture, Pattern.CASE_INSENSITIVE);
public static String encryptSensitiveInfo(String message) {
try {
if (message == null)
return message;
StringBuffer newStr = new StringBuffer(message);
int lenDiff = 0;
Matcher m = pattern.matcher(message);
// Loop over message until all sensitive data is redacted
while (m.find()) {
String keyAndValue = m.group(0);
String value = m.group(5);
String REDACTED = "REDACTED";
String replacementText = keyAndValue.replace(value, REDACTED);
// Replace the key/value in the message with the new redacted
// value, adjusting for where it is in the redacted version of
// the input
newStr = newStr.replace(m.start() - lenDiff, m.end() - lenDiff, replacementText);
lenDiff += keyAndValue.length() - replacementText.length();
}
return newStr.toString();
} catch (Exception e) {
return message;
}
}
}
I have a Query class which holds the queries i can send like so:
Class Query {
Integer product_id
Integer collection_id
Integer id
}
I use object mapper to convert my Query object to map like this:
def q = new Query(product_id: 12345)
Map <String, Object> toMap = new ObjectMapper().convertValue( q, Map )
Which then in turn i pass on my RESTClient so it is included in the request
def client = new RESTClient ('http://somewebsite.com')
client.get(path: 'somePath/anotherPath.json',
contentType: ContentType.JSON,
query: q)
After i send the request, the empty keys in the query map are also sent in the request which causes problems to the response
GET somePath/anotherPath.json?product_id=12345&collection_id=&id=
As the title says, is there a way to remove keys with empty values in map so they are not included in the request when you send a REST GET request. I want it to be like this:
GET somePath/anotherPath.json?product_id=12345
Where the key with empty values (collection_id and id) is not sent in the request.
You can use the annotation #JsonInclude(Include.NON_NULL)
Class Query {
#JsonInclude(Include.NON_NULL)
Integer product_id
...
}
See documentation here
Doesn't ObjectMapper have any parameter which would disable exporting nulls?
If not you may do:
Map <String, Object> toMap = new ObjectMapper().convertValue(q, Map).findAll { it.value != null }
I am using Gatling for load testing. When I am creating Customer profiles , Customer IDs will be generated. I am able to extract the customer ID and saving in to session variables.
But How to store these values in to a Text file.
please help me.
There's several ways to achieve that.
If you're familiar with Scala, you can:
open a file in your simulation
write into it in a exec(function): https://github.com/excilys/gatling/wiki/Session#wiki-functions
close the file (either register an ActorSystem shutdown hook, or use Gatling 2 snapshot after hook: https://github.com/excilys/gatling/issues/1475)
If this is too complex for you, the most simple way is to use logback:
declare a logger in your simulation with a special name
configure logback so this logger outputs in a dedicated FileAppender
use a pattern that just logs the message
Regarding the second solution, check the logback documentation: http://logback.qos.ch/documentation.html
In the example below I'm saving the extracted NumberIDs out of the SOAP response and then save them into the numbersIDs.csv file so that it can be fed into another Request.
//Variables to store the extracted values of the response
var numberIDs : Seq[String] = _
//Java Writer
val numberIDs_writer = {
val fos = new java.io.FileOutputStream("numbersIDs.csv")
new java.io.PrintWriter(fos,true)
}
val scn = scenario("basicSimulation")
.exec(http("Request_One")
.post("/services")
.body(StringBody(inputXMLpayLoad))
.headers(Request_One_Header)
.check(bodyString.saveAs("Request_Response")) //Save the response in a variable called Request_Response
.check((status.is(200)))
//Extract all numberIDs of the SOAP response
.check(regex("""<NumberId>([\d]+)</NumberId>""")
.findAll
.transform { string => numberIDs = string; string }
.saveAs("numberIDs")))
//Save variable numberIDs into the csv file
.foreach("${numberIDs}", "id") {
exec(session => {
numberIDs_writer.println(session("id").as[String])
session
})
}
Since 2.0.0-M1, infoExtractor hook takes a Session parameter:
https://github.com/excilys/gatling/issues/1004
There's also a builtin that adds the Session content to simulation.log records when the request fails.
https://github.com/excilys/gatling/commit/b7b6299c658d0aa1b88971208eb0c98a9808e37f
You can adapt this example if you just want to log with logback.
class MySimulation extends Simulation with Logging {
def logSessionOnFailure(status: RequestStatus, session: Session, request: Request, response: ExtendedResponse): List[String] = {
if (status == KO) logger.error(session)
Nil
}
val httpConf = http
...
.extraInfoExtractor(logSessionOnFailure)
...
}
Another possibility (not tested, but could work) is to use response transformer (call .transformResponse after .post). In the transformer body you would get a response object from which you can extract the generated ID and append it into a file, collection, etc.. Then just return the original response as the transformation result. This is however not a very nice solution from the design point of view, because your transformation has a side effect.
How do you deal with the fact, that URLs are case sensitive in xPages even for parameters? For example URL:
my_page.xsp?folderid=785478 ... is not the same as ...
my_page.xsp?FOLDERID=785478
How to make, for example, a proper check that params contain some key e.g.
param.containsKey("folderid") which desnt work when there is 'FOLDERID' in URL.
I'd suggest defining a couple convenience #Functions:
var #HasParam = function(parameter) {
var result:boolean = false;
for (var eachParam : param.keySet()) {
if (eachParam.toLowerCase() == parameter.toLowerCase()) {
result = true;
break;
}
}
return result;
};
var #GetParam = function(parameter) {
var result = "";
if (#HasParam(parameter)) {
for (var eachParam : param.keySet()) {
if (eachParam.toLowerCase() == parameter.toLowerCase()) {
result = param.get(eachParam);
break;
}
}
}
return result;
};
Then you can safely query the parameters without caring about case. For bonus points, you could add requestScope caching so that you can skip looping through the keySet if you're examining a parameter that you've previously looked at during the same request.
you may use this function:
context.getUrlParameter('param_name')
then test if it's null or not.
make sure to decide for one,so either upper or lowercase
other than that i'd suggest something like
KeyValuePair<string,string> kvp = null;
foreach(KeyValuePair<string,string> p in param)
{
if(UPPERCASE(p.Key) == UPPERCASE("folderid"))
{
kvp = p;
break;
}
}
syntax isn't correct and idk the uppercase method in c# right now,but you get the point
The easiest answer is ofcourse the obvious. Be sure that the parameters you are using througout your application are always the same on every url you are generating and know what to expect. A good approach to accomplish this is to create a ssjs function which generates url's for you according to the objects you submit.
In this function you could check which object you are receiving and with the use of keywords and so forth generate the correct url. This way generating twice a url with the same input parameters should always generate the exact same url.
another option would be just to double check with a bit of code like this
var key = "yourkey";
if(param.contains(#uppercase(key)) || param.contains(#lowercase(key)){
// do stuff
}
But should not be necesarry if the url you are parsing is generated by your own application
Edit after post of topic starter
Another option would be to grap the url directly from from the facescontext and to convert it to a string first. When it is a string you can parse the parameters yourself.
You can combine server side substitution/redirection to get around the issue that David mentioned. So a substitution rule will redirect incoming patern like this:
http://myhost/mypage/param (/mypage/* => which converts to - /dbpath/mypage.xsp?*) - substitution is tricky so please handle with care.
Also I believe I read somewhere that context.getUrlParameter is not case sensitive - can someone please confirm this.
Hope this helps.