I am currently trying to store some data streaming from a Binance Miniticker Websocket, but I can't figure out a way to do so.
I would like to append the data to an existing dictionary, so that I can access it as historical data.
def miniticker_socket(msg):
''' define how to process incoming WebSocket messages '''
if msg[0]['e'] != 'error':
for item in msg:
miniticker["{0}".format(item['s'])] = {'low': [],'high': [], 'open': [], 'close':[],'timestamp':[], 'symbol':[] }
miniticker[item['s']]['close'].append(msg[msg.index(item)]['c'])
print(miniticker[item['s']])
else:
print('there has been an issue')
bsm = BinanceSocketManager(client)
#ticker_key = bsm.start_symbol_ticker_socket(crypto, ticker_socket)
miniticker_key = bsm.start_miniticker_socket(miniticker_socket)
bsm.start()
The issue I'm having in the code above is that the data does not get appened, because every time the Websocket calls back the function, it also defines the dictionary as empty. I can't define the dictionary outside the Websocket because the name of the dictionary is given by the item['s'] element inside the socket.
I also tried returning the whole data and calling the callback function in another function but this generates another error saying "msg is not defined."
I would appreciate your feedback on this!
Thanks
Also, you can try to check if a key already exists in the dictionary miniticker:
key = "{0}".format(item['s'])
if key not in miniticker.keys():
miniticker["{0}".format(item['s'])] = {...}
So you will not redefine it as an empty dictionary each time
I think what you might want is a global variable dictionary containing dictionary values that come in from the ticker. You will need something unique for the keys of the global dictionary.
For example, you could use a string datetime:
timestamp_key = datetime.datetime.now().isoformat()
global_dict[timestamp_key] = miniticker["{0}".format(item['s'])] = {'low': [],'high': [], 'open': [], 'close':[],'timestamp':[], 'symbol':[] }
global_dict[timestamp_key][item['s']]['close'].append(msg[msg.index(item)]['c'])
The global dict would end up something like this:
{
"2020-03-25T17:14:19.382748": {
"your_data_key1": { "more": "data" }
},
"2021-03-25T17:15:19.249148": {
"your_data_key1": { "more": "data_from_another_update" }
}
}
Related
Other questions
This is a spin-off from How can I replace a key-value pair in a nested dictionary with the value from the same key-value pair? where the answer is working only in a one-time-nested dictionary.
And it is a spin-off from Loop through all nested dictionary values? which I could not get to work on this problem.
Before:
I have a dictionary that is nested many times.
dict_nested = {
"key_":{
"key0a":{
"key1a":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b"},
"key1b":"value1b"},
"key0b":{
"key_XYZ":{
"key1a":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b"},
"key1b":"value1b"}
}
}
}
After:
The result should look like this:
dict_nested_new = {
"key_":{
"key0a":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b",
"key1b":"value1b"},
"key0b":{
"key_XYZ":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b",
"key1b":"value1b"}
}
}
}
Modifying a Python dict while iterating over it
When I looped through the items of the dictionary to delete / replace, I got the error
RuntimeError: dictionary changed size during iteration
which needs somehow to be avoided.
How can I replace the "key1a":SOME_VALUE key-value pair with its value each time it occurs somewhere in the dictionary?
As I understand it, you want to recursively search for a key in a nested dict and promote its value.
This might not be super efficient, but it should work. It also does not really explore dictionaries with lists as values but your example data does not have them so I did not implement that.
import copy
import json
def find_replace(this_dict, target_key):
## optional depending on if you care that you mutate this_dict
this_dict = copy.deepcopy(this_dict)
for key in this_dict:
# if the current value is a dict, dive into it
if isinstance(this_dict[key], dict):
this_dict[key] = find_replace(this_dict[key], target_key)
# if the current key is our target merge the values and remove the key
if key == target_key:
this_dict = {**this_dict, **this_dict[key]}
del this_dict[key]
return this_dict
dict_nested = {
"key_":{
"key0a":{
"key1a":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b"
},
"key1b":"value1b"
},
"key0b":{
"key_XYZ":{
"key1a":{
"sub_key2a":"sub_value2a",
"sub_key2b":"sub_value2b",
"key1a": {
"sub_key3a":"sub_value3a",
"sub_key3b":"sub_value3b"
},
},
"key1b":"value1b"
}
}
}
}
dict_nested_new = find_replace(dict_nested, "key1a")
print(json.dumps(dict_nested_new, indent=4))
Should give you:
{
"key_": {
"key0a": {
"key1b": "value1b",
"sub_key2a": "sub_value2a",
"sub_key2b": "sub_value2b"
},
"key0b": {
"key_XYZ": {
"key1b": "value1b",
"sub_key2a": "sub_value2a",
"sub_key2b": "sub_value2b",
"sub_key3a": "sub_value3a",
"sub_key3b": "sub_value3b"
}
}
}
}
Note that I added an addition level of nesting with a sub-nested key match just to show that scenario. Additional optimizations like support for lists and avoiding updates to unchanged keys available for a reasonable fee :-P
I'm trying to write an automation script that needs to get the values from the output below. The problem is the CValue is not a constant number. It can range anywhere from 1 - x sample values. Is there a way I can store each value properly?
{
'Output': {
'Name': 'Sample',
'Version': {
'Errors': [],
'VersionNumber': 2,
'AValue': 'Hello',
'BValue': ['val:val:BVal'],
'CValue': [{
'DValue': 'aaaaa-bbbbb-cccc',
'Name': 'Sample_Name_1'
}, {
'DValue': 'aaaaa-bbbbb-ddddd',
'Name': 'Sample_Name_2'
}]
}
},
'RequestId': 'eeeee-fffff-gggg'
}
Right now, I'm doing it in the most inefficient way by storing each value separately. My code looks like something below:
def get_sample_values():
test_get = command.sdk(xxxx)
dset_1 = test_get['Output']['Version']['CValue'][0]['DValue']
dset_2 = test_get['Output']['Version']['CValue'][1]['DValue']
return dset_1, dset_2
It works but it's limited to only 2 sets of the dset. Can you please provide input on how I can do it more efficiently?
Use case is this, I need the DValues for another function that requires it. The format for that request is going to be something like:
Source = {
'SourceReference': {
'DataReference': [
{
'EValue': 'string, string, string'
'FValue': DValue1
},
'EValue': 'string, string, string'
'FValue': DValue2
}
]
}
Use a list comprehension to create a list constructed from the desired element of the CValue dicts, then return the list.
return [x['DValue'] for x in test_get['Output']['Version']['CValue']]
Does this work for you?
# either
def get_sample_values():
test_get = command.sdk(xxxx)
return test_get['Output']['Version']['CValue']
# or a generator - maybe not that useful here, but possible
def get_sample_values():
test_get = command.sdk(xxxx)
yield from test_get['Output']['Version']['CValue']
# Then you can use it
for value in get_sample_values():
print(value)
# or
print(values[3])
# for the generator
values = list(get_sample_values())
print(values[3])
For more information https://realpython.com/introduction-to-python-generators/
I am new to Gatling, I am trying to do the performance testing for couple of rest calls. In my scenario I need to extract a value from json response of the 1st call and add those values to the list after looping for few times. Again after looping for few times and adding the values into the list, I want to reuse each value in my next rest call by iterating over the values in the list. Can anyone please suggest on how to implement this. I tried something as below,
var datasetIdList = List.empty[String]
val datasetidsFeeder = datasetIdList.map(datasetId => Map("datasetId" -> datasetId)).iterator
def createData() = {
repeat(20){
feed("").exec(http("create dataset").post("/create/data").header("content-type", "application/json")
.body(StringBody("""{"name":"name"}"""))
.asJson.check(jsonPath("$.id").saveAs("userId"))))
.exec(session => { var usrid = session("userId").as[String].trim
datasetIdList:+= usrid session})
}}
def upload()= feed(datasetidsFeeder).exec(http("file upload").post("/compute-metaservice/datasets/${datasetId}/uploadFile")
.formUpload("File","./src/test/resources/data/File.csv")
.header("content-type","multipart/form-data")
.check(status is 200))
val scn = scenario("create data and upload").exec(createData()).exec(upload())
setUp(scn.inject(atOnceUsers(1))).protocols(httpConf)
}
I am seeing an exception that ListFeeder is empty when trying to run above script. Can someone please help
Updated Code:
class ParallelcallsSimulation extends Simulation{
var idNumbers = (1 to 50).iterator
val customFeeder = Iterator.continually(Map(
"name" -> ("test_gatling_"+ idNumbers.next())
))
val httpConf = http.baseUrl("http://localhost:8080")
.header("Authorization","Bearer 6a4aee03-9172-4e31-a784-39dea65e9063")
def createDatasetsAndUpload() = {
repeat(3) {
//create dataset
feed(customFeeder).exec(http("create data").post("/create/data").header("content-type", "application/json")
.body(StringBody("""{ "name": "${name}","description": "create data and upload file"}"""))
.asJson.check(jsonPath("$.id").saveAs("userId")))
.exec(session => {
val name = session("name").asOption[String]
println(name.getOrElse("COULD NOT FIND NAME"))
val userId = session("userId").as[String].trim
println("%%%%% User ID ====>"+userId)
val datasetIdList = session("datasetIdList").asOption[List[_]].getOrElse(Nil)
session.set("datasetIdList", userId :: datasetIdList)
})
}
}
// File Upload
def fileUpload() = foreach("${datasetIdList}","datasetId"){
exec(http("file upload").post("/uploadFile")
.formUpload("File","./src/test/resources/data/File.csv")
.header("content-type","multipart/form-data")
.check(status is 200))
}
def getDataSetId() = foreach("${datasetIdList}","datasetId"){
exec(http("get datasetId")
.get("/get/data/${datasetId}")
.header("content-type","application/json")
.asJson.check(jsonPath("$.dlp.dlp_job_status").optional
.saveAs("dlpJobStatus")).check(status is 200)
).exec(session => {
val datastId = session("datasetId").asOption[String]
println("request for datasetId >>>>>>>>"+datastId.getOrElse("datasetId not found"))
val jobStatus = session("dlpJobStatus").asOption[String]
println("JOB STATUS:::>>>>>>>>>>"+jobStatus.getOrElse("Dlp Job Status not Found"))
println("Time: >>>>>>"+System.currentTimeMillis())
session
}).pause(10)
}
val scn1 = scenario("create multiple datasets and upload").exec(createDatasetsAndUpload()).exec(fileUpload())
val scn2 = scenario("get datasetId").pause(100).exec(getDataSetId())
setUp(scn1.inject(atOnceUsers(1)),scn2.inject(atOnceUsers(1))).protocols(httpConf)
}
I see below error when I try to execute above script
[ERROR] i.g.c.s.LoopBlock$ - Condition evaluation crashed with message 'No attribute named 'datasetIdList' is defined', exiting loop
var datasetIdList = List.empty[String] defines a mutable variable pointing to a immutable list.
val datasetidsFeeder = datasetIdList.map(datasetId => Map("datasetId" -> datasetId)).iterator uses the immutable list. Further changes to datasetIdList is irrelevant to datasetidsFeeder.
Mutating a global variable with your virtual user is usually not a good idea.
You can save the value into the user's session instead.
In the exec block, you can write:
val userId = session("userId").as[String].trim
val datasetIdList = session("datasetIdList").asOption[List[_]].getOrElse(Nil)
session.set("datasetIdList", userId :: datasetIdList)
Then you can use foreach to iterate them all without using a feeder at all.
foreach("${datasetIdList}", "datasetId") {
exec(http("file upload")
...
}
You should put more work in your question.
Your code is not syntax-highlighted, and is formatted poorly.
You said "I am seeing an exception that ListFeeder is empty" but the words "ListFeeder" are not seen anywhere.
You should post the error message so that it's easier to see what went wrong.
In the documentation linked, there is a Warning. Quoted below:
Session instances are immutable!
Why is that so? Because Sessions are messages that are dealt with in a multi-threaded concurrent way, so immutability is the best way to deal with state without relying on synchronization and blocking.
A very common pitfall is to forget that set and setAll actually return new instances.
This is why the code in the updated question doesn't update the list.
session => {
...
session.set("datasetIdList", userId :: datasetIdList)
println("%%%% List =====>>>" + datasetIdList.toString())
session
}
The updated session is simply discarded. And the original session is returned in the anonymous function.
I have got the two following arrays.
producers = { owner_name: "testing123, url: "http://test.com/", logo_svg: "http://test.com/logo.svg"}, { owner_name: "testing456, url: "http://test456.com/", logo_svg: "http://test456.com/logo.svg"}
result = { owner_name: "testing123, true, false}
I am trying to find a way to extract the logo_svg url from producers array when the owner_name match. I am very new to JS, so I'm not very good at it.
I manage to put this together but unclear on how I can get the logo_svg from that.
image={producers.filter((producer) => producer.owner_name === result.owner_name)}
try image = producers.find(p => p.owner_name == result.owner_name).logo_svg
but bear in mind that if the owner name does not exist in producers then find returns undefined and so you'll get an error so you can make the whole expression more robust by doing
obj = producers.find(p => p.owner_name === result.owner_name)
obj_url = obj ? obj.logo_svg || 'default_url_location'
image = obj_url
According to MDN, filter produces an array, so that is why you cannot get a valid variable inside your image.
You can try something like:
image={producers[producers.findIndex(el => el.owner_name === result.owner_name)]}
This will find the index of the item you are looking for and pass it immediately as the index in the producers array.
I'm trying to select certain keys from an JSON array, and filter the rest.
var json = JSON.stringify(body);
which is:
{
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
}
Want I want:
{
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
}
}
I've checked out How to filter JSON data in node.js?, but I'm looking to do this without any packages.
Now you can use Object.fromEntries like so:
Object.fromEntries(Object.entries(raw).filter(([key]) => wantedKeys.includes(key)))
You need to filter your obj before passing it to json stringify:
const rawJson = {
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
};
// This array will serve as a whitelist to select keys you want to keep in rawJson
const filterArray = [
"FirstName",
"typeform_form_submits",
];
// this function filters source keys (one level deep) according to whitelist
function filterObj(source, whiteList) {
const res = {};
// iterate over each keys of source
Object.keys(source).forEach((key) => {
// if whiteList contains the current key, add this key to res
if (whiteList.indexOf(key) !== -1) {
res[key] = source[key];
}
});
return res;
}
// outputs the desired result
console.log(JSON.stringify(filterObj(rawJson, filterArray)));
var raw = {
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
}
var wantedKeys =["FirstName","typeform_form_submits" ]
var opObj = {}
Object.keys(raw).forEach( key => {
if(wantedKeys.includes(key)){
opObj[key] = raw[key]
}
})
console.log(JSON.stringify(opObj))
I know this question was asked aways back, but I wanted to just toss out there, since nobody else did:
If you're bound and determined to do this with stringify, one of its less-well-known capabilities involves replacer, it's second parameter. For example:
// Creating a demo data set
let dataToReduce = {a:1, b:2, c:3, d:4, e:5};
console.log('Demo data:', dataToReduce);
// Providing an array to reduce the results down to only those specified.
let reducedData = JSON.stringify(dataToReduce, ['a','c','e']);
console.log('Using [reducer] as an array of IDs:', reducedData);
// Running a function against the key/value pairs to reduce the results down to those desired.
let processedData = JSON.stringify(dataToReduce, (key, value) => (value%2 === 0) ? undefined: value);
console.log('Using [reducer] as an operation on the values:', processedData);
// And, of course, restoring them back to their original object format:
console.log('Restoration of the results:', '\nreducedData:', JSON.parse(reducedData), '\nprocessedData:', JSON.parse(processedData));
In the above code snippet, the key value pairs are filtered using stringify exclusively:
In the first case, by providing an array of strings, representing the keys you wish to preserve (as you were requesting)
In the second, by running a function against the values, and dynamically determining those to keep (which you didn't request, but is part of the same property, and may help someone else)
In the third, their respective conversions back to JSON (using .parse()).
Now, I want to stress that I'm not advocating this as the appropriate method to reduce an object (though it will make a clean SHALLOW copy of said object, and is actually surprisingly performant), if only from an obscurity/readability standpoint, but it IS a totally-effective (and mainstream; that is: it's built into the language, not a hack) option/tool to add to the arsenal.