I am trying to implement dayjs package in my node.js application. I would like my date and time to be formatted like this:
2022-09-11T17:46:00+01:00
I have my codes like this:
const dayjs = require("dayjs");
const utc = require("dayjs/plugin/utc");
const customParseFormat = require('dayjs/plugin/customParseFormat');
const dayJsUTC = dayjs.extend(utc)
const dayJsDate = dayJsUTC.extend(customParseFormat)
I am trying to check that the format comes out this way 2022-09-11T17:46:00+01:00
Here is the code:
if(!dayJsDate( 2022-09-11T17:46:00+01:00, "YYYY-MM-DD HH:mm", true).isValid()){
return res.status(500).json('invalid date')
}
It is returning invalid date. I know I am not doing it properly. How can I do it properly to scale this validation?
Also, if I want to simply create only date, hour and minute without the additional time details, how can I do that using dayjs?
Sorry for not answering your question straight, but you could get what you want by defining your own function. When you pass no args to it, it returns just time in format like you wish, when passing a string to function - it return true or false, checking format. If you pass second param as "true", function will return date, hours and minutes.
const myDate = (input, noSecondsAndOffset) => {
const
offset = -(new Date().getTimezoneOffset()),
offsetHours = '0' + offset / 60,
offsetMins = '' + offset % 60,
timeZone = `${offsetHours}:${offsetMins}${offsetMins.length === 1 ? '0' : ''}`,
currTime = `${new Date().toJSON().slice(0, -5)}+${timeZone}`,
checker = (string) => !!string.match(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2}/);
return typeof input === 'string'
? checker(string)
: !noSecondsAndOffset
? currTime
: currTime.slice(0, -9)
}
I would like to create a candle graph using GDAX api. I am currently using the HTTP request for historical data https://docs.gdax.com/#get-historic-rates bug this is marked that I should use websocket API. Unfortunately I don't know how to handle historic data through Gdax websocket api https://github.com/coinbase/gdax-node. Could someone help me ?
Here are 1m candlesticks built from match channel using GDAX websocket
"use strict";
const
WebSocket = require('ws'),
PRECISION = 8
function _getPair(pair) {
return pair.split('-')
}
let ws = new WebSocket('wss://ws-feed.pro.coinbase.com')
ws.on('open', () => {
ws.send(JSON.stringify({
"type": "subscribe",
"product_ids": [
"ETH-USD",
"BTC-USD"
],
"channels": [
"matches"
]
}))
})
let candles = {}
let lastCandleMap = {}
ws.on('message', msg => {
msg = JSON.parse(msg);
if (!msg.price)
return;
if (!msg.size)
return;
// Price and volume are sent as strings by the API
msg.price = parseFloat(msg.price)
msg.size = parseFloat(msg.size)
let productId = msg.product_id;
let [base, quote] = _getPair(productId);
// Round the time to the nearest minute, Change as per your resolution
let roundedTime = Math.floor(new Date(msg.time) / 60000.0) * 60
// If the candles hashmap doesnt have this product id create an empty object for that id
if (!candles[productId]) {
candles[productId] = {}
}
// If the current product's candle at the latest rounded timestamp doesnt exist, create it
if (!candles[productId][roundedTime]) {
//Before creating a new candle, lets mark the old one as closed
let lastCandle = lastCandleMap[productId]
if (lastCandle) {
lastCandle.closed = true;
delete candles[productId][lastCandle.timestamp]
}
// Set Quote Volume to -1 as GDAX doesnt supply it
candles[productId][roundedTime] = {
timestamp: roundedTime,
open: msg.price,
high: msg.price,
low: msg.price,
close: msg.price,
baseVolume: msg.size,
quoteVolume: -1,
closed: false
}
}
// If this timestamp exists in our map for the product id, we need to update an existing candle
else {
let candle = candles[productId][roundedTime]
candle.high = msg.price > candle.high ? msg.price : candle.high
candle.low = msg.price < candle.low ? msg.price : candle.low
candle.close = msg.price
candle.baseVolume = parseFloat((candle.baseVolume + msg.size).toFixed(PRECISION))
// Set the last candle as the one we just updated
lastCandleMap[productId] = candle
}
})
What they're suggesting is to get the historic rates https://docs.gdax.com/#get-historic-rates from this endpoint, then keep your candles up to date using the Websocket feed messages - whenever a 'match'/ticker message is received, you update the last candle accordingly.
From the docs: "The maximum number of data points for a single request is 300 candles. If your selection of start/end time and granularity will result in more than 300 data points, your request will be rejected." so probably why you can't get more than 2 days worth of data.
ps. I have a live orderbook implemented here and a basic candle stick chart - lots not hooked together fully still but its at least available for preview until its complete https://github.com/robevansuk/gdax-java/
I have been stuck with this problem for a month.
I've been trying to use firebase to create a FIFO Inventory. I am using firebase cloud function to update the FIFO inventory. However if I stress test the following code just 10 times with for loop for both insert (push) and remove (pop), it breaks because of the concurrent update.
Does anyone have another solution for this?
Insert FIFO/PUSH:
let fifoRef = admin.database().ref('fifo/' + item.itemId + '/').push();
let fifo = {
price: data.items[uniqueId].price,
in: data.items[uniqueId].quantity,
quantity: data.items[uniqueId].quantity,
}
fifoRef.set(fifo);
Get FIFO Value/POP (Here I simply update the quantity for POP):
// get fifo
let fifoReference = 'fifo/' + item.itemId;
let fifoRef = admin.database().ref(fifoReference);
fifoRef.once('value').then(currentData => {
let fifo = currentData.val();
for (let key in fifo) {
let val = fifo[key];
if (val.quantity > 0) {
// get fifo quantity
let fifoQuantityRef = admin.database().ref(fifoReference + '/' + key + '/quantity/');
// get local cache value
let fifoQuantityListener = fifoQuantityRef.on('value', () => {
// transaction start
fifoQuantityRef.transaction(function (quantity) {
if (quantity) {
if (quantity > 0 && quantitySubtotal > 0) {
if (quantity >= quantitySubtotal) {
// minus inventory amount
let amount = calculator(quantitySubtotal + "*" + val.price);
quantity = calculator(quantity + "-" + quantitySubtotal);
quantitySubtotal = 0;
// update quantity
return quantity;
} else {
let amount = calculator(quantity + "*" + val.price);
quantitySubtotal = calculator(quantitySubtotal + "-" + quantity);
return 0;
}
}
}
return quantity;
}, (error, committed, result) => {
fifoQuantityRef.off('value', fifoQuantityListener);
}, true);
});
Brainstorming:
I just need Insight on how to get the VALUE using FIFO. From my understanding Firebase is best used to insert and remove not with transaction. But if I am using only insert and remove, how to create a FIFO? If I create FIFO with the quantity of 1, the data stored will be too large.
I did try to use Google Datastore, however datastore persistence is extremely slow (over 2 seconds for writing). Which is not possible to be used with firebase which have persistence of less than 1 second. The problem arise when PUSH and POP is done within 1 seconds, datastore insert is not persisted yet.
Any other brainstorming ideas?
We are developing a client script that will auto update the rate or unit price (*this is a custom column), but it throws an error and I think its because if we change the rate it will auto update the unit price and after the unit price has been updated it will update the rate also, so on and so on until it reach the maximum call stack, We are looking for any workaround for this not to happen, please see my code below. Thanks
function fieldChanged(type, name, linenum)
{
var context = nlapiGetContext();
var user = context.getUser();
var execution = context.getExecutionContext();
try {
switch (name) {
case "custcol_cqwst_po_uprice":
if (execution == "userinterface") {
var qty = nlapiGetCurrentLineItemValue('item', 'quantity');
var taxCode = nlapiGetCurrentLineItemValue('item', 'taxcode');
if (!isNullOrEmpty(taxCode) && !isNullOrEmpty(qty)) {
var taxRate = nlapiGetCurrentLineItemValue('item', 'taxrate1');
var unitprice = nlapiGetCurrentLineItemValue('item', 'custcol_cqwst_po_uprice');
var vatRate = 1 + ((taxRate.replace('%', '')) / 100);
var unitRate = unitprice / vatRate;
nlapiSetCurrentLineItemValue('item', 'rate', unitRate);
}
}
break;
//case "taxcode":
//break;
case "rate":
if (execution == "userinterface") {
var qty = nlapiGetCurrentLineItemValue('item', 'quantity');
var taxCode = nlapiGetCurrentLineItemValue('item', 'taxcode');
if (!isNullOrEmpty(taxCode) && !isNullOrEmpty(qty)) {
var taxRate = nlapiGetCurrentLineItemValue('item', 'taxrate1');
var rate = nlapiGetCurrentLineItemValue('item', 'rate');
var vatRate = 1 + ((taxRate.replace('%', '')) / 100);
var unitPrice = rate * vatRate;
nlapiSetCurrentLineItemValue('item', 'custcol_cqwst_po_uprice', unitPrice);
nlapiLogExecution('debug', "Rate Value", "Rate: " + rate + " Vat Rate: " + vatRate + " Unit Price: " + unitPrice + " Execution: " + execution);
}
}
break;
}
}
catch (ex) {
alert("A scripting problem occurred during the onFieldChange event please inform an administrator quoting the following error: " + ((ex instanceof nlobjError) ? ex.getCode() + '\n' + ex.getDetails() : ex.toString()));
}
}
nlapiSetCurrentLineItemValues() has a fourth paramater called firefieldchanged which can be set to false to prevent exactly this issue
Using the Fire Field Changed Parameter
When creating scripts that provide the ability to watch a field for a
change, and then write back to the field that changed, a risk of
creating an infinite loop exists as follows:
The Client script watches for fieldA to change.
fieldA changes.
The script writes to fieldA, causing the Field Changed event to fire,
returning the code to step 2, and this loop repeats indefinitely.
To prevent this looping behavior, you can set the optional
firefieldchanged parameter in your client scripts.
The firefieldchanged parameter is available for all write functions.
If set to true, the parameter causes any field changed events to fire
as normal. This is the default setting. If set to false, field changed
events are NOT fired.
https://system.netsuite.com/app/help/helpcenter.nl?fid=section_N3042487.html#bridgehead_N3050839
I am wondering how can I achieve pagination using Cassandra.
Let us say that I have a blog. The blog lists max 10 posts per page. To access next posts a user must click on pagination menu to access page 2 (posts 11-20), page 3 (posts 21-30), etc.
Using SQL under MySQL, I could do the following:
SELECT * FROM posts LIMIT 20,10;
The first parameter of LIMIT is offset from the beginning of result set and second argument is amount of rows to fetch. The example above returns 10 rows starting from row 20.
How can I achieve the same effect in CQL?
I have found some solutions on Google, but all of them require to have "the last result from previous query". It works for having "next" button to paginate to another 10-results-set, but what if I want to jump from page 1 to page 5?
You don't need to use tokens, if you are using Cassandra 2.0+.
Cassandra 2.0 has auto paging.
Instead of using token function to create paging, it is now a built-in feature.
Now developers can iterate over the entire result set, without having to care that it’s size is larger than the memory. As the client code iterates over the results, some extra rows can be fetched, while old ones are dropped.
Looking at this in Java, note that SELECT statement returns all rows, and the number of rows retrieved is set to 100.
I’ve shown a simple statement here, but the same code can be written with a prepared statement, couple with a bound statement. It is possible to disable automatic paging, if it is not desired. It is also important to test various fetch size settings, since you will want to keep the memorize small enough, but not so small that too many round-trips to the database are taken. Check out this blog post to see how paging works server side.
Statement stmt = new SimpleStatement(
"SELECT * FROM raw_weather_data"
+ " WHERE wsid= '725474:99999'"
+ " AND year = 2005 AND month = 6");
stmt.setFetchSize(24);
ResultSet rs = session.execute(stmt);
Iterator<Row> iter = rs.iterator();
while (!rs.isFullyFetched()) {
rs.fetchMoreResults();
Row row = iter.next();
System.out.println(row);
}
Try using the token function in CQL:
https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useToken.html
Another suggestion, if you are using DSE, solr supports deep paging:
https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
Manual Paging
The driver exposes a PagingState object that represents where we were in the result set when the last page was fetched:
ResultSet resultSet = session.execute("your query");
// iterate the result set...
PagingState pagingState = resultSet.getExecutionInfo().getPagingState();
This object can be serialized to a String or a byte array:
String string = pagingState.toString();
byte[] bytes = pagingState.toBytes();
This serialized form can be saved in some form of persistent storage to be reused later. When that value is retrieved later, we can deserialize it and reinject it in a statement:
PagingState pagingState = PagingState.fromString(string);
Statement st = new SimpleStatement("your query");
st.setPagingState(pagingState);
ResultSet rs = session.execute(st);
Note that the paging state can only be reused with the exact same statement (same query string, same parameters). Also, it is an opaque value that is only meant to be collected, stored an re-used. If you try to modify its contents or reuse it with a different statement, the driver will raise an error.
Src: https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshPaging.html
If you read this doc "Use paging state token to get next result",
https://datastax.github.io/php-driver/features/result_paging/
We can use "paging state token" to paginate at application level.
So PHP logic should look like,
<?php
$limit = 10;
$offset = 20;
$cluster = Cassandra::cluster()->withContactPoints('127.0.0.1')->build();
$session = $cluster->connect("simplex");
$statement = new Cassandra\SimpleStatement("SELECT * FROM paging_entries Limit ".($limit+$offset));
$result = $session->execute($statement, new Cassandra\ExecutionOptions(array('page_size' => $offset)));
// Now $result has all rows till "$offset" which we can skip and jump to next page to fetch "$limit" rows.
while ($result->pagingStateToken()) {
$result = $session->execute($statement, new Cassandra\ExecutionOptions($options = array('page_size' => $limit,'paging_state_token' => $result->pagingStateToken())));
foreach ($result as $row) {
printf("key: '%s' value: %d\n", $row['key'], $row['value']);
}
}
?>
Although the count is available in CQL, so far I have not seen a good solution for the offset part...
So... one solution I have been contemplating was to create sets of pages using a background process.
In some table, I would create the blog page A as a set of references to page 1, 2, ... 10. Then another entry for blog page B pointing to pages 11 to 20, etc.
In other words, I would build my own index with a row key set to the page number. You may still make it somewhat flexible since you can offer the user to choose to see 10, 20 or 30 references per page. For example, when set to 30, you display sets 1, 2, and 3 as page A, sets 4, 5, 6 as page B, etc.)
And if you have a backend process to handle all of that, you can update your lists as new pages are added and old pages are deleted from the blog. The process should be really fast (like 1 min. for 1,000,000 rows if even that slow...) and then you can find the pages to display in your list pretty much instantaneously. (Obviously, if you are to have thousands of users each posting hundreds of pages... that number can grow quickly.)
Where it becomes more complicated is if you wanted to offer a complex WHERE clause. By default a blog shows you a list of all the posts from the newest to the oldest. You could also offer lists of posts with tag Cassandra. Maybe you want to inverse the order, etc. That makes it difficult unless you have some form of advanced way to create your index(es). On my end I have a C-like language which goes and peek and poke to the values in a row to (a) select them and if selected (b) to sort them. In other words, on my end I can already have WHERE clauses as complex as what you'd have in SQL. However, I do not yet break up my lists in pages. Next step I suppose...
Using cassandra-node driver for node js (koa js,marko js) : Pagination
Problem
Due to the absence of skip functionality, we need to work around. Below is the implementation of manual paging for node app in case of anyone can get an idea.
code for simple users list
navigate between next and previous page states
easy to replicate
There are two solutions which i am going to state here but only gave the code for solution 1 below,
Solution 1 : Maintain page states for next and previous records (maintain stack or whatever data structure best fit)
Solution 2 : Loop through all records with limit and save all possible page states in variable and generate pages relatively to their pageStates
Using this commented code in model, we can get all states for pages
//for the next flow
//if (result.nextPage) {
// Retrieve the following pages:
// the same row handler from above will be used
// result.nextPage();
//}
Router Functions
var userModel = require('/models/users');
public.get('/users', users);
public.post('/users', filterUsers);
var users = function* () {//get request
var data = {};
var pageState = { "next": "", "previous": "" };
try {
var userCount = yield userModel.Count();//count all users with basic count query
var currentPage = 1;
var pager = yield generatePaging(currentPage, userCount, pagingMaxLimit);
var userList = yield userModel.List(pager);
data.pageNumber = currentPage;
data.TotalPages = pager.TotalPages;
console.log('--------------what now--------------');
data.pageState_next = userList.pageStates.next;
data.pageState_previous = userList.pageStates.previous;
console.log("next ", data.pageState_next);
console.log("previous ", data.pageState_previous);
data.previousStates = null;
data.isPrevious = false;
if ((userCount / pagingMaxLimit) > 1) {
data.isNext = true;
}
data.userList = userList;
data.totalRecords = userCount;
console.log('--------------------userList--------------------', data.userList);
//pass to html template
}
catch (e) {
console.log("err ", e);
log.info("userList error : ", e);
}
this.body = this.stream('./views/userList.marko', data);
this.type = 'text/html';
};
//post filter and get list
var filterUsers = function* () {
console.log("<------------------Form Post Started----------------->");
var data = {};
var totalCount;
data.isPrevious = true;
data.isNext = true;
var form = this.request.body;
console.log("----------------formdata--------------------", form);
var currentPage = parseInt(form.hdpagenumber);//page number hidden in html
console.log("-------before current page------", currentPage);
var pageState = null;
try {
var statesArray = [];
if (form.hdallpageStates && form.hdallpageStates !== '') {
statesArray = form.hdallpageStates.split(',');
}
console.log(statesArray);
//develop stack to track paging states
if (form.hdpagestateRequest === 'next') {
console.log('--------------------------next---------------------');
currentPage = currentPage + 1;
statesArray.push(form.hdpageState_next);
pageState = form.hdpageState_next;
}
else if (form.hdpagestateRequest === 'previous') {
console.log('--------------------------pre---------------------');
currentPage = currentPage - 1;
var p_st = statesArray.length - 2;//second last index
console.log('this index of array to be removed ', p_st);
pageState = statesArray[p_st];
statesArray.splice(p_st, 1);
//pageState = statesArray.pop();
}
else if (form.hdispaging === 'false') {
currentPage = 1;
pageState = null;
statesArray = [];
}
data.previousStates = statesArray;
console.log("paging true");
totalCount = yield userModel.Count();
var pager = yield generatePaging(form.hdpagenumber, totalCount, pagingMaxLimit);
data.pageNumber = currentPage;
data.TotalPages = pager.TotalPages;
//filter function - not yet constructed
var searchUsers = yield userModel.searchList(pager, pageState);
data.usersList = searchUsers;
if (searchUsers.pageStates) {
data.pageStates = searchUsers.pageStates;
data.next = searchUsers.nextPage;
data.pageState_next = searchUsers.pageStates.next;
data.pageState_previous = searchUsers.pageStates.previous;
//show previous and next buttons accordingly
if (currentPage == 1 && pager.TotalPages > 1) {
data.isPrevious = false;
data.isNext = true;
}
else if (currentPage == 1 && pager.TotalPages <= 1) {
data.isPrevious = false;
data.isNext = false;
}
else if (currentPage >= pager.TotalPages) {
data.isPrevious = true;
data.isNext = false;
}
else {
data.isPrevious = true;
data.isNext = true;
}
}
else {
data.isPrevious = false;
data.isNext = false;
}
console.log("response ", searchUsers);
data.totalRecords = totalCount;
//pass to html template
}
catch (e) {
console.log("err ", e);
log.info("user list error : ", e);
}
console.log("<------------------Form Post Ended----------------->");
this.body = this.stream('./views/userList.marko', data);
this.type = 'text/html';
};
//Paging function
var generatePaging = function* (currentpage, count, pageSizeTemp) {
var paging = new Object();
var pagesize = pageSizeTemp;
var totalPages = 0;
var pageNo = currentpage == null ? null : currentpage;
var skip = pageNo == null ? 0 : parseInt(pageNo - 1) * pagesize;
var pageNumber = pageNo != null ? pageNo : 1;
totalPages = pagesize == null ? 0 : Math.ceil(count / pagesize);
paging.skip = skip;
paging.limit = pagesize;
paging.pageNumber = pageNumber;
paging.TotalPages = totalPages;
return paging;
};
Model Functions
var clientdb = require('../utils/cassandradb')();
var Users = function (options) {
//this.init();
_.assign(this, options);
};
Users.List = function* (limit) {//first time
var myresult; var res = [];
res.pageStates = { "next": "", "previous": "" };
const options = { prepare: true, fetchSize: limit };
console.log('----------did i appeared first?-----------');
yield new Promise(function (resolve, reject) {
clientdb.eachRow('SELECT * FROM users_lookup_history', [], options, function (n, row) {
console.log('----paging----rows');
res.push(row);
}, function (err, result) {
if (err) {
console.log("error ", err);
}
else {
res.pageStates.next = result.pageState;
res.nextPage = result.nextPage;//next page function
}
resolve(result);
});
}).catch(function (e) {
console.log("error ", e);
}); //promise ends
console.log('page state ', res.pageStates);
return res;
};
Users.searchList = function* (pager, pageState) {//paging filtering
console.log("|------------Query Started-------------|");
console.log("pageState if any ", pageState);
var res = [], myresult;
res.pageStates = { "next": "" };
var query = "SELECT * FROM users_lookup_history ";
var params = [];
console.log('current pageState ', pageState);
const options = { pageState: pageState, prepare: true, fetchSize: pager.limit };
console.log('----------------did i appeared first?------------------');
yield new Promise(function (resolve, reject) {
clientdb.eachRow(query, [], options, function (n, row) {
console.log('----Users paging----rows');
res.push(row);
}, function (err, result) {
if (err) {
console.log("error ", err);
}
else {
res.pageStates.next = result.pageState;
res.nextPage = result.nextPage;
}
//for the next flow
//if (result.nextPage) {
// Retrieve the following pages:
// the same row handler from above will be used
// result.nextPage();
//}
resolve(result);
});
}).catch(function (e) {
console.log("error ", e);
info.log('something');
}); //promise ends
console.log('page state ', pageState);
console.log("|------------Query Ended-------------|");
return res;
};
Html side
<div class="box-footer clearfix">
<ul class="pagination pagination-sm no-margin pull-left">
<if test="data.isPrevious == true">
<li><a class='submitform_previous' href="">Previous</a></li>
</if>
<if test="data.isNext == true">
<li><a class="submitform_next" href="">Next</a></li>
</if>
</ul>
<ul class="pagination pagination-sm no-margin pull-right">
<li>Total Records : $data.totalRecords</li>
<li> | Total Pages : $data.TotalPages</li>
<li> | Current Page : $data.pageNumber</li>
</ul>
</div>
I am not very much experienced with node js and cassandra db, this solution can surely be improved. Solution 1 is working example code to start with the paging idea. Cheers
a detailed blog.
Our use case was similar. Pull everything from a Cassandra table (cassandra does it smartly by fetching ~5000 in one go and return a cursor), heavy personalized processing on each row, and keep going. Once our iteration reaches close to 5000, it again fetches the next chunk of 5000 rows internally and adds it to the result cursor. It does it so brilliantly that we don’t even feel this magic happening behind the scene.
but It became a bottleneck for us.As iterating over the chunk took some time and till it reached the end of the chunk, Cassandra thought the connection was not being used and closed the connection automatically yelling, its timeout. So we implemented with page state.
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
from cassandra.query import SimpleStatement
# connection with cassandra
cluster = Cluster(["127.0.0.1"], auth_provider=PlainTextAuthProvider(username="pankaj", password="pankaj"))
session = cluster.connect()
# setting keyspace
session.set_keyspace("my_keyspace")
# set fetch size
fetch_size = 100
# It will print first 100 records
next_page_available = True
paging_state = None
data_count = 0
while next_page_available is True:
# fetches a new chunk with given page state
result = fetch_a_fresh_chunk(paging_state)
paging_state = results.paging_state
for result in results:
# process payload here.....
# payload processed
data_count += 1
# once we reach fetch size, we stop cassandra to fetch more chunk, internally
if data_count == fetch_size:
i = 0
break
# fetches a fresh chunk with given page state
def fetch_a_fresh_chunk(paging_state = None)
query = "SELECT * FROM my_cute_cassandra_table;"
statement = SimpleStatement(query, fetch_size = fetch_size)
results = session.execute(statement, paging_state=paging_state)