I am using the edgar api to retrieve XBRL form data such as 10-K, 8-K etc.
I start by hitting the https://data.sec.gov/submissions/CIK${this.cik}.json which provides the recent filings. A recent filing has the following props:
type RecentFile = {
accessionNumber: string;
filingDate: string;
reportDate: string;
acceptanceDateTime: string;
act: string;
form: string;
fileNumber: string;
filmNumber: string;
items: string;
size: number;
isXBRL: number;
isInlineXBRL: number;
primaryDocument: string;
primaryDocDescription: string;
}
I am mainly interested in the XBRL documents so I am filtering the results:
const XBRLs = recentFilings.filter(file => file.isXBRL);
This is where the story gets a little tricky, sometimes the primaryDocument ends with .xml which is easy and I can parse it by constructing the following URL:
${xbrlApiUrl}/${Number(this.cik)}/${accessionNumber.replaceAll("-", "")}/${primaryDocument}
where:
const xbrlApiUrl = "https://www.sec.gov/Archives/edgar/data";
Number(this.cik) so I drop the leading zeros.
and accessionNumber needs to have its - dropped
but when the primaryDocument ends with .htm sometimes the corresponding XBRL XML document is ${primaryDocument.replace(".htm", "_htm")}.xml and some other times there is no consistent rule, like the corresponding XBRL XML has dropped the form code from the .htm file name, so if primaryDocument was tsla-10q_20180930.htm then the matching XBRL document is tsla-20180930.xml and finally I also have seen cases where for example html is d668062d10k.htm but there is no obvious matching XML file in the directory.
My question is, am I missing some simple concept in order to construct the XBRL xml document URL or I just need to build a case for every edge case until all combinations are covered?
Related
Im new to dynamoDB and im trying to build an ecommerce store. I have a table with a user, product and order.
My access patterns are:
get all products in a users order
I can then use this for a similar issue with the users cart. But im not sure how. My user to order relationship is one to many and my product to order relationship is many to many.
My data looks like this:
type Variant = {
size: Sizes;
quantity: number;
price: number;
}
type OrderProduct = {
id: string;
orderId: string;
product: Product;
status: string;
trackingId: string;
}
export type Product = {
id: string;
name: string;
description: string;
category: string;
createdAt: string;
variants: Variant[];
}
export type Order = {
id: string;
userId: string;
products: OrderProduct[];
createdAt: string;
}
export type User = {
id: string;
name: string;
address: string;
}
Ive seen this on aws for many to many relationships: aws many to many relationships
But this doesnt really explain how to do a one to many and then many to many query. Any advice and help with the query would be great!
DynamoDB only allows you to query by partition key (and ranged key), or to query by indexes.
If you have different tables, you cannot do a join query. You might need to create a global secondary index and then do a query on that.
So, for instance, if your Product had a secondary index over a field called "order_id", you coud do:
const documentClient = new AWS.DynamoDB.DocumentClient();
const orderId = 1234; // the real order id
const options = {
TableName: 'Product',
IndexName: 'OrderIdIndex',
KeyConditionExpression: 'order_id = :order_id',
ExpressionAttributeValues: {
':order_id': orderId
}
};
const response = await documentClient.query(options)
Keep in mind that this example is modifying your original structure.
You might need to add that new index and attribute
Edit
Keep in mind that there might be some delay for the index propagation. For example, if you insert a new Product, and you immediately want to search using the Index by order_id, DynamoDB might tell you that there is no product (because its propagating the data). If that small delay is not acceptable, you might prefer to first query the Order, and then query each product by Id (you could use batchGet if needed)
You do not do relationship queries in Dynamo. It is not a Relational Database, it is a document database.
This means most importantly, your normal way of storing data in multiple tables and usually by some whatever unique auto incrimented identifier in an SQL is a terrible way to do it in a dynamo
Instead, you need to store your data based on your access patterns - and this may feel very weird coming from SQL! You may even feel like you are duplicating data at times.
Since a Dynamo query requires you to know what the Partition Key is in order to query (you cannot do a search or a conditional on the PK) then the PK needs to be what you have to start your query.
so with your access pattern described, your PK must be the user. Then, a separate entry for each item in their cart would be the way to proceed - basically something like:
(EDIT: you can switch User for OrderID very easily too of course)
PK: User
SK: ITEM#123456123
PK: User
SK: ITEM#123491239
PK: User
SK: Item#113322
and maybe even a
PK: User
SK: META
with attribiutes like "total items" or "login time" or "sales offered" or whatever else needs to be tracked.
then if you query against the PK of USER, you get back a list of all their items. They remove an item, you remove the SK document associated with that item. They increase the amount, then you increase that items quantity attribute. ect.
This is in effect a One to Many relationship: One (the PK of User) and Many (SK's prefixed with ITEM#) - you can then do a query of PK=User, SK (beginsWith) ITEM# to retrieve all the items of a user.
But as you may be able to see, this can get very complex very fast if you are trying to do many different relationships - dynamo is not built for that. If you need to do anything deeper than a single relationship like this or need to be able to dynamically decide the relationships/queries at run time, then Dyanmo is not the solution, SQL is.
I am new to Nest js in this we have some topics like Dto,interfaces,and schema can anyone provide clear Information on these topics.
Dto's, interfaces, and Schema are not exclusive to Nestjs.
A Data Transfer Object is an object that is used to encapsulate data, and send it from one subsystem of an application to another. In layman's terms, it formats data in a defined manner.
Example: You want data for the signup Route which is passed in Body. Then you can use DTO to filter out only the required information and striping out the rest.
//signUp.dto.ts
export class signUpDto {
#IsNotEmpty({message: "Email cannot be empty."})
#IsEmail() //class-validators can be ignored here.
email: string;
#IsNotEmpty({message: "Password cannot be Empty."})
#MinLength(6,{message: "Password must be 6 characters."})
#MaxLength(128,{message: "Password must be less than 128."})
password: string;
}
Now when you validate your incoming request body it would check for these two fields in the body and also the class validator rules if defined.
Interfaces: interfaces are used for type-checking and defining the types of data that can be passed to a controller or a Nest service. From NestJs documentation :
An Interface is an abstract type that includes a certain set of fields that a type must include to implement the interface
suppose you have a human interface you can implement a Doctor, professor, and everyone
Example:
interface Human {
eyeColor: string;
hairColor: string;
}
class Doctor implements Human{
eyeColor: string;
hairColor: string;
}
Also, Schema is how you define your entities, integrity constraints, relations, etc. in a database.
A schema can have multiple tables and have many relations such as OneToMany, ManyToOne, ManyToMany.
Hope this clears your doubts.
Given two classes:
export class Person {
public name: string;
public lastName: string;
public address: Address;
}
export class Address {
public city: string;
public street: string;
}
How would one query to see the changes to a Person address field? For example if you would like to check where some one lived in the year 2000
If you use MongoDB to hold your world state, you can perform so called Mango queries to retrieve all records that satisfy your request. If you write your application in Node.js, you could use the sqltomango package to convert SQL queries to Mango.
All these options however only apply on the current state data, not historical. To see how an object has been modified, or to see if any object at any point in time matched your search criteria, you can iterate through the historical blocks and check the transactions contained in them.
I am getting the following error with my query:
Class constructor CitizenDTO cannot be invoked without 'new'
I have tried finding information on the .from() syntax and I have not found any information oh what the first parameter should be. I have a plain old javascript type, an arrow function, and a class as shown below.
const lastProcessedAt = '2019-08-01';
await this.getConnection();
const results = await this.connection
.createQueryBuilder()
.select('*', 'citizen')
.from(CitizenDTO, 'citizen')
.where('citizen.cit_last_status_change_date >= :date', { date: lastProcessedAt })
.getMany();
and the dto:
#Entity("citizen")
export class CitizenDTO {
#PrimaryGeneratedColumn()
public cit_id: string;
#Column()
public cit_uid_serial: string;
#Column()
public cit_last_status_change_date: string;
}
I'm not sure but I think my CitizenDTO isn't designed correctly.
I found a few links, such as this one, but they didn't answer my question. The typeorm github shows the from being used this way but they did not explain the structure of their DTO either.
Okay, so I'm starting to dig into graphql a little bit, and I've built an api using koa, type-graphql, and sequelize-typescript. Everything works pretty well.... I managed to get a query working, and even managed to optimize a little bit by using graphql-fields to filter the columns I query in the database... However when I've aliased a field name, I can't seem to get the mapped name.....
For example, given the following ObjectType/Sequelize Model....
export interface IDepartment {
departmentId: number;
name: string;
description: string;
}
#ObjectType()
#Table({ underscored: true })
export class Department extends Model<Department> implements IDepartment {
#Field({ name: 'id' })
#PrimaryKey
#Column({ field: 'department_id'})
public departmentId: number;
#Field()
#Length({ max: 100 })
#Column
name: string;
#Field()
#Length({ max: 100 })
#AllowNull
#Column
description: string;
}
and sample query....
query {
department(name: "Test Dept") {
id
name,
description
}
}
sample resolver...
async department(#Arg('name') name: string, #Info() info: GraphQLResolveInfo) {
return Department.findOne({
where: { name }
});
}
This works just fine.... but when I do
async department(#Arg('name') name: string, #Info() info: GraphQLResolveInfo) {
let fields = Object.keys(getFields(info))
return Department.findOne({
attributes: fields,
where: { name }
});
}
(getFields is graphql-fields), it fails because the query specified field name id, which is what graphql-fields returns, but the column name is department_id (sequelize model name departmentId).
I've gone through the schema with a fine tooth comb, using the introspectionFromSchema function to see a detailed copy of my schema, but nowhere is there a mention of departmentId or department_id.... However I know it's out there somewhere because when I exclude the attributes field from my sequelize query, even though sequelize returns departmentId as the property name, when I return it from my resolver and it reaches the client, the property name is id.
Any help would be appreciated.... I'm trying to optimize everything by only fetching requested properties and not the entire object. I could always store the maps as separate constants and use those in my #Field definition, but I want to do that as a last resort, however if I can I'm trying to keep the code as lean as possible....
Thank you all in advance.
Unfortunately, the name option was introduced mostly to support resolvers inheritance. Using this for mapping the schema field names is a kinda undocumented feature so it's doesn't provide any mapping or exposing mapping metadata.
Using the name option for input or args types will be even worse - it will result in no access to the fields and the properties being undefined.
For now my recommendation is to just keep it simple and don't map the field names until a proper fix arrives.