sqlalchemy and I'm trying to understand the finer workings of the way the objects work. I've got some test code that just seems to work differently than the tutorials and I am getting confused. I think this is a case of the tensorflow 1 vs 2 docs confusion happening
the code I posted, I'm told works for other people, and the last line of that is database.session.commit(), the code is the last code block.
This is a brand new Debian Stable VM with only VSCode, terminator and bpython installed beyond what is necessary for the application. THIS SHOULD WORK.
the database file is created but not populated. the tables are created but not columns or rows.
sqlite3 shell shows that nothing is in the database but the tables are there
database.session.flush()
does not add the stuff either
database.session.query(User).all()
also returns an empty thingamabob
and
database.session.query(User).filter(User.user_id)
returns : SQLAlchemy object has no attribute "query"
I'm trying to make a function that returns the user object based on its user_id and then change a variable in that user object using
object.field = "blorp"
I can make an object and access it like
user = User()
user.user_id
but for some strange reason, even though I have defaults set, it doesn't have those fields populated with those defaults. I have to explicitly define the fields during the declaration
but I can assign stuff like:
user.user_id = 2
this object returns ALL of the users right?
>>> asdf = database.session.query(User)
>>> asdf
<flask_sqlalchemy.BaseQuery object at 0x7f49aacff780>
the "all()" method returns an empty array for both class.query and database.session.query
>>> users = User.query.all()
>>> users
[]
>>> users = database.session.query(User).all()
>>> users
[]
Using the following test code:
from flask.config import Config
from flask import Flask, render_template, Response, Request ,Config
from flask_sqlalchemy import SQLAlchemy
HTTP_HOST = "gamebiscuits"
ADMIN_NAME = "Emperor of Sol"
ADMIN_PASSWORD = "password"
ADMIN_EMAIL = "game_admin"
DANGER_STRING= "TACOCAT"
class Config(object):
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + HTTP_HOST + '.db'
SQLALCHEMY_TRACK_MODIFICATIONS = True
solar_empire_server = Flask(__name__ , template_folder="templates" )
solar_empire_server.config.from_object(Config)
database = SQLAlchemy(solar_empire_server)
def update_database(thing):
database.session.add(thing)
database.commit()
def user_by_id(id_of_user):
return User.query.filter_by(user_id = id_of_user).first()
#without the flask migrate module, you need to instantiate
# databases with default values. That module wont be loaded
# yet during the creation of a NEW game
class User(database.Model):
user_id = database.Column(database.Integer, default = 0, primary_key = True)
username = database.Column(database.String(64), default = "tourist", index=True, unique=True)
email = database.Column(database.String(120), default = DANGER_STRING , index=True, unique=True)
password_hash = database.Column(database.String(128), default = DANGER_STRING)
turns_run = database.Column(database.Integer, default = 0)
cash = database.Column(database.Integer, default = 1000)
def __repr__(self):
return '<User id:{} name: {} >'.format(self.user_id , self.username)
class UserShip(User):
ship_id = database.Column(database.String(128), default = "1")
ship_name = database.Column(database.String(128), default = "goodship moop")
def __repr__(self):
return '<User id:{} name: {} >'.format(self.ship_id , self.ship_name)
admin = User(username=ADMIN_NAME, user_id = 1, email=ADMIN_EMAIL , password_hash = ADMIN_PASSWORD)
guest = User(username='guest', user_id = 2, email='test#game.net' , password_hash = 'password')
user = User()
usership = UserShip()
adminship = UserShip()
guestship = UserShip()
database.create_all()
database.session.add(admin)
database.session.add(guest)
database.session.add(user)
database.session.add(usership)
database.session.add(adminship)
database.session.add(guestship)
database.session.commit()
Okay, so when you see default here it's going to be on INSERT (see the SQLAlchemy default documentation).
To quote the key part here:
Column INSERT and UPDATE defaults refer to functions that create a default value for a particular column in a row as an INSERT or UPDATE statement is proceeding against that row, in the case where no value was provided to the INSERT or UPDATE statement for that column.
So these defaults will only be initialized on adding to the database, flushing won't have any effect if an object that hasn't been added (session.add). So User() is only going to create an object, it will not create a DB row until you add (and flush / commit as necessary).
Related
I have an issue regarding factory boy using in the testing of my Lets assume I have this three models:
Class Company(models.Model):
name = str
Class Domain(models.Model):
company = ForeignKey(ref=Company)
name = str
created_at = datetime
Class Record(models.Model):
domain = ForeignKey(ref=Domain)
name = str
created_at = datetime
CompanyFactory(factory.django.DjangoModelFactory):
name = str
DomainFactory(factory.django.DjangoModelFactory):
company = factory.SubFactory(CompanyFactory)
name = str
created_at = datetime
RecordFactory(factory.django.DjangoModelFactory):
domain = factory.SubFactory(DomainFactory)
name = str
created_at = datetime
Having this, when I'm testing the Record views, at the begginning of every view I check that the Domain object is, in fact, related to the Company object such as:
try:
domain = Domain.objects.get(domain=domain_id, company__id=company_id)
except ObjectDoesNotExist:
return Response(
data={"message": "Domain isn't related to the company provided."}, status=status.HTTP_403_FORBIDDEN
)
But this code always returns an ObjectDoesNotExist exception when I make the testing with pytest+factory-boy but when I do manual testing runs fine. Have you experienced something similar? What I'm missing here?
Thanks in advance.
As requested per #gbrennon I'm adding the test code:
Hi! Thanks for answering.
My test code is as it follows:
class RecordCompanyAdminTests(CompanyAdminUser):
def setUp(self):
super(RecordCompanyAdminTests, self).setUp()
self.domain = DomainFactory.create()
self.record = RecordFactory.create()
def test_record_list_get(self):
url = reverse("autoprovisioning:record_list", kwargs={"company_id": self.company.id, "domain_id": self.domain.id})
response = self.client.get(url, format="json")
self.assertEqual(response.status_code, status.HTTP_200_OK)
how are u doing?
the test code wasn't included but i'll try to infer things
when I do manual testing runs fine
it seems like u already populated ur database! but listen, when u run ur test suite django will be using a "test database" in favor of isolation!
what u need to do is to create, using the factory_boy lib, ur "data fixtures"!
my suggestion without any context:
class MyTestCase(TestCase):
def setUp(self):
self.existing_domain_in_database = DomainFactory.create(...) # insert here the data to populate this model
and the desired data should already exists in ur "test database" for every test that is going to be run inside of this class
The following question I present is the following: I'm doing a code that is been tested in the Scriptrunner console and tried into Jira. What I want to reach is the following: I have an issue in any project, at any moment a workflow transition allows me to copy the current issue and all the linked issues in a new project.
The cfProjectPicker is a custom field Project Picker that allows me to select a project.
What I'm having trouble right now on the code is the following:
1- projectKey is returning null valor but projectPicker is returning the correct project selected in the custom field.
2- createNewIssue is not allowing me to recognize the "params" field as a mutable issue, and it is a mutable (if I replace with newIssue it does take it)
3- the list "issuesCopied" is not allowing me to issuesCopied.id.toString() but I'm guessing is giving error due to the point 2.
Any help regarding the matter would be much appreciated because I'm not understanding what I'm doing wrong at moment.
def issueMgr = ComponentAccessor.getIssueManager();
def issueTypeSchemeManager = ComponentAccessor.getIssueTypeSchemeManager()
def customFieldManager = ComponentAccessor.getCustomFieldManager();
def projectMgr = ComponentAccessor.getProjectManager();
def groupManager = ComponentAccessor.getGroupManager();
def issueFactory = ComponentAccessor.getIssueFactory();
IssueLinkManager issueLinkManager = ComponentAccessor.getIssueLinkManager();
def cfProjectPicker = customFieldManager.getCustomFieldObjectByName("Seleccionar Proyecto") //cf project picker
def issue = issueMgr.getIssueObject("MART1-1")
ApplicationUser user = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser() //User Admin
ApplicationUser currentUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser()
def currentIssue = issueFactory.getIssue()
//def issue = issueMgr.getIssueObject()
def newIssue = issueFactory.getIssue()
def exit = false
// Array list of the issues created to save them
List<String> issuesCopied = new ArrayList<>();
// ***** CREATE ISSUE *****
if (issuesCopied.isEmpty()){
String projectPicker = issue.getCustomFieldValue(cfProjectPicker)
log.error("Gengar")
log.error(projectPicker)
def projectKey = projectMgr.getProjectObjByKey(projectPicker)
log.error(projectKey)
newIssue.setProjectObject(projectKey)
newIssue.setIssueType(issue.getIssueType())
newIssue.setSummary(issue.getSummary())
newIssue.setDescription(issue.getDescription())
newIssue.setReporter(currentUser)
def params = ["issue":newIssue]
log.error(params)
def createNewIssue = issueMgr.createIssueObject(user,params);
issuesCopied.add(createNewIssue.id.toString())
}
I try to modify many instance of some model (like User model), and this changes is different (I don't want to use update QuerySet method and not works for my scenario).
For example some user need to change first_name and some user need to change last_name and get users like : all_user = User.objects.all()
I think if I use save method for each instance after change, Django sent one query for save that!
How can I save all changes to database in one query instead of use foreach on models and save that one by one?
Given the comment from #iklinac, I would thoroughly recommend implementing django's own approach to bulk updates detailed here
It's quite similar to my original answer, below, but it looks like the functionality is now built in.
# bulk_update(objs, fields, batch_size=None)
>>> objs = [
... Entry.objects.create(headline='Entry 1'),
... Entry.objects.create(headline='Entry 2'),
... ]
>>> objs[0].headline = 'This is entry 1'
>>> objs[1].headline = 'This is entry 2'
>>> Entry.objects.bulk_update(objs, ['headline'])
Original answer
There's a package called django-bulk-update which is similar to bulk create which is builtin to django.
An example of where I use this, is part of an action in an admin class;
#admin.register(Token)
class TokenAdmin(admin.ModelAdmin):
list_display = (
'id',
'type'
)
actions = (
'set_type_charity',
)
def set_type_charity(self, request, queryset):
for token in queryset:
token.type = Token.Type.CHARITY
bulk_update(
queryset,
update_fields=['type', 'modified'],
batch_size=1000
)
Usage, taken from their readme;
With manager:
import random
from django_bulk_update.manager import BulkUpdateManager
from tests.models import Person
class Person(models.Model):
...
objects = BulkUpdateManager()
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
person.name = random.choice(random_names)
Person.objects.bulk_update(people, update_fields=['name']) # updates only name column
Person.objects.bulk_update(people, exclude_fields=['username']) # updates all columns except username
Person.objects.bulk_update(people) # updates all columns
Person.objects.bulk_update(people, batch_size=50000) # updates all columns by 50000 sized chunks
With helper:
import random
from django_bulk_update.helper import bulk_update
from tests.models import Person
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
person.name = random.choice(random_names)
bulk_update(people, update_fields=['name']) # updates only name column
bulk_update(people, exclude_fields=['username']) # updates all columns except username
bulk_update(people, using='someotherdb') # updates all columns using the given db
bulk_update(people) # updates all columns using the default db
bulk_update(people, batch_size=50000) # updates all columns by 50000 sized chunks using the default db
This code opens a twitter listener, and the search terms are in the variable, upgrades_str. Some searches work, and some don't. I added AMZN to the upgrades list just to be sure there's a frequently used term since this is using an open Twitter stream, and not searching existing tweets.
Below, I think we only need to review numbers 2 and 4.
I'm using Python 3.5.2 :: Anaconda 4.0.0 (64-bit) on Windows 10.
Variable searches
Searching with: upgrades_str: ['AMZN', 'SWK', 'AIQUY', 'SFUN', 'DOOR'] = returns tweets such as 'i'm tired of people'
Searching with: upgrades_str: ['$AMZN', '$SWK', '$AIQUY', '$SFUN', '$DOOR'] = returns tweets as as 'Chicago to south Florida. Hiphop lives'. This search is the one I wish worked.
Explicit searches
Searching by replacing the variable 'upgrades_str' with the explicit string: ['AMZN', 'SWK', 'AIQUY', 'SFUN', 'DOOR'] = returns 'After being walked in on twice, I have finally figured out how to lock the door here in Sweden'. This one at least has the search term 'door'.
Searching by replacing the variable 'upgrades_str' with the explicit string: ['$AMZN', '$SWK', '$AIQUY', '$SFUN', '$DOOR'] = returns '$AMZN $WFM $KR $REG $KIM: Amazon’s Whole Foods buy puts shopping centers at risk as real'. So the explicit call works, but not the identical variable.
Explicitly searching for ['$AMZN'] = returns a good tweet: 'FANG setting up really good for next week! Added $googl jun23 970c avg at 4.36. $FB $AMZN'.
Explicitly searching for ['cool'] returns 'I can’t believe I got such a cool Pillow!'
import tweepy
import dataset
from textblob import TextBlob
from sqlalchemy.exc import ProgrammingError
import json
db = dataset.connect('sqlite:///tweets.db')
class StreamListener(tweepy.StreamListener):
def on_status(self, status):
if status.retweeted:
return
description = status.user.description
loc = status.user.location
text = status.text
coords = status.coordinates
geo = status.geo
name = status.user.screen_name
user_created = status.user.created_at
followers = status.user.followers_count
id_str = status.id_str
created = status.created_at
retweets = status.retweet_count
bg_color = status.user.profile_background_color
blob = TextBlob(text)
sent = blob.sentiment
if geo is not None:
geo = json.dumps(geo)
if coords is not None:
coords = json.dumps(coords)
table = db['tweets']
try:
table.insert(dict(
user_description=description,
user_location=loc,
coordinates=coords,
text=text,
geo=geo,
user_name=name,
user_created=user_created,
user_followers=followers,
id_str=id_str,
created=created,
retweet_count=retweets,
user_bg_color=bg_color,
polarity=sent.polarity,
subjectivity=sent.subjectivity,
))
except ProgrammingError as err:
print(err)
def on_error(self, status_code):
if status_code == 420:
return False
access_token = 'token'
access_token_secret = 'tokensecret'
consumer_key = 'consumerkey'
consumer_secret = 'consumersecret'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
stream_listener = StreamListener()
stream = tweepy.Stream(auth=api.auth, listener=stream_listener)
stream.filter(track=upgrades_str, languages=['en'])
Here's the answer, in case someone has the problem in the future: "Note that punctuation is not considered to be part of a #hashtag or #mention, so a track term containing punctuation will not match either #hashtags or #mentions." From: https://dev.twitter.com/streaming/overview/request-parameters#track
And for multiple terms, the string, which was converted from a list, needs to be changed to ['term1,term2']. Just strip out the apostrophes and spaces:
upgrades_str = re.sub('[\' \[\]]', '', upgrades_str)
upgrades_str = '[\''+format(upgrades_str)+'\']'
I'm writing my first groovy script to update the whole database table. Basically I want the number is the ID field to be put into ANOTHER_ID field
import groovy.sql.Sql
def url = project.getProperty("database.url")
def driver = project.getProperty("database.driver")
def user = project.getProperty("database.user")
def pass = project.getProperty("database.pass")
def output = properties['output']
def sql = Sql.newInstance(url, user, pass, driver)
def outwriter = new PrintWriter(output)
sql.eachRow("select * from THE_TABLE", { row ->
def id = row.id
outwriter.println "Update THE_TABLE set (ANOTHER_ID)=(${id});"
} );
outwriter.close()
sql.close()
I'm posting to make sure I'm doing this correctly or if there is a better way of updating my table.
Cheers
There's a problem with the SQL you're generating, it should be:
outwriter.println "UPDATE THE_TABLE SET ANOTHER_ID = ${row.id} WHERE id = ${row.id} ;"
However, you can replace this whole loop and everything with
outwriter.println "UPDATE THE_TABLE SET ANOTHER_ID = id ;"
Which will set every row in a single query instead of needing an UPDATE per row