I'm trying to figure out if it is possible to prevent resource updates when one of the Resource.Schema attributes changes.
Essentially I'm building a provider that manages infrastructure. I've got a resource that updates firmware. Something like:
resource "redfish_simple_update" "update" {
transfer_protocol = "HTTP"
target_firmware_image = "/home/mikeletux/BIOS_FXC54_WN64_1.15.0.EXE"
}
As you can see, target_firmware_image does refer to the full path of my firmware package. I want to be able to change directories without triggering an update. I.e. changing above target_firmware_image by /home/mikeletux/Downloads/BIOS_FXC54_WN64_1.15.0.EXE for instance.
I don't know if this is possible. If done my own research and I found the CustomDiff functions to be added to the schema, but I think that thing doesn't match my scenario.
Do you think of something else I could do?
Thanks!
Just posting here how I finally did it.
To avoid triggering an update when the path changes but not the filename, I've found out that DiffSuppressFunc function becomes very handy here:
"target_firmware_image": {
Type: schema.TypeString,
Required: true,
Description: "Target firmware image used for firmware update on the redfish instance. " +
"Make sure you place your firmware packages in the same folder as the module and set it as follows: \"${path.module}/BIOS_FXC54_WN64_1.15.0.EXE\"",
// DiffSuppressFunc will allow moving fw packages through the filesystem without triggering an update if so.
// At the moment it uses filename to see if they're the same. We need to strengthen that by somehow using hashing
DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool {
if filepath.Base(old) == filepath.Base(new) {
return true
}
return false
},
}
By checking old and new value using filepath.Base(), I can figure out if filename is the same, no matter what path the file is placed in.
I'd like to improve that behavior in the future by implementing file hashing, so even the filename doesn't matter, but that's something I'll leave for a new version.
Thanks!
Related
context: I'm adding a new resource to TF Provider (using SDKv2) with roughly the following schema:
resource "player" "football" {
type = "FOOTBALL"
...
config = {
"dribbling" = "50"
"speed" = "90"
"position" = "GOALKEEPER"
}
}
that I represent as:
"config": {
Type: schema.TypeMap,
Elem: &schema.Schema{
Type: schema.TypeString,
},
Required: true,
ForceNew: true,
},
The important detail here for different palyer instances' types there'll be a different set of required attributes (dribbling, speed, position for football and height, can_dunk, arm_span for basketball) -- all players share the same API endpoint so I introduced just one resource to cover them all.
I'd like to support the ability of importing players and apparently READ response includes a bunch of fields that are optional on create (and I suspect most of the users won't have them in Terraform configuration file) which results in the fact that I've got a state difference when saving the whole config like:
d.Set("config", player.GetConfig()) # GetConfig includes a bunch of new attributes (optional on a create or even computed)
So I've got a question: which of the following 2 options is preferable:
Implement DiffSuppressFunc for a config attribute where I'll be ignoring these optional fields (the downside is I'll have an implicit drift between main.tf and TF state file).
Be more restrictive when writing configs to TF state file:
instead of
d.Set("config", player.GetConfig())
# filtered config will match config in main.tf
filteredConfig = ...
d.Set("config", filteredConfig)
In some other Terraform providers that deal with similar situations (where a particular argument has a mixture of configuration-provided and remote-system-provided nested values), the resource type implementation takes a compromise position of effectively exposing the same data in two different attributes, where one of them represents what the user configured and the other represents the full data returned by the remote system. For example, you might have config to be set in the configuration, and expanded_config representing the full set of elements the server decided.
There is a challenge with that approach in that you'll probably need a special rule in your Read function to somehow decide if a change you detect in the remote system constitutes "drift" relative to the configuration or if it's just an additional element added by the server.
From what you described it seems like the rule could be that any key that's present in config in the prior state (that is, the values visible to d.Get inside Read before you call d.Set) would have its value overwritten by what the server returned, but any keys that were not present before are ignored entirely. This would create the effect then that any key the author specified in the configuration is considered "managed by Terraform" while any other key is only read by Terraform and not directly managed.
If you adopt that strategy then it's worth keeping in mind what will happen in a situation where the user has changed the configuration to include a new key or to remove a previously-present key. The Read operation is in terms of the previous state rather than the configuration, so that function will see the keys that were present at the end of the last apply, not the keys currently present in the configuration. In particular this means that if an author adds a new key that the server was already tracking then it will appear in the subsequent plan as being added, even though it might technically be more appropriate to show it as an in-place update ~ or a no-op. This is an example of the compromises we sometimes need to make in order to adapt remote APIs to fit within Terraform's model of resource instances.
I've added our infrastructure setup to puppet, and used roles and profiles method. Each profile resides inside a group, based on their nature. For example, Chronyd setup and Message of the day are in "base" group, nginx-related configuration is in "app" group. Also, on the roles, each profile is added to the corresponding group. For example for memcached we have the following:
class role::prod::memcache inherits role::base::debian {
include profile::app::memcache
}
The profile::app::memcached has been set up like this :
class profile::app::memcache {
service { 'memcached':
ensure => running,
enable => true,
hasrestart => true,
hasstatus => true,
}
}
and for role::base::debian I have :
class role::base::debian {
include profile::base::motd
include profile::base::chrony
}
The above structure has proved to be flexible enough for our infrastructure. Adding services and creating new roles could not been easier than this. But now I face a new problem. I've been trying to separate data from logic, write some yaml files to keep the data there, using Hiera version 5. Been looking through internet for a couple of days, but I cannot deduct how to write my hiera files based on the structure I have. I tried adding profile::base::motd to common.yaml and did a puppet lookup, it works fine, but I could not append chrony to common.yaml. Puppet lookup returns nothing with the following common.yaml contents :
---
profile::base::motd::content: This server access is restricted to authorized users only. All activities on this system are logged. Unauthorized access will be liable to prosecution.'
profile::base::chrony::servers: 'ntp.centos.org'
profile::base::chrony::service_enable: 'true'
profile::base::chrony::service_ensure: 'running'
Motd lookup works fine. But the rest, no luck. puppet lookup profile::base::chrony::servers returns with no output. Don't know what I'm missing here. Would really appreciate the community's help on this one.
Also, using hiera, is the following enough code for a service puppet file?
class profile::base::motd {
class { 'motd':
}
}
PS : I know I can add yaml files inside modules to keep the data, but I want my .yaml files to reside in one place (e.g. $PUPPET_HOME/environment/production/data) so I can manage the code with git.
The issue was that in init.pp file inside the puppet module itself, the variable $content was assigned a value. Removing the value fixed the problem.
The Situation
I recently started working on a new project using nodejs. I have a background of using Python/Django and C#/.NET (not a huge fan of the latter). Node is awesome, but I must say I miss the ease of building models and automating migrations in Django. I am currently using the AdonisJS framework which leverages Knex. Knex is a powerful library, but the migrations all need to be manually built. Additionally, the AdonisJS ORM that manages the Models is independent of Knex (migration manager). You also do not define field attributes on the Models, which can have benifits for dynamically doing things in the front and back end. All things considered, there is a lot of room for human error, miscommunication and a boat load more typing required. I know the the hot thing these days is to keep it loose and fast, but for this specific project, I am looking for a bit more structure than loosely defined models.
Current State
What I have landed on is building a new Class called tableModel and a field class to define the fields within table model. I have already completed this and I am successfully writing the migration files leveraging mustache. I plan on also automatically writing the Models which I shouldn't have a problem with (fingers crossed).
The Problem
Here is where it gets a little tough and where I need help...I need to track what has been added or removed via migration so I can effectively write ups and downs as the tableModels change over time.
So let's say I add a "tableModel" which creates a migration to create table Foo with fields {id (bigint), user_id(int), name(string255)}
Later I want to add a field called description so I would simply add it to my "tableModel" and then run a build command which would build out the migration.
How do I check what has already been created though so I only do an up() for description?
Then I want to remove the name field so I mark it out in my "tableModel" and run a build migration command. How do I check what has been migrated that now needs to be added in to the down().
Edit: I would add a remove field to the up and the corresponding roll back to the down.
Bonus Round
Let's say I want to change user_id from an int to a bigint, because who makes a foreign key just an int? How do I check not just what needs to be added to the up and down, but also checks if I need to change a property on a field.
Edit: would just write the up. and a corresponding roll back to the down
The Big Question
Basically, how do I define dirty "tableModels" classes
Possible Solution?
I am thinking that maybe I should capture some type of registry or snapshot and then run the comparison when building the migrations and or models, then recapture/snapshot. If this is the route, should I store in a json file, write this to the DB itself, or is there another/better option.
If I create the tableModel instances as constants, could I actually write back to the JS file and capture the snapshot as an attribute? IF this is an option, is Node's file system the way to go and what's the best way to do this? Node keep suprising me so I wouldn't be baffled if any of these are an option.
Help!
If anyone has gone down this path before or knows of any tools I could leverage, I would greatly appreciate it and thank you in advance. Also, if I am headed in a completely wrong direction, then please let me know, I both handle and appreciate all types of feedback.
Example
Something to note, when I define the "tableModel" for a given migration or model, it is an instance of the class, I am not creating an extended class since this is not my orm.
class tableModel {
constructor(tableName, modelName = tableName, fields = []) {
this.tableName = tableName
this.modelName = modelName
this.fields = fields
}
// Bunch of other stuff
}
fooTableModel = new tableModel('fooTable', 'fooModel', fields = [
new tableField.stringField('title'),
new tableField.bigIntField('related_user_id'),
new tableField.textField('description','Testing Default',false,true)
]
)
which equates to:
tableModel {
tableName: 'fooTable',
modelName: 'fooModel',
fields:
[ stringField {
name: 'title',
type: 'string',
_unique: false,
allow_null: null,
fieldAttributes: {},
default_value: null },
bigIntField {
name: 'related_user_id',
type: 'bigInteger',
_unique: false,
allow_null: null,
fieldAttributes: {},
default_value: 0 },
textField {
name: 'description',
type: 'text',
_unique: false,
allow_null: true,
fieldAttributes: {},
default_value: 'Testing Default' } ]
You have the up and down notation mixed up. Those are for migrating the "latest" (runs the up function) and doing rollbacks (runs the down function). Up and down to not relate to dropping or adding table columns.
The migrations up is for any change, and the down is to reverse those changes. So if you wanted to drop a column from some table, you write the command in the up, then write the opposite in the down (you'd add it back in...), such that you can "rollback" and the change is effectively reversed. You have to be careful with such things though, as you can put yourself in a situation where you actually lose data.
Want to add a column? Write it in the up, and drop the column in the down.
One of the major points behind the migrations mechanism is to track the state of changes of your database, as time goes forward. So generally, if you created a table in some migration, then a day or so later you realize you need to drop/add columns, you normally don't go back and edit the existing migration, especially if the migration has already been run. You'd just write a new migration to drop/add your column.
Since you're using knex, there are a couple "knex" tables that get created. By default the one you're looking for is knex_migrations, unless someone specifically modified the settings to change the name of it. This table holds all the migrations that have run against your DB, per batch. From the CLI, assuming you have knex.js installed globally, you can run knex migrate:latest, and that will push all the migrations that exist in your directory to the target database, if they have not yet been run. It does this by way of examining that knex_migrations table. If you roll a change and don't like it, and assuming you've properly done the down function, you can invoke knex migrate:rollback to reverse the change. If there are 3 migration files that have NOT yet been run, invoking knex migrate:latest will run all 3 of those migration files under a new batch #, which is 1 higher than the most recent batch number. Conversely, if you invoke a knex migrate:rollback, it will find the highest batch number (there could be more than 1 migration in a batch...), and invoke the down function on all those files, effectively rollback those changes.
All that said, knex is a "query builder" tool. It's got a ton of helper functions to help build the sql for you. Personally, I find this to be a major distraction. Why spend hours on hours figuring out all the helper functions when I can just go crank out raw SQL and run that. Thus, that's what we've done in our system. we use knex.raw('') and write our own DDL and DML. It works great and does exactly what we need it to. We don't need to go figure out the magic of the query building.
The short answer is that knex will automatically know what has and has not been run for you (again, via that knex_migrations table it creates for you...).
Things can get weird though when it start involving git and different branches. I recommend that if you're writing migrations on some branch, and you need to go do other work, always remember to first perform a rollback of any migrations you've done in that branch BEFORE switching branches. Otherwise you will be in weird DB states that don't coincide with the application code.
I would personally just deal with updating models independently of writing migrations. For example, if I'm adding a description column to some table, then I probably want to manually update the ORM to reflect the change of the new db schema. Generally, I've found trying to use a tool that automagically does that for you (rather, if I change the orm, stuff happens to write all the underlying sql...) usually winds me up in a heap of trouble and I just spend more time trying to un-fudge stuff. But, that's just my 2 cents :)
Here is where it gets a little tough and where I need help...I need to track what has been added or removed via migration so I can effectively write ups and downs as the tableModels change over time.
You could store changes in a DB/txt file and those can act as snapshots. So when you want to rollback to a particular migration, you would find the changes (up/down) made for that mutation and adjust accordingly.
Later I want to add a field called description so I would simply add it to my "tableModel" and then run a build command which would build out the migration. How do I check what has already been created though so I only do an up() for description?
Here you either call the database itself directly and check what fields have already been created. If a field is already their and the attributes are the same, you can either ignore it or stop the transaction all together.
Bonus Round Let's say I want to change user_id from an int to a bigint, because who makes a foreign key just an int? How do I check not just what needs to be added to the up and down, but also checks if I need to change a property on a field.
Again, call the DB itself on the table in question. I know the SQL call would be:
describe [table_name];
After reading the end, I think you answered this yourself, but I think capturing these changes would work best in a NoSql database since you're using Node or PostGres with it's json field.
I just updated Node after not doing so for a while and had to reinstall MongoDB and other modules. Where I previously would only get an array of database objects when using the find() function, I'm now getting a JSON object that includes "results", "ops", "insertedCount", and "insertedIds". I can't remember what I might have done when I initially set it up or maybe this is just an annoying change with Mongo, but I'd like to return to only getting an array of database objects so that I don't have to test my entire server. I've tried several npm parse modules with no success.
Here's an example:
{ result: { ok: 1, n: 1 },
ops:
[ { user: '595ee2fec2924e5435dfdd2d'},
_id: 595f0fe55e84fa2468b17ce8 } ],
insertedCount: 1,
insertedIds: [ 595f0fe55e84fa2468b17ce8 ] }
Whereas previously, it would have only returned:
[ { user: '595ee2fec2924e5435dfdd2d'},
_id: 595f0fe55e84fa2468b17ce8 } ]
You can simply get the ops array.
result.ops;
You may also need to make sure to follow your call stack correctly as those objects are only returned on an insert.
You can take the resulting object and strip it down to just the array by accessing the ops attribute.
runMyQuery().then(function(res) {
return res.ops;
});
The previous example makes a lot of assumptions, so don't expect a copy-paste solution.
The most correct solution would be to continue running your project with the exact versions of everything your package.json depends on.
That said, I'm assuming you're encountering this issue is because you're running Node locally on your system and needed to upgrade it for another project or security fix. If this is the case, you may want to consider using a Node version management tool like nvm or nodenv. These allow you to have multiple versions of Node installed and associate them with individual projects so you don't run into compatibility issues.
For an even more powerful variation of this, you might want to virtualize your entire development environment using a virtual machine such as VirtualBox or a container system like Docker. Both let you create files that define how you'd like to provision your VMs or containers (what OS, what versions of software are installed, etc.). They're the most robust way of ensuring that when you come back to a project months, or even years later, it will still run exactly the way you left it.
Sounds like you inadvertently upgraded from MongoDB 3.X drivers to MongoDB 4.X drivers. The insert operations now return a much more developer friendly "InsertManyResult" object. Unfortunately, as far as I can see, this change did not make the documentation on breaking changes.
I have versioning and checkin/checkout enabled in a SharePoint 2010 list, and for business-related reasons, would like to update information in a field: overwriting the field data for all versions in a similar manner and in place so that the version numbers do not change.
If I call SPListItemVersion.ListItem.UpdateOverwriteVersion(), it fails stating that I need to check out the item before making changes to it. Makes sense. So I precede the update statement with SPListItemVersion.ListItem.CheckOut() statement, attempt the update, and receive an error that I cannot overwrite a published version. I've also attempted to precede the check out with SPSite.AllowUnsafeUpdates = true, setting it back to false after the update call, but the latter error still occurs. Any ideas?
Stripped code below:
foreach (SPListItemVersion itemVersion in item.Versions){
itemVersion.ListItem.File.CheckOut(SPFile.SPCheckOutType.Online, itemVersion.ListItem.File.TimeLastModified.ToString());
site.AllowUnsafeUpdates = true;
itemVersion.ListItem["FieldName"] = "changed value here";
itemVersion.ListItem.UpdateOverwriteVersion();
site.AllowUnsafeUpdates = false;
itemVersion.ListItem.File.CheckIn("Updated list item version", SPCheckinType.OverwriteCheckIn);
}
To anyone that may find themselves facing this issue, you do not need to toggle the AllowUnsafeUpdates field. What worked for me is replacing the itemVersion.ListItem.UpdateOverwriteVersion() statement with itemVersion.ListItem.SystemUpdate(false). The parameter passed in tells SharePoint to not create a new version when updating.
Edit: This answer only updates the current item again. SPListItemVersion field references are available via a get only; it appears this is not possible in the object model.