Is it "taboo" to programatically create system restore points? I would be doing this before I perform a software update. If there is a better method to create a restore point with just my software's files and data, please let me know.
I would like a means by which I can get the user back to a known working state if everything goes kaput during an update (closes/kills the update app, power goes out, user pulls the plug, etc.)
private void CreateRestorePoint(string description)
{
ManagementScope oScope = new ManagementScope("\\\\localhost\\root\\default");
ManagementPath oPath = new ManagementPath("SystemRestore");
ObjectGetOptions oGetOp = new ObjectGetOptions();
ManagementClass oProcess = new ManagementClass(oScope, oPath, oGetOp);
ManagementBaseObject oInParams = oProcess.GetMethodParameters("CreateRestorePoint");
oInParams["Description"] = description;
oInParams["RestorePointType"] = 12; // MODIFY_SETTINGS
oInParams["EventType"] = 100;
ManagementBaseObject oOutParams = oProcess.InvokeMethod("CreateRestorePoint", oInParams, null);
}
Is it "taboo" to programatically create system restore points?
No. That's why the API is there; so that you can have pseudo-atomic updates of the system.
Whether it's a good idea or not really depends on how much you're doing. A full system restore point is weighty - it takes time to create, disk space to store, and gets added to the interface of restore points, possibly pushing earlier restore points out of storage.
So, if your update is really only changing your application (i.e. the data it stores, the binaries that make it up, the registry entries for it), then it's not really a system level change, and I'd vote for no restore point. You can emulate the functionality by just backing up the parts you're changing, and offering a restore to backup option. My opinion is that System Restore should be to restore the system when global changes are made that might corrupt it (application install, etc).
The counter argument that one should just use the system service doesn't hold water for me; I worry that, if you have to issue a number of updates to your application, the set of system restore points might get so large that important, real "system wide" updates might get pushed out, or lost in the noise.
No, it's not Taboo - in fact, I'd encourage it. The OS manages how much hard drive takes, and I'd put money down on Microsoft spending more money & time testing System Restore than you the money & time you're putting into testing your setup application.
If you are developing an application for Vista you can use Transactional NTFS, which supports a similar feature to what you are looking for.
http://en.wikipedia.org/wiki/Transactional_NTFS
Wouldn't installer packages already include this type of rollback support, though? I'm not terribly familiar with most of them so I am not sure.
Finally, Windows will typically automatically create a restore point anytime you run a setup application.
Take a look at the following link: http://www.calumgrant.net/atomic/
The author described "Transactional Programming". This is analogous to the transactions in data bases.
Example:
Start transaction:
Step 1
Step 2
Encounter error during step 2
Roll back to before transaction started.
This is a new framework, but you can look at it more as a solution rather then using the framework.
By using transactions, you get the "Restore Points" that you're looking for.
I don't think a complete system restore would be a good plan. Two reasons that quickly come to mind:
Wasted disk space
Unintended consequences from a rollback
Related
We have a large repository of files that we want to keep in sync between one central location and multiple remote locations. Currently, this is being done using rsync, but it's a slow process mainly because of how long it takes to determine the changes.
My current thought is to find a VCS-like solution where instead of having to check all of the files, we can check the diffs between revisions to determine what gets sent over the wire. My biggest concern, however, is that we'd have to re-sync all of the files that are currently in-sync, which is a significant effort. I've been told that the current repository is about .5 TB and consists of a variety of files of different sizes. I understand that an initial commit will most likely take a significant amount of time, but I'd rather avoid the syncing between clusters if possible.
One thing I did look at briefly is git-annex, but my first concern is that it may not like dealing with thousands of files. Also, one thing I didn't see is what would happen if the file already exists on both systems. If I create a repo using git-annex on the central system and then set up repos on the remote clusters, will pushing from central to a remote repo cause it to sync all of the files?
If anyone has alternative solutions/ideas, I'd love to see them.
Thanks.
Space Issues in a filesystem on Linux
Lets call it FILESYSTEM1
Normally, space in FILESYSTEM1 is only about 40-50% used
and clients run some reports or run some queries and these reports produce massive files about 4-5GB in size and this instantly fills up FILESYSTEM1.
We have some cleanup scripts in place but they never catch this because it happens in a matter of minutes and the cleanup scripts usually clean data that is more than 5-7 days old.
Another set of scripts are also in place and these report when free space in a filesystem is less than a certain threshold
we thought of possible solutions to detect and act on this proactively.
Increase the FILESYSTEM1 file system to double its size.
set the threshold in the Alert Scripts for this filesystem to alert when 50% full.
This will hopefully give us enough time to catch this and act before the client reports issues due to FILESYSTEM1 being full.
Even though this solution works, does not seem to be the best way to deal with the situation.
Any suggestions / comments / solutions are welcome.
thanks
It sounds like what you've found is that simple threshold-based monitoring doesn't work well for the usage patterns you're dealing with. I'd suggest something that pairs high-frequency sampling (say, once a minute) with a monitoring tool that can do some kind of regression on your data to predict when space will run out.
In addition to knowing when you've already run out of space, you also need to know whether you're about to run out of space. Several tools can do this, or you can write your own. One existing tool is Zabbix, which has predictive trigger functions that can be used to alert when file system usage seems likely to cross a threshold within a certain period of time. This may be useful in reacting to rapid changes that, left unchecked, would fill the file system.
I have a QSPI flash on my embedded board.
I have a driver + process "Q" to handle reading and writing into.
I want to store variables like SW revisions, IP, operation time, etc.
I would like to ask for suggestions how to handle the different access rights to write and read values from user space and other processes.
I was thinking to have file for each variable. Than I can assign access rights for those files and process Q can change the value in file if value has been changed. So process Q will only write into and other processes or users can only read.
But I don't know about writing. I was thinking about using message queue or zeroMQ and build the software around it but i am not sure if it is not overkill. But I am not sure how to manage access rights anyway.
What would be the best approach? I would really appreciate if you could propose even totally different approach.
Thanks!
This question will probably be downvoted / flagged due to the "Please suggest an X" nature.
That said, if a file per variable is what you're after, you might want to look at implementing a FUSE file system that wraps your SPI driver/utility "Q" (or build it into "Q" if you get to compile/control source to "Q"). I'm doing this to store settings in an EEPROM on a current work project and its turned out nicely. So I have, for example, a file, that when read, retrieves 6 bytes from EEPROM (or a cached copy) provides a MAC address in std hex/colon-separated notation.
The biggest advantage here, is that it becomes trivial to access all your configuration / settings data from shell scripts (e.g. your init process) or other scripting languages.
Another neat feature of doing it this way is that you can use inotify (which comes "free", no extra code in the fusefs) to create applications that efficiently detect when settings are changed.
A disadvantage of this approach is that it's non-trivial to do atomic transactions on multiple settings and still maintain normal file semantics.
I have an application that runs on an embedded Linux device and every now and then changes are made to the software and occasionally also to the root file system or even the installed kernel.
In the current update system the contents of the old application directory are simply deleted and the new files are copied over it. When changes to the root file system have been made the new files are delivered as part of the update and simply copied over the old ones.
Now, there are several problems with the current approach and I am looking for ways to improve the situation:
The root file system of the target that is used to create file system images is not versioned (I don't think we even have the original rootfs).
The rootfs files that go into the update are manually selected (instead of a diff)
The update continually grows and that becomes a pita. There is now a split between update/upgrade where the upgrade contains larger rootfs changes.
I have the impression that the consistency checks in an update are rather fragile if at all implemented.
Requirements are:
The application update package should not be too large and it must also be able to change the root file system in the case modifications have been made.
An upgrade can be much larger and only contains the stuff that goes into the root file system (like new libraries, kernel, etc.). An update can require an upgrade to have been installed.
Could the upgrade contain the whole root file system and simply do a dd on the flash drive of the target?
Creating the update/upgrade packages should be as automatic as possible.
I absolutely need some way to do versioning of the root file system. This has to be done in a way, that I can compute some sort of diff from it which can be used to update the rootfs of the target device.
I already looked into Subversion since we use that for our source code but that is inappropriate for the Linux root file system (file permissions, special files, etc.).
I've now created some shell scripts that can give me something similar to an svn diff but I would really like to know if there already exists a working and tested solution for this.
Using such diff's I guess an Upgrade would then simply become a package that contains incremental updates based on a known root file system state.
What are your thoughts and ideas on this? How would you implement such a system? I prefer a simple solution that can be implemented in not too much time.
I believe you are looking wrong at the problem - any update which is non atomic (e.g. dd a file system image, replace files in a directory) is broken by design - if the power goes off in the middle of an update the system is a brick and for embedded system, power can go off in the middle of an upgrade.
I have written a white paper on how to correctly do upgrade/update on embedded Linux systems [1]. It was presented at OLS. You can find the paper here: https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf
[1] Ben-Yossef, Gilad. "Building Murphy-compatible embedded Linux systems." Linux Symposium. 2005.
I absolutely agree that an update must be atomic - I have started recently a Open Source project with the goal to provide a safe and flexible way for software management, with both local and remote update. I know my answer comes very late, but it could maybe help you on next projects.
You can find sources for "swupdate" (the name of the project) at github.com/sbabic/swupdate.
Stefano
Currently, there are quite a few Open Source embedded Linux update tools growing, with different focus each.
Another one that is worth being mentioned is RAUC, which focuses on handling safe and atomic installations of signed update bundles on your target while being really flexible in the way you adapt it to your application and environment. The sources are on GitHub: https://github.com/rauc/rauc
In general, a good overview and comparison of current update solutions you might find on the Yocto Project Wiki page about system updates:
https://wiki.yoctoproject.org/wiki/System_Update
Atomicity is critical for embedded devices, one of the reasons highlighted is power loss; but there could be others like hardware/network issues.
Atomicity is perhaps a bit misunderstood; this is a definition I use in the context of updaters:
An update is always either completed fully, or not at all
No software component besides the updater ever sees a half installed update
Full image update with a dual A/B partition layout is the simplest and most proven way to achieve this.
For Embedded Linux there are several software components that you might want to update and different designs to choose from; there is a newer paper on this available here: https://mender.io/resources/Software%20Updates.pdf
File moved to: https://mender.io/resources/guides-and-whitepapers/_resources/Software%2520Updates.pdf
If you are working with the Yocto Project you might be interested in Mender.io - the open source project I am working on. It consists of a client and server and the goal is to make it much faster and easier to integrate an updater into an existing environment; without needing to redesign too much or spend time on custom/homegrown coding. It also will allow you to manage updates centrally with the server.
You can journal an update and divide your update flash into two slots. Power failure always returns you to the currently executing slot. The last step is to modify the journal value. Non atomic and no way to make it brick. Even it if fails at the moment of writing the journal flags. There is no such thing as an atomic update. Ever. Never seen it in my life. Iphone, adroid, my network switch -- none of them are atomic. If you don't have enough room to do that kind of design, then fix the design.
I've got a WinCE device powered over ethernet (PoE) and I want to prevent File-System corruption following a potential power loss, e.g. user pulling the plug.
As a side note, I'm already using TexFAT which is supposed to prevent FS corruptions. While the later certainly does help reducing FS corruptions (over using plain old FAT), it doesn't entirely prevent some to still occur from time to time... So, I'm considering using a small rechargeable backup battery that would give WinCE enough time to cleanly shut down. Now, I can't find any info on the shutdown process: how to trigger it, how long it takes, and so on... MSDN is pretty quiet on this topic. Any idea?
The powerdown sequence is totally platform dependent.
The following answer is relevant to Windows CE 6. It may be different for previous versions of CE.
If you include the power manager component in your system, then the sequence is plus minus this:
Send go to D4 to all the drivers that are powermanageable and that reported they support this state. Otherwise, the driver gets the lowest powerstate it supports.
XXX_PowerDown is called, but it is not commonly used in Windows CE 6.
In between the registry is flushed in case you have a Hive Based registry and you enabled the registry flush thread. You should disable this in a fragile system such as yours
OEMPowerOff
device down
Just found a post by Bruce Eitman on what happens when Suspending. He puts it better than I do.
The Suspend sequence is what you'd do before loosing power.