Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: EatThePath on June 12, 2021, 06:56:07 pm
-
OR: More stable stables through more stables
Over the last couple months it's become increasingly clear that some things have slipped through the cracks that make the last couple stables not so stable for some prominent mods, to the point where they can't be made to work on a recent version without significant redevelopment on the mod's part. That leaves those mods either waiting multiple months for the next stable and hoping that this one will work, or having to resort to running on a nightly. Nightlies both make for a lot of modder work hunting through versions to find one that's good, and open the door for further versioning headaches. Failing that, modders are left with throwing in the towel and giving up.
I'm not trying to point any fingers about this. FSO is a big beast and the mods built against it are highly varied. Some of these breaks aren't obvious right away. But it's put a lot of strain on modders, in some cases evidently and understandably intolerable amounts. Continuing on as always and just hoping the next stables live up to the name seems unwise, to me.
To my mind the solution most likely to actually make things better is to establish a procedure for minor-version bugfix releases. If a mod identifies an issue that is keeping them from using the stable and a fix for that issue is developed, make a maintenance branch off the last stable release with that fix and that fix alone(or as little else as practical as the case may be).
It might be ideal to set some guidelines about...
- How quickly a fix version is to be released after you identify the fix. You'd likely want to give it at least a few days to get some testing, make sure it actually fixes the issue and doesn't create new ones, and incorporate new fixes if other critical issues come up.
- How frequently they can be expected. If a fix version drops and a new critical issue comes out the next day, do you immediately kick in to the process again, or is there a cooldown period to make sure the team doesn't burn out on bug whack-a-mole and has more time to make sure they get it right this time?
I don't know what the optimal answers to these questions are, and situational flexibility would be needed. but I think having established and public answers to them would help set community expectations. It'd also make it clearer when it's appropriate to prod the dev team about something; sometimes life happens and things fall through the cracks, but from the outside it's hard to tell if that's happened or if the wheels of code are just turning slowly.
I'm sure there's considerations here I'm overlooking, but if a good procedure for something like this can be found I think it will help bring down stress levels for everyone a bit in the end.
-
Speaking only for myself and not on SCP's behalf....
I can't speak to how well bugfix releases alone will address long-term mod compatibility issues (for example, Phantom Hoover's concerns here (https://www.hard-light.net/forums/index.php?topic=97606.msg1909981#msg1909981)) but I think regular maintenance releases between stable releases could be a good idea.
A more stable engine can make modders (and players) happier, and as the wiki once said, happy modders make more content.
One idea is a temporary maintenance branch between stable releases, like EatThePath suggested, but with all important fixes ported to it (definition of "important" TBD), rather than only a fix for a specific mod. But I understand that including any non-essential fix carries some risk for a mod.
As EatThePath said, waiting a few days after the fix appears in a nightly before porting to the maintenance branch could be a good idea.
To simplify maintenance releases, I suggest a predictable release schedule (monthly?) but I guess a one-off hotfix could be released for a critical issue.
Maybe it's a bad idea, but my $0.02, for whatever it's worth.
At least we're past the not-so-good old days of stable releases occurring once a year (or even less).
-
The quarterly release cycle is intended to improve FSO's stability in precisely this way. We have a mini-code freeze before every release, precisely for the purpose of squashing any outstanding bugs.
It doesn't take much effort to post an official build, but it does take a fair amount of effort to manage the release cycle. I don't know that we have enough manpower to produce maintenance releases with everything else going on. PR reviews take a lot of work too, and there are a lot of pull requests currently outstanding that need reviewing.
-
What could work is for newer members to look at bug fixes and see if there are any that should be applied, wait a few nightles to make sure that the bug is really fixed, then just recommends that they be back ported to the stable release.
With the amount of time that I've had, I barely feel like I've been able to do anything, so I know I wouldn't be able to do that.
-
Not everything that causes compatibility issues is so clearly a bug. I've brought up this example before and at the risk of sounding like a broken record, I'm going to do so again because it's the perfect example of the kind of breakage mods encounter without expecting it. It's also a pretty good example of how SCP handles these types of issues (see: they don't).
BtA had a mission with an event tied to when the AI fires missiles. The idea was that the event would trigger when the AI hostiles get within firing range and actually start firing. There's a thousand ways to FRED such a thing, but this is the way the mission designer chose. This bug did not exist at release or for several years following. Then at some point unknown the AI started firing as soon as they arrive in the mission even though they are out of range of any valid target. This caused the event tied to AI missiles firing to trigger too early, breaking the mission.
It's easy to fix in FRED, which is what I did. But the implications of this small AI change are pretty broad. Outside of just the sexp, this implies that across the board the AI has fundamentally changed when they choose to fire missiles which can have far reaching effects on missions and mission balance. I know SCP is aware of the issue. Asteroth scolded me for months about not fixing the mission in BtA (Notice how he expected me to fix it in the mission even though it's the engine that broke it). Is anyone in SCP even interested in looking at why this behavior changed? I suspect not and that we're just gonna roll with it now.
To be clear. I don't need this fixed. I've already spent many months revamping my old missions to work on newer builds. But this is the kind of stuff that changes frequently enough to be frustrating between stable builds. For mods that are in active development that want to use newer features, there's no way around this. The breakage almost always comes from difficult to debug or otherwise very muddy actions in the engine. The kind of stuff I would be very wary of backporting to anything resembling 'stable'.
And no, this bug is not on GitHub. My last couple of bug reports have gone completely ignored, sooo... ¯\_(ツ)_/¯
-
And no, this bug is not on GitHub. My last couple of bug reports have gone completely ignored, sooo... ¯\_(ツ)_/¯
I helped you with two of those recently. You have two open non-mantis bugs on github. (thank you, again, for transferring those mantis bugs over) One of them was admittedly ignored, but so have the 6 surrounding bugs, meaning that no one has had time to address them. The only other one that hasn't been resolved since you posted it is the bug for the ships going ridiculous speeds, but we were never able to reliably reproduce that one.
All the members are stretched and we're falling behind on pretty much everything. Only Goober and Asteroth have been able to regularly submit pr's. The most I've been able to do is review the stuff Asteroth has submitted. He's been submitting features and bugfixes.
-
Issue 3375 and 3367. Both of which I've had to come up with workarounds for. But that's not the point of this thread.
-
The quarterly release cycle is intended to improve FSO's stability in precisely this way. We have a mini-code freeze before every release, precisely for the purpose of squashing any outstanding bugs.
It doesn't take much effort to post an official build, but it does take a fair amount of effort to manage the release cycle. I don't know that we have enough manpower to produce maintenance releases with everything else going on. PR reviews take a lot of work too, and there are a lot of pull requests currently outstanding that need reviewing.
The proposal here is basically to make minor patch releases to stable builds if and when regressions are caught. The problem with quarterly builds is that proactive testing before release doesn't seem to be enough to prevent a couple of big 'showstopper' bugs for one mod or another slipping through (the very subtle resequencing of AI, physics and goal code which broke BP checkpoints in 21.0.0 is a great example), and it's frankly embarrassing to be telling modders hit by that kind of thing that they'll have to use the next quarterly stable build, which may have introduced more regressions with new features, to fix it. I would really like to see incremental patches, say 21.0.1 for instance, that have precision fixes to major regressions found in the wild. I really hope that we can get the release requirements for these down enough to be practical, because I get an alarming sense of "two steps forward, one step back" with every release under the current arrangement.
-
I really hope that we can get the release requirements for these down enough to be practical, because I get an alarming sense of "two steps forward, one step back" with every release under the current arrangement.
I agree. It's not a great system.
-
Not everything that causes compatibility issues is so clearly a bug. I've brought up this example before and at the risk of sounding like a broken record, I'm going to do so again because it's the perfect example of the kind of breakage mods encounter without expecting it. It's also a pretty good example of how SCP handles these types of issues (see: they don't).
BtA had a mission with an event tied to when the AI fires missiles. The idea was that the event would trigger when the AI hostiles get within firing range and actually start firing. There's a thousand ways to FRED such a thing, but this is the way the mission designer chose. This bug did not exist at release or for several years following. Then at some point unknown the AI started firing as soon as they arrive in the mission even though they are out of range of any valid target. This caused the event tied to AI missiles firing to trigger too early, breaking the mission.
It's easy to fix in FRED, which is what I did. But the implications of this small AI change are pretty broad. Outside of just the sexp, this implies that across the board the AI has fundamentally changed when they choose to fire missiles which can have far reaching effects on missions and mission balance. I know SCP is aware of the issue. Asteroth scolded me for months about not fixing the mission in BtA (Notice how he expected me to fix it in the mission even though it's the engine that broke it). Is anyone in SCP even interested in looking at why this behavior changed? I suspect not and that we're just gonna roll with it now.
If you're referring to the Mefistofele issue, I undertook an extensive debugging session, at the time that you posted the issue, to figure out the root cause. It was caused by SEXPs, not AI, and it was a buggy SEXP that had its behavior masked by a bug in FSO. (I believe Lepanto was the author of the buggy SEXPs in question.)
When the FSO bug was fixed, the SEXP bug revealed itself. But I also described how the SEXP bug could be fixed, again at the time it was posted. This is all documented on Discord.
The quarterly release cycle is intended to improve FSO's stability in precisely this way. We have a mini-code freeze before every release, precisely for the purpose of squashing any outstanding bugs.
It doesn't take much effort to post an official build, but it does take a fair amount of effort to manage the release cycle. I don't know that we have enough manpower to produce maintenance releases with everything else going on. PR reviews take a lot of work too, and there are a lot of pull requests currently outstanding that need reviewing.
The proposal here is basically to make minor patch releases to stable builds if and when regressions are caught.
I don't disagree with the proposal, per se, I'm just saying that it would involve a certain amount of work. In addition to branching and patching, one would have to collect a list of bugs, filter out the fixes to be applied, and merge them.
-
Not everything that causes compatibility issues is so clearly a bug. I've brought up this example before and at the risk of sounding like a broken record, I'm going to do so again because it's the perfect example of the kind of breakage mods encounter without expecting it. It's also a pretty good example of how SCP handles these types of issues (see: they don't).
BtA had a mission with an event tied to when the AI fires missiles. The idea was that the event would trigger when the AI hostiles get within firing range and actually start firing. There's a thousand ways to FRED such a thing, but this is the way the mission designer chose. This bug did not exist at release or for several years following. Then at some point unknown the AI started firing as soon as they arrive in the mission even though they are out of range of any valid target. This caused the event tied to AI missiles firing to trigger too early, breaking the mission.
It's easy to fix in FRED, which is what I did. But the implications of this small AI change are pretty broad. Outside of just the sexp, this implies that across the board the AI has fundamentally changed when they choose to fire missiles which can have far reaching effects on missions and mission balance. I know SCP is aware of the issue. Asteroth scolded me for months about not fixing the mission in BtA (Notice how he expected me to fix it in the mission even though it's the engine that broke it). Is anyone in SCP even interested in looking at why this behavior changed? I suspect not and that we're just gonna roll with it now.
If you're referring to the Mefistofele issue, I undertook an extensive debugging session, at the time that you posted the issue, to figure out the root cause. It was caused by SEXPs, not AI, and it was a buggy SEXP that had its behavior masked by a bug in FSO. (I believe Lepanto was the author of the buggy SEXPs in question.)
When the FSO bug was fixed, the SEXP bug revealed itself. But I also described how the SEXP bug could be fixed, again at the time it was posted. This is all documented on Discord.
Well I apologize then. Of course none of that Discord documentation is available to me. Thank you mageking.
-
It's okay. The discussion was more than a year ago so one wouldn't necessarily remember the details.
Neither of the issues you mentioned above are within my usual areas of expertise, but I might be able to figure out 3367, so I'll take a look at that.
-
It doesn't take much effort to post an official build, but it does take a fair amount of effort to manage the release cycle. I don't know that we have enough manpower to produce maintenance releases with everything else going on. PR reviews take a lot of work too, and there are a lot of pull requests currently outstanding that need reviewing.
The proposal here is basically to make minor patch releases to stable builds if and when regressions are caught.
I don't disagree with the proposal, per se, I'm just saying that it would involve a certain amount of work. In addition to branching and patching, one would have to collect a list of bugs, filter out the fixes to be applied, and merge them.
While I'm glad Goober agrees on the merits of the idea, I do have to agree with him that it would take a non-trivial amount of dedicated work to keep a series of maintenance releases going, and finding someone among the current active SCP coders who can do that is hard. :blah: Unless, of course, someone currently not on SCP would like to step up? ;7 I don't think it'd require any serious/deep coding skills, just ability/willingness to learn and dedicate to it. Just saying :nod:
Mjn, I don't know anything about the code pertaining to Issue #3375 (https://github.com/scp-fs2open/fs2open.github.com/issues/3375), but I can take a look. Maybe I can at least narrow down the cause.
-
Well I had the fun of going through the forums and the two SCP channels on discord to get up to speed on this discussion, but I believe I have enough information now to grasp what is going on.
The proposal here is to add an official build that is based on a stable release and includes bugfixes to critical game-breaking issues that mods may stumble upon between release cycles, thus called a point release (<year>.<quarter>.<point>). This is to address unexpected issues that show up and are critical to the health of a mod, especially regressions that break older, already released mods.
So lets lay down some points:
- Point releases are bugfix-only builds
- Point releases are done outside the normal release cycle process, but are treated in similar modus operande
- Point releases are done only on the most recent stable release
- Point releases are done by the discretion of the team
- Point releases have the option of reverting a feature entirely for the sake of stability
Item 1: Point releases are bugfix only builds.
No additional features, no extra QOL. If the client wants either new feature or new QOL feature, then they'll have to wait until the next release. This is because we want the point release to isolate off any possible vector of a new bug from occurring, which is largely from new features but also from previous bugfixes. We can't predict if a bugfix nor a feature is the cause of a new bug, but isolating fixes from features should help keep the team focused on the not-fun gruntwork that bugfixing is.
Item 2: Point releases are done outside the normal release cycle process, but are treated similarly.
To keep things simple, the "point release cycle" should work in a very similar fashion to what we do with the release cycle. That is:
1. Make the fix
2. Publish the fix as a candidate
3. Test it for a week
4. If no further bugs found, and the fix is confirmed as working, the candidate is released and the point version is incremented by 1.
5. If new bugs were found or the fix did not work fully, repeat 1 through 5.
Now, some key similarities and differences here:
- Point release cycles are all-hands-on-deck situations, as are Release cycles.
- Point releases are essentially feature frozen just as with release cycles, but master is not feature frozen.
- This implies that point releases have their own branch based on the published released that diverge from master. Master can still have features worked on and merged into it while a point release is going on, but its not recommended to do so (since everybody should be focused on the bugs).
- Point releases are stable, and are only based on a stable release build or another point release
- Point release cycles are "its done when its done," just like with release cycles. The difference here is that only critical bugs and fixes will be addressed.
- Definition of what constitutes a critical bug is explained later
- Release cycles are planned, scheduled events. Point release cycles are planned, unscheduled events and are irregular in nature.
Item 3: Point fixes are done only on the most recent stable.
Bugfixing is not fun for coders. It really isn't. It can be hard, grueling, and frustrating to track down what's causing a bug, and sometimes it is even an adventure to reproduce the bug reliably in the first place. I'm not inclined to backport bugfixes from a new stable release into an older stable because somebody doesn't want the features of a newer build. For that matter, if a new feature is neither opt-in nor opt-out, it probably should not have been merged in the first place.
Item 3 also has an effect on the repository depending on the branching strategy we wind up with. If we go with having a special release branch that's based on the stable release but is isolated from master, then we can wind up with a number of vestigial branches that are left behind (as compared to having a simple, lightweight collection of tags). Merging the fixes from the point release branch into master doesn't make the point release branch go away.
Note: we still should attempt merging the fixes from the point release back into master so the fix doesn't have to be remade. This is not guaranteed to always work without conflicts, however.
If we go with having a dev branch (master) and a release branch that runs parallel to master but maintained, we can use tags on the release branch. Essentially, the release branch is all stable and after a release cycle is finished, the features are logically squashed and tidied and then pushed from dev into release. Should a point release be needed, the fixes are branched off and merged back into the release branch directly.
The dev/release dual trunklines is nice and tidy in theory, but it requires a higher degree of management than does the single dev branch and tagged releases, and risks having unstable commits accidentally merged into stable without careful review of where the commit is going. (Github doesn't make it clear enough that "User wants to merge into Master" and "User wants to merge into Stable" with their UI). Additionally our automation may need changes to support the switch.
Item 4. Point releases are done by the discretion of the SCP team.
If the team feels that it's better to wait for the next release cycle to start than to try to start a point-release cycle, then there's no need to start a point release cycle. This is a grey area depending on the state of its team members (yes, we are a volunteer organization. No, an emergency for you is not strictly an emergency for me), and exhaustive back-to-back point release cycles will make the team less likely to put their all into the next stable release cycle. Release cycles will always be our priorities, since they are more regular and thus can be scheduled for time-off to focus from paying jobs and personal schedules should the team members decide to do so. Point release cycles are only for special circumstances and should be kept at a minimum, since some team members that have been scheduled to be elsewhere won't be able to contribute to the point release cycle, and at worse cause a deadlock because of specialist knowledge about the affected code.
Item 5. Point releases have the option of reverting a feature entirely for the sake of stability
If a long point release cycle occurs, the team may decide to revert a feature that is causing them. This feature will then have to wait for the next release cycle to be available on a stable build, thus giving time for its creator to rethink its design. This is to try to make a stable build, actually stable. If a feature is so bug ridden or otherwise a pain to work with, then the team shouldn't have to exert excessively. As with Item 4, this option is at the discretion of the team.
Now, for managing the point release cycle.
- A new priority column in the Bug Triage kanban will be made for critical bugs, the kanban process for bug triage is otherwise the same. (Diagnose, Prioritize, Test a fix, Publish)
- "Point Release Candidate" and "Not a Point Release Candidate" tags go away. These are not helpful in the issue tracker after they've been merged in and can increase tag clutter. Especially since priorities are already given in the Bug Triage kanban
- Point release cycles are special, irregular circumstances that are entered the moment a critical bug is identified and does not have a feasible workaround that'll keep a mod going until the next release cycle. The Support Ninjas will be deployed and assassinate the critical bug ASAP.
- SCP members on discord will get a ping when a critical bug is diagnosed and confirmed in the kanban. This is different from regular release cycles because critical bugs are special circumstances and need to be fixed yesterday.
Finally, what defines a critical bug?
A critical bug is one in which breaks expected behavior of any mod that has been published and released (thus, a regression), or
a bug that significantly blocks the development of a new mod and can not have a feasible workaround to keep it working until the next release cycle.
Regressions are especially nasty to the SCP if ignored, as it demoralizes players and its creators and in general makes us look like assholes. Dev-stopping bugs for new mods are similarly bad for our public relations, as its creators can have enough stress as-is trying to make the mod they want without needing the additional stress from an indifferent, uncaring monolith they have to plead with to get their mod back on track. This, along with a "stable that isn't stable" goes against the moral of the community and can make the place not fun to be in if the behavior becomes expected, leading to community death.
While this may seem like an aside, the goal of point-releases is to reduce the stress on players and mod makers so they can get back to playing the game they love so much without needing to cuss out or plead the SCP to fix their beloved again. And, for that matter, to better retain players and mod makers in a community that "actually cares."
-
Regressions are especially nasty to the SCP if ignored, as it demoralizes players and its creators and in general makes us look like assholes. Dev-stopping bugs for new mods are similarly bad for our public relations, as its creators can have enough stress as-is trying to make the mod they want without needing the additional stress from an indifferent, uncaring monolith they have to plead with to get their mod back on track. This, along with a "stable that isn't stable" goes against the moral of the community and can make the place not fun to be in if the behavior becomes expected, leading to community death.
While this may seem like an aside, the goal of point-releases is to reduce the stress on players and mod makers so they can get back to playing the game they love so much without needing to cuss out or plead the SCP to fix their beloved again. And, for that matter, to better retain players and mod makers in a community that "actually cares."
I'm not going to have the ability to usefully respond to all of this till the weekend at least, but I just want to say this hits the nail on the head on why I proposed this. I think everyone cares here, in their own ways, and I hope efforts like this can help solidify that.
-
So my take on some issues:
On the effort needed to build these point/bugfix release: it is undeniable that this draws resources away from other places and will slow main line development. I’m okay with that. I’m as feature hungry as anyone, but if having more mods working immediately and more modders able to have confidence in being able to use features current or upcoming means slowing the stream of new innovations, I think that’s a net positive for the community. Obviously there’s a tipping point where the tradeoff isn’t worth it, but I think we’re at a point where adjusting priorities towards stability somewhat is appropriate.
On MJN’s point about compatibility issues that aren’t bugs strictly : It’s a grey area, but my view is this that changes to the fundamental behavior of the game that aren’t behind flags are in principle undesirable, and if they can be identified and old gameplay restored that would be suitable for a ‘point release’ especially if the effect is significant. However, it is a case of tradeoffs. Sometimes that change might be a necessary evil that enables some other change, or sometimes the old behavior might be too fragile and hard to quantify that it just can’t be pinned down, or possibly even the change has been in place long enough that you’re breaking things either way. Someone needs to make a judgement call in the end, and I think the biggest help to the community here would be some documentation of the identified or suspected issues like this and the SCP current stance on them.
All in all I think z64555’s summary of the situation and potential plan of action is generally solid and workable. The details of git branching strategy are kind of above my metaphorical pay grade, I haven’t used git for much fancy, so as long as the proper procedure is documented somewhere, whatever works for the team I’m not going to make too much fuss about.
The one thing I’d add is that I think it would probably be wise to have a thread here either for the submission and discussion of potential critical issues, or for the announcement of critical issues that have been identified/submitted with github issues. Either way, the goal is to create a stronger channel of discussion between mod and engine developers, make sure that the issues one side is aware of are known to the other side, and make both side aware of each other’s views on those issues. I’m sure there will be some contention from time to time in such a thread, but it is I think better than people being/feeling in the dark. The exchange in this very thread about the BtA issue highlights how that can increase friction needlessly.
-
Ok, so I think the effort involved in point releases has been really overblown. There's a better way to think about and organize it than what z64555 is suggesting.
Here's what should be done instead:
Rule 1:
Point releases should be limited to fixes already merged in master. This allows two channels for testing and prevents extra coder overhead.
Rule 2:
If an AI bug is found that for some reason really needs to be addressed by a point release, then the SCP needs to decide how to hide that behavior behind a flag so that in fixing one thing we don't break other mods that be that depend on the old behavior. In general those type of AI fixes would be rare, since mods would testing against the AI decisions of that stable version. So fixing the missile launching AI difference that mjn is talking about would be hidden behind a compatibility flag.
The point release cycle should be this process:
1. The SCP creates a new branch based on last stable about half of the time period to the next scheduled stable release.
2. The SCP looks at the bug fixes submitted since the last stable and filters out any that would be suspected of changing AI and balance. Fixes for new features or QOL changes written since the last release would, of course, also be filtered out.
3. Modders are giving the ability to comment on things they would like to see fixed but which aren't addressed via a form thread, GitHub or discord. So if there is an AI or other fix that was filtered out which modders really want, it can be added and put behind a compatibility flag. A list of the approved fixes would be kept on the forum thread.
4. The SCP applies those fixes to the point release branch and puts out a release candidate.
5. The community is given about 2 weeks to try out that build, since they are not the main focus of what the SCP is doing or what modders are doing.. the testing would basically be there to determine if fixing anything change the expected behavior of a mod that depends on that build.
6. Based on testing, either small adjustments are made and another release candidate put out, or the build is released.
Anything more than that is probably over-complicating the task, because the goal is to apply fixes to previous builds so that modders do not have to upgrade their mod to take advantage of fixes.
Besides the startup tasks, like creating a forum thread format, the SCP would only have the only added responsibilities of
1. Filtering already existing, applied and tested bugfixes
2. Creating a build
3. Listening to the community
4. Possibly putting a fix behind an optional flag
-
Ok, from my understanding, your proposal has the following differences from mine:
- Regular cycles halfway between full release cycles instead of as-needed
- Cherry-picks all bugfixes from master, not just critical bugfixes
- Seems to focus on AI and balance-breaking bugs rather than the broader regression category
- Hide broken features behind an opt-in flag instead of removal
- Testing cycle is 2 weeks versus the "standard" 1 week
Some disorganized comments:
Doing a regular cycle every 2 months may be too frequent, would have to test to see how well this holds up.
Cherry-picking all bugfixes from master may prove challenging, especially on merged fixes that have not been squashed.
AI and balance-breaking bugs should of course be a priority, but I feel like it leaves out regressions of existing behavior/bugfixes which can leave modders feel jaded and frustrated.
Hiding broken features behind opt-in "experimental" flags might allow for volunteer testing from other than the dev, which is good.
The 2-week testing period may be too long or too different from our release cycle's 1-week period. This is more of a habit management issue in that people could inadvertently get used to one or the other and end up disrupting the workflow. Would have to test over 6 months or so to see how well it turns out.