Repository navigation
stratum-apps: refine shutdown signal handling (support SIGTERM) - #915
Conversation
There was a problem hiding this comment.
IIUC the stop_signal on docker-compose.yml was a workaround for the lack of proper SIGTERM handling, so it's no longer needed (cc @lucasbalieiro )
yes, exactly; historical context : #299 (comment)
--
tACK the docker deployment:
running the changes locally the containers now exit with a graceful shutdown (code 0) without the stop_signal
Container tproxy_sv2 Stopped
tproxy_sv2 exited with code 0
Container jd_client_sv2 Stopped
jd_client_sv2 exited with code 0
Container pool_sv2 Stopped
pool_sv2 exited with code 0
bit-aloo
left a comment
There was a problem hiding this comment.
Solid improvement on signal handling. Few suggestions
e5f55ad to
da8af71
Compare
bit-aloo
left a comment
There was a problem hiding this comment.
Few suggestions, once we do that. Its good to go. Thanks.
| pub async fn wait(&mut self) -> (&'static str, i32) { | ||
| tokio::select! { | ||
| _ = self.sigint.recv() => ("SIGINT", 130), // 128 + 2 | ||
| _ = self.sigterm.recv() => ("SIGTERM", 143), // 128 + 15 | ||
| } | ||
| } |
There was a problem hiding this comment.
Instead of vanilla strs and error codes, lets have this:
/// A signal asking the process to stop.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum StopSignal {
/// `SIGINT`, sent by an interactive terminal on Ctrl+C.
Interrupt,
/// `SIGTERM`, sent by service managers and container runtimes.
Terminate,
}
impl StopSignal {
/// The exit status conventionally associated with this signal.
///
/// That status is `128 + the signal number` (`SIGINT` is 2, `SIGTERM` is 15), which is what a
/// shell or a container runtime reports when a signal terminates a process. Exiting with it
/// is not the same outcome as being terminated by the signal, which a supervisor can tell
/// apart, but it is the number tooling already reads as "stopped by an operator".
pub fn exit_code(self) -> i32 {
match self {
Self::Interrupt => 130,
Self::Terminate => 143,
}
}
}
impl fmt::Display for StopSignal {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(match self {
Self::Interrupt => "SIGINT",
Self::Terminate => "SIGTERM",
})
}
}|
|
||
| impl ShutdownSignal { | ||
| /// Starts listening for `SIGINT` and `SIGTERM`. | ||
| /// |
There was a problem hiding this comment.
Lets have a new API here:
pub async fn run_until_stopped(mut self, graceful: impl Future<Output = ()>) {
let signal = self.wait().await;
info!("{signal} received — initiating graceful shutdown...");
tokio::select! {
_ = graceful => {}
signal = self.wait() => {
warn!("{signal} received again — abandoning graceful shutdown");
std::process::exit(signal.exit_code());
}
}
}This we can use for all apps startup, kinda similar code everywhere.
There was a problem hiding this comment.
I'm "ponytail" skeptic against the suggestion and I don't want to use this PR as an opportunity to introduce a new API pattern just because it feels cool
there's already repetition on main.rs of all apps (all benign IMO), and if we want to address that we should do it in a consistent way
843a008 to
0de16f2
Compare
The applications waited only on SIGINT, so SIGTERM, the conventional stop signal from service managers and container runtimes, terminated them outright with no chance to release resources. ShutdownSignal listens for both and reports whichever arrives first. Listening starts when the listener is built rather than when it is first awaited, so a caller can install it before it begins serving and cannot miss a signal that arrives during startup. Building it is fallible: a process that can observe neither signal has no shutdown path at all, and callers are expected to treat that as fatal.
Pool, JDC and tProxy now build a ShutdownSignal before calling start(), and exit when it cannot be built: carrying on without any way to observe a stop signal would leave the process killable only by SIGKILL. Listening for a signal installs a process-wide handler that is never removed, so SIGTERM no longer terminates these processes on its own. Supervisors that need a guaranteed stop still fall back to SIGKILL once their grace period expires.
The compose services overrode the stop signal with SIGINT, because the applications waited only on that one. They now handle SIGTERM as well, so the override is redundant: dropping it also means an image stopped outside this compose file, by plain `docker stop`, systemd or kubernetes, shuts down gracefully instead of being killed outright. The ENTRYPOINT comment no longer names a specific signal, since the point it makes about `exec` and PID 1 holds for either one.
0de16f2 to
fa4d57e
Compare
follow-up to #689
implements some suggestions from #914, except the tProxy sv1 draining part
IIUC the
stop_signalondocker-compose.ymlwas a workaround for the lack of properSIGTERMhandling, so it's no longer needed (cc @lucasbalieiro )