Capstone: Design, Trace, and Defend a Neural Network

A wearable records accelerometer and gyroscope windows. The product must recognize ordinary activities, flag possible falls, and abstain when sensor quality is poor.

Nothing in the contract below is invented. The input shape is the shape of a published public benchmark, and the decision policy is the one a shipped consumer watch already runs.

The system must run on-device within a latency and energy budget. False urgent alerts create alarm fatigue, while missed falls carry higher harm.

Alarm fatigue is a counted harm, not a figure of speech. The Joint Commission counted 98 alarm-related events in its sentinel event database between January 2009 and June 2012: “Of the 98 reported events, 80 resulted in death, 13 in permanent loss of function, and five in unexpected additional care or extended stay.” The same 2013 alert estimates that “between 85 and 99 percent of alarm signals do not require clinical intervention”. Drew and colleagues then measured the volume directly. They instrumented 461 adult intensive-care patients and logged 2,558,760 unique alarms in 31 days — 1,154,201 arrhythmia, 612,927 parameter, 791,632 technical. That is an audible burden of 187 per bed per day, reported in PLOS ONE in 2014. Of the 12,671 nurse-annotated arrhythmia alarms, 88.8% were false positives. Of the 168 true ventricular-tachycardia alarms, 93% were not sustained long enough to warrant treatment. That last number is the argument for temporal confirmation. Even the true detections were mostly not events worth waking anyone for.

Users differ in gait, device placement, mobility, and sensor quality. Evaluation must therefore preserve person-level separation and include deployment-like missingness. The size of that requirement has been measured on this very dataset. Saeb and colleagues ran random forests on the same 30-subject activity data — 10,299 records, an average of 343 per subject — and split it two ways. Subject-wise cross-validation error started at 27% with two subjects and fell to about 7%. Record-wise error “already started at a value of 2% when using data from only two subjects, and did not significantly change”. Their systematic review of 62 papers found 28 of them, about 45%, using record-wise cross-validation. Across the papers that reported accuracy, “for subject-wise CV papers, the median classification error (1 - accuracy) was 13.00%, more than twice that of record-wise CV papers, which was 5.60%”. That was 2017. Tougui and colleagues independently reproduced the direction of the effect on a Parkinson's-disease smartphone-audio dataset. A 2% error rate that is really 27% is not a small bookkeeping difference. It is the whole product decision.

There is a published measurement of this exact gap for this exact product. Bagalà and colleagues benchmarked thirteen already-published accelerometer fall-detection algorithms. The test set was 29 real-world falls. They were recorded from high-fall-risk patients in the SensAction-AAL project. They report in PLOS ONE, in 2012: “The SP average of the thirteen algorithms, was (mean +/- std) 83.0%+/- 30.3% (maximum value = 98%). The SE was considerably lower (SE = 57.0%+/- 27.3%, maximum value = 82.8%)”. Those figures are “much lower than the values obtained on simulated falls”. Over one day of monitoring, false alarms ranged from 3 to 85. Every one of those algorithms had been published as accurate, on falls performed deliberately by healthy volunteers.

The design uses shared temporal features with separate heads for activity, risk, and quality. The shared trunk is the cheap part of this drawing. What the heads are allowed to take from it is set by the loss weights, and the sections below show that this is where most of the measured variation lives.

Trace shapes before writing training code. Better still, prefer a shape somebody has already published and measured.

The UCI Human Activity Recognition Using Smartphones dataset was built from 30 volunteers aged 19 to 48, each wearing a waist-mounted Samsung Galaxy S II. It records a triaxial accelerometer and a triaxial gyroscope — six channels — at 50 Hz. “The time signals were then sampled in fixed-width sliding windows of 2.56 sec and 50% overlap between them”, Anguita and colleagues wrote in 2013. At 50 Hz a 2.56-second window is exactly 128 readings. So step 1 of the ledger is not [B, T, C] in the abstract. It is a batch of 128-reading windows over six channels, in a benchmark you can download: 10,299 instances, 561 engineered features, 6 activities.

The same record settles the split as well. The UCI entry states that the partition was made so that “70% of the volunteers was selected for generating the training data and 30% the test data”. The separation is by person, written into the benchmark before any model touches it.

Each head serves a different target, and each may have different label availability. The numbers that combine those losses are not bookkeeping. They are one of the largest levers in the design.

The controlled version of that experiment has been run. One encoder-decoder architecture was held fixed on Tiny CityScapes at 128x256, and only the classification/depth loss weights changed: 1.0/0.0 gave 59.4% IoU, 0.85/0.15 gave 60.4%, an equal 0.5/0.5 gave 56.3%, and 0.1/0.9 collapsed to 42.7%. The architecture was identical in all four runs. Kendall and colleagues open their 2018 paper on the point: “In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss.” Learning the weights from homoscedastic task uncertainty reached 62.7% IoU and 0.533 RMS inverse-depth error — better than the best fixed weighting, 0.573, and better than either single-task model. The tidy round number 0.5/0.5 finished below the learned weighting.

The quality head carries the other kind of risk, and sensing has a regulated precedent for it. Hidden hypoxemia — a normal pulse-oximeter reading masking a low arterial saturation — turned up in 1,785 of 26,032 Black patients, 6.8%, against 2,822 of 57,632 White patients, 4.9%. In-hospital mortality among the affected Black patients was 21.1% against 15.0%. Wong and colleagues found that across 5 databases and 87,971 patients, in JAMA Network Open in 2021. In January 2025 the FDA published draft guidance requiring manufacturers to evaluate pulse-oximeter accuracy across skin pigmentation, using both the Monk Skin Tone Scale and individual typology angle, with a labelling statement when comparable performance is demonstrated. “Our draft recommendations are based on the best available science to help address concerns of disparate performance of pulse oximeters based on an individual's skin pigmentation,” said Michelle Tarver, director of the FDA's Center for Devices and Radiological Health. A quality head fed by a sensor whose error depends on who is wearing it will learn the subgroup, unless the labels and the slices stop it.

Use activation-aware initialization, normalization that remains stable at the intended on-device batch size, and a loss reduction that accounts for valid timesteps and missing head labels.

Monitor per-head loss, gradient norms into the shared encoder, and conflict among head updates. If the safety head is drowned out, change the weighting only after verifying label quality and sampling. Treat that weighting as a measurement rather than a preference, and the reason is the sweep above. One fixed architecture produced 60.4% IoU under 0.85/0.15 and 42.7% under 0.1/0.9. A weighting learned from task uncertainty beat every fixed pair in that sweep, at 62.7%. If a weight pair can move a result that far, it deserves the same logging and the same ablation as a layer.

The capstone is incomplete until it includes tests that could falsify the design. Note what the strongest failure in this lesson was not. In the civil-war case discussed further down, no layer was wrong and no optimizer diverged. The model had simply seen the data it was scored on, and every reported margin was an artifact of that.

Organize failures around the information pathway rather than one aggregate accuracy number.

Population shift is the easiest of these to assert and the hardest to face, so take the measured case. Apple's documentation states plainly that “Apple Watch cannot detect all falls”, and gates the feature accordingly: it turns on automatically only for users aged 55 and over, and is unavailable under 18. One boundary of that sentence has been measured. Twenty-five young adults wore an Apple Watch Series 5 and performed 12 intentional falls each out of a wheelchair. Of the 300 fall trials, the watch detected 14 — “a sensitivity of 4.7%, a false negative rate of 95.3%”, Abou and colleagues reported in 2022. Nothing was broken. The population was outside the one the detector was built and gated for, and the failure was near-total rather than graceful.

Sensor failure has the same structure. The pulse-oximeter discrepancy above was a hardware error rate that depended on who was wearing the device. That is why a quality head needs independent evidence rather than the model's own confidence.

Produce a concise dossier that another engineer can challenge and reproduce. Each of the five items below should carry a number rather than an intention. The input contract is a batch of 128-reading windows over six channels, because 2.56 seconds at 50 Hz is 128 readings. The split is by volunteer, because subject-wise error was 27% where record-wise error was 2%. The loss weighting is a sweep with a reported table, because one weight pair moved a fixed architecture from 42.7% to 62.7% IoU. The verification plan includes a leakage check, because a whole literature's margin over logistic regression did not survive one. And the deployment decision names its abstention delay, because the shipped comparison waits about a minute and then counts down 30 seconds before it calls anyone.