Перейти к содержанию

Работа без лаборатории · Средний

Report computational, machine learning and engineering research

Design fair computational and engineering evaluations with reproducible methods.

13 мин чтения · 26 окт. 2026 г.

Эта страница ещё не переведена, поэтому вы читаете английскую версию.

Главное

  • Prevent information leakage.
  • Use meaningful baselines and metrics.
  • Separate demonstration from external validation.
Содержание

Design fair computational and engineering evaluations with reproducible methods.

A computational or engineering paper must explain what was built, what question it addresses, how it was evaluated and where its evidence stops. An impressive demonstration is not automatically a fair comparison. A high prediction score can reflect information leakage. A precise sensor reading can conceal calibration uncertainty. A useful paper exposes these possibilities so readers can judge the contribution.

The examples in this tutorial are synthetic or hypothetical teaching cases. Their numbers illustrate reporting and calculation, not the performance of an actual device or model. The principles also apply to computational science and social science projects using prediction, simulation or large datasets.

Define the task, target and contribution

Write the input, output, intended setting and success criteria. “We built a better classifier” is incomplete. A clearer question is: “How accurately can a model detect a specified machine fault in recordings from devices absent from model development?” This identifies the positive class, the data source and the intended generalisation target.

Separate a contribution in methodology from a contribution in performance or application. A simpler algorithm may be valuable even if its accuracy is similar, provided its cost or interpretability is evaluated. A new dataset may be valuable because it covers a missing condition, even before a model achieves a record score. A prototype may demonstrate feasibility while leaving durability unresolved. State which contribution the evidence supports.

Define the benchmark, meaning the evaluation task and conditions used to compare methods. Specify what makes a case independent. Multiple recordings from one device do not necessarily provide independent evidence about performance on new devices. Similarly, thousands of simulated time steps from one trajectory do not equal thousands of independent experiments.

Describe the system so readers can reconstruct the comparison

For a physical device, report its relevant dimensions, materials, components, assembly, firmware, operating conditions and measurement procedure. Use units consistently. State tolerances when manufacturing variation could affect performance. A photograph can orient the reader, but it cannot replace dimensions, circuit details or a specification needed to understand the mechanism.

For a computational method, define variables, objective functions, assumptions, algorithm steps and stopping criteria. Distinguish parameters learned from data from hyperparameters, which govern model structure or fitting and are selected through a development procedure. Explain defaults that materially affect the result. “Standard settings were used” becomes ambiguous when software versions change.

For simulations, describe governing equations, boundary and initial conditions, parameter ranges, numerical solver and convergence criteria. Explain how scenarios were chosen. A simulation containing only easy cases may estimate performance accurately for those cases while failing to address the intended operating conditions.

Distinguish verification from validation

Verification asks whether the implementation or product meets its specified requirements. Validation asks whether it is suitable for its intended use under relevant conditions. NASA's systems engineering guidance makes this distinction between conformity to requirements and suitability in realistic operation. Neither word should be used merely to mean that a graph looked plausible. S509

For a numerical solver, verification might compare a simple case with a known analytical solution and examine whether estimates stabilise as the time step or mesh is refined. Validation might compare predictions with independent measurements. Agreement after parameters were fitted to those same measurements is calibration evidence; it is weaker evidence of prediction in new conditions.

A fictional heat transfer paper could report that the solver reproduced an analytical slab solution within a defined tolerance, then compare predictions against separate measurements from an instrumented sample. If the measurements cover only one temperature range, the validation claim should retain that boundary. Do not claim validation of a complete physical process from a code check alone.

Protect evaluation data from learning decisions

Information leakage occurs when information unavailable at the intended prediction moment influences model development or evaluation. It can arise through duplicate records, future observations, preprocessing fitted on all data, feature selection using test labels or repeated tuning against the final test score.

Fit scaling, imputation, feature selection and dimensionality reduction within the training portion of each evaluation split. Apply the fitted transformations to the evaluation portion. A software pipeline can help preserve this separation, but it must still be used with the correct splitting strategy. The scikit learn documentation demonstrates how preprocessing before splitting can produce misleading performance. S23

Suppose a hypothetical dataset contains ten recordings from each machine. A random split of recordings can place the same machine in training and testing. The model may recognise machine specific signals rather than generalise to unseen machines. If the scientific question concerns new machines, group by machine identifier. If the intended use predicts future events, a split respecting time may be necessary. The correct unit follows the claim, not the easiest command.

Training data support fitting and tuning. A separate evaluation set is used after decisions are fixed. Group identifiers remain within one split.

Cross validation repeatedly separates development data into fitting and validation subsets. Nested cross validation places hyperparameter selection inside an inner loop and evaluates the selected procedure in an outer loop. It can help avoid evaluating a model on the same folds used to select it. Grouped, stratified and temporal strategies serve different purposes; preserving class balance does not resolve dependence between observations. S523

Keep a final test set untouched until the planned decisions are fixed when the design provides one. If repeated inspection of that set changes the method, it has become part of development. Report this honestly and obtain new evaluation data when needed rather than calling the same set unseen.

Compare against credible baselines

A baseline is a reference method that makes the contribution interpretable. Include a simple baseline and relevant established approaches. A fault classifier should at least be compared with a rule reflecting class prevalence or a simple model; a controller comparison needs consistent environments and comparable tuning effort. Explain the resources and information available to each method.

An ablation study removes or modifies a component to investigate its contribution. If removing a component also changes model capacity, training time and parameter count, the comparison does not isolate a single mechanism. State the actual difference. Avoid calling every performance change proof that a particular internal explanation is correct.

Report tradeoffs. Lower error may come with longer latency, greater energy use or more expensive sensing. Describe the units and evaluation conditions for each outcome, and identify primary outcomes before inspecting the results. A method that wins one metric while losing another requires a decision relevant interpretation, not the word “superior” without qualification.

Choose metrics from the consequences of errors

For binary classification, define the positive class and report the confusion matrix. Sensitivity, also called recall, is the proportion of actual positives detected. Specificity is the proportion of actual negatives correctly rejected. Precision, or positive predictive value, is the proportion of predicted positives that are actual positives. Accuracy is the proportion of all cases classified correctly. These answer different questions. Software metric documentation provides the corresponding definitions and averaging conventions. S511

Consider this synthetic evaluation of 1,000 independent cases, with 50 actual faults.

Actual class Predicted fault Predicted no fault Total
Fault 40 10 50
No fault 95 855 950
Total 135 865 1000

Sensitivity is 40/50 = 80%. Specificity is 855/950 = 90%. Precision is 40/135, approximately 29.6%. Accuracy is 895/1000 = 89.5%. A rule that never predicts a fault achieves 95% accuracy here but detects none of the faults. Accuracy alone therefore gives the wrong impression of the task's useful performance.

A useful result paragraph would read: “In the synthetic evaluation, the classifier detected 40 of 50 faults but generated 95 false alarms among 950 cases without faults. Precision was 29.6%, indicating that most alarms were false under the evaluated class prevalence. The practical value would depend on the relative consequences of missed faults and unnecessary inspection.” A real paper should also report suitable uncertainty and describe threshold selection.

The decision threshold converts a score into a class label. Choose it using development data and application requirements. Do not search the final test set for the threshold producing the most attractive result. Precision changes when prevalence changes, even when sensitivity and specificity remain similar. Performance measured in an artificially balanced dataset may therefore not describe operational alarm burden.

Separate discrimination, calibration and numerical error

Discrimination concerns separating or ranking cases. Probability calibration concerns whether predicted probabilities agree with observed frequencies. If a model repeatedly assigns probabilities around 0.8, calibration asks whether roughly that proportion of comparable cases has the event. A model can rank cases well while assigning probabilities that are too extreme. Calibration curves and appropriate scoring rules provide complementary information, and recalibration itself must use data separate from final evaluation. S512

For a continuous prediction, define the error metric and units. Mean absolute error averages absolute deviations; root mean squared error takes the square root of the average squared deviation and gives more influence to large errors. Neither should be reported without identifying the reference values, evaluation cases and aggregation. A mean across subjects can differ from a mean across all observations when subjects contribute unequal numbers of records.

For physical measurement, distinguish repeatability from accuracy and uncertainty. Repeated readings may vary little while all share a calibration offset. Measurement uncertainty describes uncertainty about the quantity being measured, not simply variation among repeated readings. NIST Technical Note 1297 distinguishes components evaluated statistically from those evaluated using other information; both can contribute to a combined uncertainty. S510

In a synthetic linear example, two resistance estimates are 100 Ω and 200 Ω with independent standard uncertainties 0.2 Ω and 0.3 Ω. For their sum, the estimate is 300 Ω and the combined standard uncertainty is sqrt(0.2² + 0.3²), about 0.36 Ω. A coverage factor of 2 gives an expanded uncertainty of about 0.72 Ω. Report the factor and assumptions; a factor of 2 is not automatically an exact 95% interval. Correlation between components would change this calculation.

Describe uncertainty at the correct level

Different repetitions answer different questions. Repeating training with several seeds examines some algorithmic variation. Repeating measurements on one specimen examines measurement variation. Evaluating independent devices or sites examines variation at those levels. None automatically substitutes for the others.

Report the number of independent evaluation units and how repeated observations were handled. If uncertainty is estimated by resampling, resample at a level consistent with the design. A large number of correlated rows can otherwise produce an implausibly narrow interval. Cross validation folds share training data, so a standard deviation across fold scores should not automatically be treated as the standard error of an independent sample of experiments.

The supplied controller example in Article 32 uses twelve constructed paired layout summaries. A real version would also need the number of runs within each layout, a rule for producing each summary and a justification for treating layouts as independent. It should report failures under the defined inclusion rules rather than omit difficult runs to improve the mean.

Make the computational record usable

Preserve data provenance, licences, version identifiers, preprocessing code, split assignments, environment versions, model configuration and evaluation outputs. Document random seeds and hardware when relevant. A seed alone may not ensure identical output across hardware or library versions. Explain expected variation and any nondeterministic operations.

A reproducibility README should tell another person how to obtain permitted inputs, install dependencies, run the analysis and locate outputs. Provide a small demonstration when the full calculation requires substantial resources. Separate reusable code from credentials, identifying information and restricted data. Where sharing is limited, describe the actual access process and supply permitted documentation rather than promising unrestricted access.

Sandve and colleagues' guidance on reproducible computational research highlights preserving the connection between inputs, code and outputs, avoiding undocumented manual changes, and recording the computational environment. The goal is a checkable chain of work, not merely a repository containing assorted scripts. S524

Write the paper around the evaluation question

The Introduction explains the problem, limitations of relevant approaches and specific contribution. The Methods describes the system, data, evaluation units, comparators, splitting, metrics and uncertainty. The Results reports all planned comparisons, failures and tradeoffs. The Discussion considers distribution shift, untested environments, fairness where relevant, and practical constraints.

Distribution shift means that the evaluation or deployment data differ from the development data in a relevant way. A model evaluated on one institution's records has not automatically been evaluated elsewhere. A sensor validated indoors has not automatically been validated outdoors. Name the next evaluation needed to support the broader claim.

A defensible conclusion is: “The prototype reduced the specified error in the evaluated conditions, while power consumption increased. Additional testing under changing environmental conditions is needed to determine whether that tradeoff persists.” Avoid “The system is reliable and ready for use” when durability, failure modes or external conditions were not evaluated.

Practice and model answer

Task: A student scales all observations, selects features using all labels, splits records randomly, tunes repeatedly on the test score and reports only the best random seed. The records contain repeated measurements from the same devices. What must change?

Model answer: Define whether the target is new measurements from known devices or new devices. Split using that target, keeping related observations together where required. Fit preprocessing and feature selection within training data, and tune through an appropriate validation procedure. Reserve independent evaluation for fixed decisions. Report the planned repetition procedure and its distribution of results rather than selecting the most favourable seed. Reevaluate after these changes because the original score does not estimate the intended generalisation performance.

Trace your own pipeline from raw input to final figure and mark every step that learns a parameter or selects an option. Use Template 4 and Template 6 to document the evaluation and reproduction plan.

Sources

S23: Scikit learn developers. Common pitfalls and recommended practices. Current official documentation, accessed 11 November 2026.

S509: NASA. Systems Engineering Handbook, section 5.3: Product Verification. Official guidance, accessed 11 November 2026.

S523: Scikit learn developers. Cross validation: evaluating estimator performance. Current official documentation, accessed 11 November 2026.

S511: Scikit learn developers. Metrics and scoring: quantifying the quality of predictions. Current official documentation, accessed 11 November 2026.

S512: Scikit learn developers. Probability calibration. Current official documentation, accessed 11 November 2026.

S510: Taylor BN, Kuyatt CE. 1994. Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. NIST Technical Note 1297.

S524: Sandve GK, Nekrutenko A, Taylor J, Hovig E. 2013. Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology 9(10):e1003285.

Файлы

Войдите, чтобы скачать эти файлы.