Machine learning vs rules-based software: which should each feature use?
Any feature can be written as explicit rules or trained as a model, and most products need both. The choice is made per feature, not per product, and it changes how the feature is tested, how it is changed and who can explain a wrong answer.
Should a feature use machine learning or rules-based software?
Quick answer
Machine learning vs rules-based software is a per-feature choice. Use explicit rules when a domain expert can write the logic and every output must be explained. Train a model when the pattern can only be learned from labeled examples and keeps shifting. Agnotic Technologies builds both, often with rules guarding the model.
| Criterion | Rules | Machine learning |
|---|---|---|
| Can a domain expert write the rule? | Yes: the logic fits in a decision table or a set of conditions an expert can review | No: experts recognise the right answer but cannot state the rule, as with defect photos or free-text triage |
| Is there labeled data? | Not needed; the rules are the specification | Required: labeled examples of each outcome, from the population the feature will serve |
| Must each output be explained? | Each decision traces to the rule that fired | Explanations are approximate; plan for attributions or a human reviewer |
| How often does the pattern change? | Rarely, or on a known schedule such as a new policy year | Continuously, as with fraud patterns that adapt to your controls |
| What does a wrong answer cost? | A wrong rule fails the same way every time and is fixed once | Errors vary case by case; set a threshold and send low-confidence outputs to a person |
| How is it tested? | Unit tests per rule, with fixed inputs and exact expected outputs | A held-out evaluation set, scored against thresholds agreed before training |
| How is it changed? | Edit the rule, review the diff, rerun the tests, release | Retrain, re-evaluate on the same set, compare with the live model, release behind a flag |
| What is monitored in production? | Rule hit counts and inputs that match no rule | Input drift, output distribution and accuracy on newly labeled samples |
What they are
What is Rules
Rules-based software runs logic that people wrote down: conditions, thresholds, lookup tables and workflow steps. Its behaviour stays fixed until someone edits the code or the rule configuration, so the same input always gives the same output.
What is Machine learning
Machine learning software learns a mapping from examples. A model is trained on labeled data, evaluated on data it has not seen and then predicts outputs for new inputs. Its behaviour depends on the training data as much as on the code.
Key differences
Why does testing look different for each?
A rule is tested like any other code: fixed inputs, exact expected outputs, and a failing test blocks the release. A model is tested statistically. You hold out an evaluation set the model never trained on, agree thresholds per outcome before training, and release only when the new model meets them. NIST's AI Risk Management Framework names this gap, listing the difficulty of deciding what to test as one way AI risk differs from traditional software risk.
What does change control involve?
Changing a rule means editing it, reviewing the diff and rerunning the suite. Changing a model means retraining on new data, which can shift outputs you did not mean to touch. Keep the training data version, code version and evaluation results together for every model release, so you can say why an output changed between two dates. NIST also notes that AI systems may need more frequent maintenance because of data, model or concept drift.
How do regulated products handle model updates?
For medical devices, the FD&C Act lets FDA approve or clear a predetermined change control plan (21 U.S.C. 360e-4). Changes that match the plan's described modifications, methods and performance requirements then do not need a new supplement or 510(k) each time. If a feature might ever be a device function, design the retraining pipeline so each step could be written into such a plan.
When is a hybrid the better design?
Often. Two patterns cover most cases. Rules first, model second: hard constraints and known cases go through rules, and only the remainder reaches the model. Rules as guardrails: the model proposes, and rules reject outputs that break a constraint, such as a triage suggestion that skips a mandatory escalation or a schedule that double-books a room. The rules stay testable, and the model does the part rules cannot.
When to use each
When to use Rules-based software
- Scheduling constraints, such as which rooms, staff and equipment a booking needs.
- Eligibility, pricing and policy checks where the rule text already exists.
- Any output an auditor or regulator will ask you to justify line by line.
- Features with little or no labeled history to learn from.
When to use Machine learning
- Visual inspection, where a model flags defects in photos that no rule could describe.
- Fraud or anomaly flags, where patterns move faster than people can rewrite rules.
- Triage of free-text messages or notes into the right queue.
- Ranking and forecasting, where a calibrated score works better than a fixed cutoff.
How do you move a feature from rules to a model?
1. Log first. Run the rules in production and record inputs, outputs and every case a person overrode. Those overrides become your first labeled examples.
2. Build the evaluation set before the model. Freeze a sample of real cases with agreed correct answers, and write down the thresholds a model must meet.
3. Run the model in shadow mode. It scores live traffic next to the rules, and nobody acts on its output until it beats the rules on the evaluation set.
4. Keep the rules as guardrails. Release the model behind the rules that encode hard constraints, and route low-confidence outputs to a person.
5. Retrain on a trigger. Watch input drift and accuracy on newly labeled samples, and retrain when they move, not only on a calendar.