Repository navigation
Conversation
…anded notices Model 2.7 scored 83% on a 100-message bench where TypeSafe (jev-1.13.0, reading docs/LABELING_GUIDE.md) scored 96-98%. Every miss was a data gap. 2.9 trains on the teacher's verdicts for the whole 137k corpus with soft targets, plus generated data for what the corpus lacked, and matches the teacher on the bench. Data (evals/distill_labels.py): TypeSafe answered the guide's label question for 154k unique texts (cached in dataset/curated/typesafe_corpus_answers.jsonl). Eval texts are excluded by exact match, template shape and token Jaccard; 5% of the corpus is held out by hash. Rows below 0.6 teacher confidence are dropped. Generated sets, each kept only where the teacher agrees: - evals/generate_archetypes.py: new-number family scams, code-relay asks, wrong-number openers, fake orders with a callback number, benign twins. - evals/generate_legit_notices.py: 12k legitimate bank, telco, courier, retail, government, health and travel notices in 14 languages with real domains. The labeled corpus had 40 phishing rows with a link per ham row with a link, so the first candidate blocked every real bill and delivery notice. --twins-only emits the minimal-pair lure for each notice, including link-stripped and KYC variants. Training (evals/finetune_distill.py): 2.5 base, (1-a) CE + a KL(teacher), a=0.5, lr 2e-5, 2 epochs, batch 32, padding bucketed to multiples of 16 for MPS. Three seeds, 66-68 min each on an M4 Pro; shipped checkpoint is seed 13. Results (evals/results/DISTILL_2.9.md, three-seed mean vs 2.7): - bench100 accuracy 83% -> 96.7%, false blocks 17.6% -> 1.0% - hard_legit false blocks 32.5% -> 1.7% - teacher held-out accuracy 80.1% -> 98.8%, Mishra 89.4% -> 94.7% - errors no longer at 0.99 confidence; calibration finds nothing to fix UCI 3-class accuracy drops by design (prize lures are phishing under the guide). Serving: settings.py and .dockerignore point at mbert_ots_model_2.9.pth. The image passes the API unit tests, the bench through /predict, and both SMPP integration suites. run_eval.py, calibrate_thresholds.py and the new trainer run on the GPU; run_eval keys generic CSV sets by filename so several can share a run. label_audit no longer treats the held-out prediction dump as an eval set. test_advanced.js uses a realistic message for the DLR batch test; its old synthetic string scores as spam at the proxy threshold under 2.9.
…builds The live unit on ots.telecomsxchange.com pins an image tag rather than following latest, and the instance is linux/amd64. An image built with plain docker build on Apple Silicon is arm64-only and fails there with exec format error, so the update steps now try the new tag on a side port first and the README says to build with buildx for both platforms.
…age claim Clicking the status pill used to open the raw /health JSON. It now opens a small panel with the state, API version, model, architecture, response time and last check in plain rows, with links to the changelog and the raw endpoint. Closes on Escape or a click outside; keyboard and screen-reader friendly. New changelog.html summarises what each model version brought (2.9, 2.7, 2.5 and the 2024 releases) with the numbers behind it, in the demo page's design and theme. Linked from the footer and the status panel. The page and README said "more than 100 languages". That is the mBERT base; a language-detector pass over the 2.9 training set finds about 20 languages, with nine carrying nearly all of it. Both now say: built on a 104-language model and trained on messages in 20 languages.
…ase workflows evals/probe/probe.py invents 150 realistic SMS a day with Claude Haiku 4.5 (rotating sectors, languages and styles, each attack with a benign twin, de-duplicated across days), classifies them through the live API, and has Claude Opus 5 grade every answer against docs/LABELING_GUIDE.md. It writes a JSONL log per day, an improvements log with one line per miss, a training_additions.csv of graded misses with their gold labels for the next fine-tune, and REPORT.md with accuracy, false-block and missed-threat rates by day, category and language. Runs on the public host from a systemd timer; sync.sh pulls the logs back into the repo. .github/workflows/model-gate.yml scores the checkpoint settings.py points at on bench100, hard_legit and the clean fable5 split on every PR that touches a checkpoint, the model settings or the eval tooling, and fails below evals/gate_thresholds.json (evals/gate.py). The checkpoint is cached by its LFS object id so LFS bandwidth is spent once per checkpoint. .github/workflows/release-image.yml builds the image for linux/amd64 and linux/arm64 on a v* tag or by hand, pushes it, and fails unless the manifest lists both platforms. Needs DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+17
to
+55
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| lfs: false | ||
|
|
||
| - name: Find the default checkpoint | ||
| id: ckpt | ||
| run: | | ||
| p=$(grep -o 'mBERT/training/model-training/mbert_ots_model_[0-9.]*\.pth' src/api_interface/config/settings.py | head -1) | ||
| echo "path=src/$p" >> "$GITHUB_OUTPUT" | ||
| echo "oid=$(git show HEAD:src/$p | grep oid | cut -d: -f2)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Cache the checkpoint and vocab | ||
| id: cache | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: | | ||
| ${{ steps.ckpt.outputs.path }} | ||
| evals/assets/bert-base-multilingual-cased-vocab.txt | ||
| key: lfs-${{ steps.ckpt.outputs.oid }} | ||
|
|
||
| - name: Pull the checkpoint from LFS | ||
| if: steps.cache.outputs.cache-hit != 'true' | ||
| run: git lfs pull --include "${{ steps.ckpt.outputs.path }},evals/assets/bert-base-multilingual-cased-vocab.txt" | ||
|
|
||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install CPU torch and transformers | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install torch --index-url https://download.pytorch.org/whl/cpu | ||
| pip install transformers | ||
|
|
||
| - name: Score and gate | ||
| run: python evals/gate.py --model "${{ steps.ckpt.outputs.path }}" |
Comment on lines
+22
to
+58
| runs-on: ubuntu-latest | ||
| timeout-minutes: 90 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| lfs: true | ||
|
|
||
| - name: Work out the tags | ||
| id: tags | ||
| run: | | ||
| if [ "${{ github.event_name }}" = "push" ]; then tag="${GITHUB_REF_NAME#v}"; latest=true; else tag="${{ inputs.tag }}"; latest="${{ inputs.latest }}"; fi | ||
| t="telecomsxchange/opentextshield:$tag" | ||
| [ "$latest" = "true" ] && t="$t,telecomsxchange/opentextshield:latest" | ||
| echo "tag=$tag" >> "$GITHUB_OUTPUT"; echo "list=$t" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - uses: docker/setup-qemu-action@v3 | ||
| - uses: docker/setup-buildx-action@v3 | ||
| - uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
|
||
| - uses: docker/build-push-action@v6 | ||
| with: | ||
| context: . | ||
| platforms: linux/amd64,linux/arm64 | ||
| push: true | ||
| tags: ${{ steps.tags.outputs.list }} | ||
| cache-from: type=gha | ||
| cache-to: type=gha,mode=max | ||
|
|
||
| - name: Verify both platforms are in the manifest | ||
| run: | | ||
| m=$(docker buildx imagetools inspect "telecomsxchange/opentextshield:${{ steps.tags.outputs.tag }}") | ||
| echo "$m" | grep -q "linux/amd64" || { echo "amd64 missing"; exit 1; } | ||
| echo "$m" | grep -q "linux/arm64" || { echo "arm64 missing"; exit 1; } | ||
| echo "manifest ok: amd64 + arm64" |
…pported temperature argument
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Model 2.7 scored 83% on a 100-message hand-written bench where TypeSafe (jev-1.13.0, reading
docs/LABELING_GUIDE.md) scored 96–98%. Every miss was a data gap: prize and job lures labeled spam, real branded notices blocked, conversational social engineering passed. 2.9 trains on TypeSafe's verdicts for the whole 137k corpus with soft targets, plus generated data for what the corpus lacked, and matches the teacher on the bench.Full write-up:
evals/results/DISTILL_2.9.md.Results (three seeds, mean; 2.7 scored by the same harness)
UCI and Mishra 3-class accuracy shift by design: prize lures are phishing under the guide, and some 2004-era ringtone receipts labeled spam in those sets now pass. Errors are no longer made at 0.99 confidence; calibration finds nothing to fix and the SMPP threshold stays at 0.7.
What changed
evals/distill_labels.py: label the corpus with TypeSafe (cached, resumable), exclude eval texts, build the distillation set with soft targets.evals/generate_archetypes.py,evals/generate_legit_notices.py: the missing archetypes and 12k legitimate branded notices in 14 languages with lure twins. The labeled corpus had 40 phishing rows with a link per ham row with a link; that prior was the whole false-block problem.evals/finetune_distill.py: KL-distillation trainer on the GPU, padding bucketed for MPS.evals/score_candidate.sh,evals/train_distill_seeds.sh,evals/distill_report.py,evals/bench_typesafe.py: multi-seed training, scoring and reporting.run_eval.pyandcalibrate_thresholds.pyrun on the GPU; generic CSV sets are keyed by filename.settings.py,.dockerignore: default checkpoint ismbert_ots_model_2.9.pth(LFS).test_advanced.js: realistic text for the DLR batch test; the old synthetic string scores as spam at the threshold under 2.9.Verified locally
/metricsrespond, both SMPP integration suites pass against the container.Deploy
Push
telecomsxchange/opentextshield:2.9and retaglatest; the public host pullslatest. Roll out to one instance first and read a day of the audit log. Rollback isOTS_MBERT_MODEL_PATHpointing at the 2.7 checkpoint.