Use direnv to add bin to your path
PATH_add binBring up the environment
# Build the base image
docker compose build
# Install the gems
nabu_run bundle
# Bring up all the containers
docker compose upThis brings up the following containers
- app - the rails app
- search - OpenSearch instance for search (dev + test), pinned to production's version
- db - mysql data base (dev + test)
- s3 - s3 mock
You can then easily run all the standard commands by prefixing with nabu
nabu_run bundle install
nabu_run bundle exec rake db:prepareOpenSearch was previously latest (3.x) locally. A data volume written by 3.x can't be opened by 2.19, so reset it once
and reindex development data. This briefly breaks search specs in every other checkout, so stop other test runs first.
docker compose rm --stop --force search
docker volume rm nabu_search-data
docker compose up --detach search
nabu_run bin/rake searchkick:reindex:allTest indices are recreated by the next bin/test run.
nabu_run bin/test # the whole suite, in parallel
nabu_run bin/test spec/models/item_spec.rb # specific files or examples
nabu_run bin/test --only-failures # what failed last timebin/test prepares the test databases before running RSpec, so a fresh checkout or a branch switch needs no manual step.
With no arguments bin/test runs the suite across parallel workers with parallel_tests, each with its own databases, search indices and bucket
(e.g. nabu_test_2). It uses 4 workers, so that concurrent runs leave each other room; set PARALLEL_TEST_PROCESSORS to change that,
as CI does to use every core. Any argument runs plain RSpec in a single process.
Each linked git worktree gets its own test namespace: its own test databases, search indices and catalogue bucket
on the shared containers, named after the worktree (e.g. nabu_test_<worktree>). The main checkout keeps the plain nabu_test names.
Set NABU_TEST_NAMESPACE to choose a namespace explicitly, or to an empty string for the plain names.
nabu_run bin/cibin/ci runs rubocop, brakeman and bundle-audit, then bin/test if those passed. It reports each step's runtime and exits non-zero if any fails.
The steps live in config/ci.rb. GitHub Actions runs the same checks, with linting, security scanning and the tests as separate jobs.
Namespaces outlive their worktrees. Run bin/test_prune on the host to list the test databases, search indices and catalogue buckets
that belong to no current worktree, then bin/test_prune --delete to drop them. The main checkout's names are never touched.
A namespace set by hand with NABU_TEST_NAMESPACE counts as orphaned unless it matches a current worktree's name.
The application is designed to be deployed with containers into an AWS account using CDK
To bootstrap a new account
# Setup an AWS account and credentials as per your preferred method and set the environment to use it
AWS_PROFILE=nabu
REGION=ap-southeast-2
ACCOUNT=$(aws sts get-caller-identity | jq -r .Account)
cdk bootstrap aws://$ACCOUNT/$REGIONIf ECR complains about access
ACCOUNT=$(AWS_PROFILE=nabu-stage aws sts get-caller-identity | jq -r .Account)
AWS_PROFILE=nabu-stage aws ecr get-login-password --region ap-southeast-2 | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.ap-southeast-2.amazonaws.comUse CDK to deploy new code via docker as well as any infrastructure changes
bin/release stage
bin/release prodIf necessary:
bin/aws/ecs_rake app db:migrate
bin/aws/ecs_rake app search:reindexbin/aws/db_sync dumps the production database to S3, restores it into your
local nabu_devel, and resets every user's password to password. See
docs/runbooks/db-sync.md for the full design (it can also overwrite staging).
bin/aws/db_sync # choose target 1) dev
nabu_run bin/rails db:environment:set RAILS_ENV=development
nabu_run bin/rake db:migrate
nabu_run bin/rake searchkick:reindex:allEvery Language is brought into line with its Source by the Language Refresh, which runs monthly on the first Tuesday and emails a report. It reads ISO 639-3 from SIL, Glottolog from its latest release and AUSTLANG from AIATSIS, regenerates the advisory Equivalents, fills empty Bounding boxes and lists what needs a person.
To run one by hand:
nabu_run bin/rake languages:refreshOLAC available at:
The feeds that OLAC harvests:
- http://catalog.paradisec.org.au/oai/item?verb=ListRecords&metadataPrefix=olac
- http://catalog.paradisec.org.au/oai/item?verb=Identify (Archive identification)
- http://catalog.paradisec.org.au/oai/item?verb=ListMetadataFormats
- http://catalog.paradisec.org.au/oai/item?verb=ListIdentifiers&metadataPrefix=olac
Individual item:
RIF-CS available at:
-
http://catalog.paradisec.org.au/oai/collection
use resulting server on an OAI repository explorer:
-
http://www.language-archives.org/register/register.php (OLAC)
-
http://oval.base-search.net/ (OAI-PMH validator)
-
http://validator.oaipmh.com/ (OAI-PMH validator)
-
http://repox.gulbenkian.pt/repox/jsp/testOAI-PMH.jsp (test protocol)
URLs to test:
-
[http://localhost:3000/oai/collection?verb=Identify
-
[http://localhost:3000/oai/collection?verb=ListMetadataFormats
-
[http://localhost:3000/oai/collection?verb=ListSets
-
[http://localhost:3000/oai/collection?verb=ListIdentifiers
The feed that ANDS harvests:
Test at ANDS:
Feed for a single collection:
To validate our XML output as per OLAC
- Download https://xerces.apache.org/mirrors.cgi#binary
- Extract it
java -cp xercesImpl.jar:xercesSamples.jar sax.Counter -n -np -v -s -f item.xmlaws secretsmanager list-secrets
aws secretsmanager put-secret-value --secret-id ARN --secret-string "{\"site_key\":\"***\", \"secret_key\":\"***\"}"We should regularly make sure we are running the latest versions of third-party packages
# Ruby gems
nabu_run bundle outdated
nabu_run bundle update
# node modules
nabu_run pnpm up -i
# New rails version
rails new nabu --database=mysql --javascript=esbuild --css=sass --skip-action-cable --skip-kamal