feat(lapis): add sequence position fields syntax to aggregated endpoint by fhennig · Pull Request #1768 · GenSpectrum/LAPIS

fhennig · 2026-07-14T15:21:12Z

Summary

Adds Name[N] syntax (e.g. S[501]) to the fields parameter of /aggregated, allowing users to request specific sequence positions as group-by columns
Shorthand [N] is supported for single-segmented genomes and resolves to the single nucleotide sequence
Sequence names are validated case-insensitively against both genes and nucleotide segments from the reference genome
Generates a .map({"S[501]":="S".at(501)}) SaneQL step before .groupBy(...) when position fields are present — the user-facing name is used directly as the SaneQL alias, so no post-processing is needed
Response column keys mirror the input exactly: S[501] in → S[501] out, [501] in → [501] out

Example

POST /sample/aggregated
{"fields": ["country", "S[501]"]}

Generates:

default.filter(true).map({"S[501]":="S".at(501)}).groupBy({"count":=count()}, {"country", "S[501]"})

Test plan

Unit tests for Name[N] parsing in SequenceFiltersRequestWithFieldsTest
Error cases: unknown sequence name, position ≤ 0, shorthand on multi-segmented genome
SaneQL generation tests in SiloQueryToSaneQlTest for position-only and mixed fields
Full test suite passes

🤖 Generated with Claude Code

Allow users to request specific sequence positions as group-by fields using `Name[N]` syntax (e.g. `S[501]`). For genomes with a sequence named `main`, the shorthand `[N]` is also accepted. Position fields generate a `.map(...)` step in SaneQL before `.groupBy(...)`, and appear in the response under the alias `Name_N` (e.g. `S_501`). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel · 2026-07-14T15:21:18Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
lapis	Ready	Preview, Comment	Jul 15, 2026 12:24pm

Echo S[501] back as the response column key instead of the internal SaneQL alias S_501, so input and output are symmetric for callers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ack in response Shorthand [N] now validates via isSingleSegmented() (consistent with how the rest of the codebase handles single-segment-only features) rather than checking for a hardcoded 'main' name. SequencePositionField gains isSingleSegment flag so the response key mirrors the input: [501] in → [501] out, S[501] in → S[501] out. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…field aliasing SaneQlAssignment now quotes its name (consistent with SaneQlIdentifier, safer against injection). Position fields use their user-facing name (e.g. S[501]) directly as the SaneQL alias, so SILO returns the column under that name already — no post-processing remapping needed in the controller. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Adds support for requesting specific sequence positions (e.g. S[501], and [501] for single-segment genomes) as group-by columns in the /aggregated endpoint by parsing them from the fields request parameter and generating the appropriate SaneQL .map(...).groupBy(...) pipeline.

Changes:

Introduces RequestField with a new SequencePositionField type and updates request parsing to recognize Name[N] / [N] syntax.
Extends aggregated SaneQL generation to add a .map(...) step that computes requested positions and includes them in .groupBy(...).
Updates SaneQL AST rendering/tests to quote assignment keys (e.g. {"count":=count()}) and adds unit coverage for new parsing/generation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lapis/src/main/kotlin/org/genspectrum/lapis/request/Field.kt	Adds `RequestField` + `SequencePositionField`, and extends field conversion to parse/validate position syntax against the reference genome schema.
lapis/src/main/kotlin/org/genspectrum/lapis/request/SequenceFiltersRequestWithFields.kt	Updates request model to store `fields` as `List<RequestField>` and keeps deserialization via the field converter.
lapis/src/main/kotlin/org/genspectrum/lapis/silo/SiloQuery.kt	Adds `sequencePositionFields` to aggregated actions and emits `.map(...)` before `.groupBy(...)` when present.
lapis/src/main/kotlin/org/genspectrum/lapis/silo/SaneQlAst.kt	Changes assignment rendering to quote/escape keys (e.g. `"name":=value`) to align with SaneQL expectations and harden output.
lapis/src/main/kotlin/org/genspectrum/lapis/model/SiloQueryModel.kt	Splits `RequestField` values into metadata fields vs. sequence-position fields for aggregated query construction.
lapis/src/main/kotlin/org/genspectrum/lapis/controller/LapisController.kt	Ensures response headers/field lists use `outputColumnName` (aggregated) and filters non-metadata fields out of details fields handling.
lapis/src/main/kotlin/org/genspectrum/lapis/controller/ControllerDescriptions.kt	Updates aggregated endpoint parameter description to document the new position-field syntax.
lapis/src/main/kotlin/org/genspectrum/lapis/request/PhyloTreeSequenceFiltersRequest.kt	Adjusts phylo-tree parsing to work with the generalized `FieldConverter<RequestField>` while enforcing a `Field` result.
lapis/src/test/kotlin/org/genspectrum/lapis/request/SequenceFiltersRequestWithFieldsTest.kt	Adds unit tests for parsing valid/invalid `Name[N]` / `[N]` field forms.
lapis/src/test/kotlin/org/genspectrum/lapis/silo/SiloQueryToSaneQlTest.kt	Updates expected SaneQL strings and adds coverage for aggregated SaneQL with sequence position fields.
lapis/src/test/kotlin/org/genspectrum/lapis/silo/SaneQlAstTest.kt	Updates AST rendering assertions to match quoted assignment output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fhennig · 2026-07-15T08:45:55Z

~~Ok, something weird with the test? But still reviewable I think. I'll look into it.~~

…ns.kt The AGGREGATED_GROUP_BY_FIELDS_DESCRIPTION const was missing its closing triple-quote, which caused the compiler to swallow the following declaration as part of the string and fail with unresolved reference and syntax errors.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Previously SequencePositionField entries were silently dropped, causing /details to return a 200 with all metadata fields instead of erroring. Now it returns a 400 with a clear message, per Copilot review feedback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

lapis/src/main/kotlin/org/genspectrum/lapis/request/Field.kt:35

FieldConverter.validatePhyloTreeFields(...) is currently unused (no call sites in the repo) and duplicates the already-used validatePhyloTreeField(...) function below. Keeping two parallel validation APIs (one DB-config based, one converter based) risks divergence and makes it unclear which one should be used.

fun interface FieldConverter<T> {
    fun convert(source: String): T

    fun validatePhyloTreeFields(source: String): T = convert(source)
}

lapis/src/main/kotlin/org/genspectrum/lapis/request/Field.kt:98

With FieldConverter.validatePhyloTreeFields(...) removed, this override becomes dead code as well. All phylo-tree validation currently goes through validatePhyloTreeField(...) (used by controllers and request deserializers).

    override fun validatePhyloTreeFields(source: String): RequestField {
        val converted = convert(source)
        if (converted !is Field) {
            throw BadRequestException(
                "Position fields like '$source' cannot be used as phylo tree fields",
            )
        }
        val validFields = caseInsensitiveFieldsCleaner.getPhyloTreeFields()
        if (converted.fieldName !in validFields) {
            throw BadRequestException(
                "Field '${converted.fieldName}' is not a phylo tree field, " +
                    "known phylo tree fields are [${validFields.joinToString(", ")}]",
            )
        }
        return converted
    }

+Sequence positions can be requested using the syntax `SequenceName[position]` (e.g. `S[501]` for position 501 of sequence `S`).
+For single-segmented genomes, the shorthand `[position]` (e.g. `[501]`) can be used.
+Position field column names in the response use the canonical sequence name from the reference genome (case-insensitive input), e.g. `s[501]` -> `S[501]`."""


fengelniederhammer

I'm also missing an e2e test ;)

fengelniederhammer · 2026-07-16T06:55:03Z

+Sequence positions can be requested using the syntax `SequenceName[position]` (e.g. `S[501]` for position 501 of sequence `S`).
+For single-segmented genomes, the shorthand `[position]` (e.g. `[501]`) can be used.
+Position field column names in the response use the canonical sequence name from the reference genome (case-insensitive input), e.g. `s[501]` -> `S[501]`."""


We should probably also add something in the llms.txt?

fengelniederhammer · 2026-07-16T06:59:03Z

+Sequence positions can be requested using the syntax `SequenceName[position]` (e.g. `S[501]` for position 501 of sequence `S`).
+For single-segmented genomes, the shorthand `[position]` (e.g. `[501]`) can be used.
+Position field column names in the response use the canonical sequence name from the reference genome (case-insensitive input), e.g. `s[501]` -> `S[501]`."""


Usually we tried to avoid those formulations (if single segmented ...). Instead, we generate them at run time: then we know in which case we are.

fengelniederhammer · 2026-07-16T09:26:17Z

    }

-    override fun validatePhyloTreeFields(source: String): Field {
+    override fun validatePhyloTreeFields(source: String): RequestField {


Isn't this function actually duplicate? Below, we have another validatePhyloTreeField that looks like basically the same.

I think this one isn't even used anywhere...

fengelniederhammer · 2026-07-16T09:29:21Z

+private val SEQUENCE_POSITION_REGEX = Regex("""^([A-Za-z][A-Za-z0-9_]*)\[(\d+)\]$""")
+private val SHORTHAND_POSITION_REGEX = Regex("""^\[(\d+)\]$""")


Suggested change

private val SEQUENCE_POSITION_REGEX = Regex("""^([A-Za-z][A-Za-z0-9_]*)\[(\d+)\]$""")

private val SHORTHAND_POSITION_REGEX = Regex("""^\[(\d+)\]$""")

private val SEQUENCE_POSITION_REGEX = Regex("""^([A-Za-z][A-Za-z0-9_]*)\[(\d+)]$""")

private val SHORTHAND_POSITION_REGEX = Regex("""^\[(\d+)]$""")

apparently those closing brackets don't need to be escaped.

fengelniederhammer · 2026-07-16T09:31:29Z

+data class SequencePositionField(
+    val sequenceName: String,
+    val position: Int,
+    val isSingleSegment: Boolean = false,
+) : RequestField {
+    /** Name used both as the SaneQL alias and as the response column key, e.g. `S[501]` or `[501]` for shorthand. */
+    val userFacingName: String get() = if (isSingleSegment) "[$position]" else "$sequenceName[$position]"
+    override val outputColumnName: String get() = userFacingName
+}


Suggested change

data class SequencePositionField(

val sequenceName: String,

val position: Int,

val isSingleSegment: Boolean = false,

) : RequestField {

/** Name used both as the SaneQL alias and as the response column key, e.g. `S[501]` or `[501]` for shorthand. */

val userFacingName: String get() = if (isSingleSegment) "[$position]" else "$sequenceName[$position]"

override val outputColumnName: String get() = userFacingName

}

data class SequencePositionField(

val sequenceName: String?,

val position: Int,

) : RequestField {

override val outputColumnName: String = if (sequenceName == null) {

"[$position]"

} else {

"$sequenceName[$position]"

}

}

We can encode isSingleSegment into sequenceName by making it nullable.

fengelniederhammer · 2026-07-16T09:33:12Z

+        val shorthandMatch = SHORTHAND_POSITION_REGEX.matchEntire(source)
+        if (shorthandMatch != null) {
+            val position = shorthandMatch.groupValues[1].toIntOrNull()
+                ?: throw BadRequestException("Invalid position in '$source': must be a positive integer")
+            if (position <= 0) {
+                throw BadRequestException("Invalid position in '$source': must be a positive integer, got $position")
+            }
+            if (!referenceGenomeSchema.isSingleSegmented()) {
+                throw BadRequestException(
+                    "Shorthand position syntax '[N]' can only be used for single-segmented genomes",
+                )
+            }
+            val canonicalName = referenceGenomeSchema.nucleotideSequences.first().name
+            return SequencePositionField(canonicalName, position, isSingleSegment = true)
+        }
+
+        val positionMatch = SEQUENCE_POSITION_REGEX.matchEntire(source)
+        if (positionMatch != null) {
+            val name = positionMatch.groupValues[1]
+            val position = positionMatch.groupValues[2].toIntOrNull()
+                ?: throw BadRequestException("Invalid position in '$source': must be a positive integer")
+            if (position <= 0) {
+                throw BadRequestException("Invalid position in '$source': must be a positive integer, got $position")
+            }
+            val canonicalName = referenceGenomeSchema.getSequenceNameFromCaseInsensitiveName(name)
+                ?: throw BadRequestException(
+                    "Unknown sequence '$name' in '$source', known sequences are: " +
+                        (referenceGenomeSchema.getNucleotideSequenceNames() + referenceGenomeSchema.getGeneNames())
+                            .joinToString(", "),
+                )
+            return SequencePositionField(canonicalName, position)
+        }


Could we collapse those two branches by making the sequence name in the regex optional? The position parsing is the same, and we could streamline the sequence name parsing.

fengelniederhammer · 2026-07-16T09:35:10Z

+            throw BadRequestException(
+                "Sequence position fields are not supported for this endpoint: " +
+                    positionFields.joinToString(", ") { it.userFacingName },
+            )


I find it a bit weird to first allow a SequencePositionField while parsing and then reject it. Wouldn't it be better to differentiate already in the types that the controller methods accept?

fengelniederhammer · 2026-07-16T09:41:14Z

+                                        SaneQlAssignment(
+                                            field.userFacingName,
+                                            SaneQlMethodCall(
+                                                SaneQlIdentifier(field.sequenceName),
+                                                "at",
+                                                listOf(SaneQlInt(field.position)),
+                                            ),
+                                        )


Do we need to deduplicate those somewhere?

vercel Bot deployed to Preview July 14, 2026 15:21 View deployment

feat(lapis): return position fields under user-facing name in response

15df660

Echo S[501] back as the response column key instead of the internal SaneQL alias S_501, so input and output are symmetric for callers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel Bot deployed to Preview July 14, 2026 15:43 View deployment

vercel Bot deployed to Preview July 14, 2026 15:58 View deployment

taepper linked an issue Jul 15, 2026 that may be closed by this pull request

Add aggregation by sequence positions #1766

Open

vercel Bot deployed to Preview July 15, 2026 07:34 View deployment

chore(lapis): fix ktlint warnings (redundant braces, import ordering)

bf2554e

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel Bot deployed to Preview July 15, 2026 07:41 View deployment

fhennig requested a review from fengelniederhammer July 15, 2026 07:44

fhennig self-assigned this Jul 15, 2026

fengelniederhammer requested a review from Copilot July 15, 2026 07:45

Copilot started reviewing on behalf of fengelniederhammer July 15, 2026 07:45 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread lapis/src/main/kotlin/org/genspectrum/lapis/controller/ControllerDescriptions.kt Outdated

fhennig mentioned this pull request Jul 15, 2026

New co-occurences over time component GenSpectrum/dashboard-components#1129

Open

Potential fix for pull request finding

08df65e

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

vercel Bot deployed to Preview July 15, 2026 07:53 View deployment

vercel Bot deployed to Preview July 15, 2026 09:15 View deployment

fengelniederhammer requested a review from Copilot July 15, 2026 09:21

Copilot started reviewing on behalf of fengelniederhammer July 15, 2026 09:22 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread lapis/src/main/kotlin/org/genspectrum/lapis/request/Field.kt

Comment thread lapis/src/main/kotlin/org/genspectrum/lapis/controller/LapisController.kt

vercel Bot deployed to Preview July 15, 2026 12:24 View deployment

fengelniederhammer requested a review from Copilot July 16, 2026 06:46

Copilot started reviewing on behalf of fengelniederhammer July 16, 2026 06:46 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

fengelniederhammer requested changes Jul 16, 2026

View reviewed changes

		private val SEQUENCE_POSITION_REGEX = Regex("""^([A-Za-z][A-Za-z0-9_]*)\[(\d+)\]$""")
		private val SHORTHAND_POSITION_REGEX = Regex("""^\[(\d+)\]$""")

Uh oh!

Conversation

fhennig commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example

Test plan

Uh oh!

vercel Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

fhennig commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

fengelniederhammer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fhennig commented Jul 14, 2026 •

edited

Loading

vercel Bot commented Jul 14, 2026 •

edited

Loading

fhennig commented Jul 15, 2026 •

edited

Loading