Repository navigation
[BHO-30]: add presigned S3 upload URL helper - #236
Closed
devluharuwalla wants to merge 19 commits into
Closed
devluharuwalla wants to merge 19 commits into
devluharuwalla wants to merge 19 commits into
Conversation
* Start database core tables and schema (#3) * Add initial database schema and migration (#3) Entities and generated migration for users, imports, daily/hourly/scheduled observations, daily records and audit log, plus docs/database-schema.md mapping each column to its source cell in the example sheets. * Remove unclear users note from schema doc * Keep PR to the schema doc; remove entities and migration Add s3Key to imports in the doc. * simplify types * update observibility * Add stations and possible sunshine to schema doc; require importId Client confirmed more stations are coming, so stationId joins the keys of the three daily tables now rather than in a later migration. Every row comes from an upload, so importId is not null on observations and records. * Add stationId to daily_records; explain visibility and present weather Keying records by station now avoids changing their primary key later if records become per-station. The hourly notes now say what column H's fractions and column I's weather codes mean. * Add average and normal temperature to daily_observations * Add barometer readings to scheduled_observations * Add notes to daily_records * Define users table in schema doc * Add initial schema migration and Kysely table types * Return date columns as YYYY-MM-DD strings * Note that imports rows are never updated or deleted * Correct date parser comment --------- Co-authored-by: ryaken-nakamoto <ryakenNakamoto@gmail.com>
* feature: added handler + handler test * enabled lint + enable jest and node types in tsconfig * add @types/aws-lambda dev dependency
* draft * more concise
* ticket-empty ingestion lambda * fix-changes
* Start historical data parser (#10) * Add shared parse result and historical record types (#10) Map @bho/weather-parsing in tsconfig.base.json so the lambdas resolve the library from source; nothing installs the file: dependency today. * Add historical records parser (#10) parseHistorical reads the All Daily Records CSV and returns one record per day, or every row error in the file. The fixture is the synthetic test sheet exported from Drive. * Rename parser tests to .test.ts to match the backend * Reject a gust direction or estimated flag without a gust
* Start historical data parser (#10) * Add shared parse result and historical record types (#10) Map @bho/weather-parsing in tsconfig.base.json so the lambdas resolve the library from source; nothing installs the file: dependency today. * Add historical records parser (#10) parseHistorical reads the All Daily Records CSV and returns one record per day, or every row error in the file. The fixture is the synthetic test sheet exported from Drive. * Rename parser tests to .test.ts to match the backend * Reject a gust direction or estimated flag without a gust * add initial daily parsing script * Refactor daily and hourly observation types, implement reporting for misformatted cells * add tests for daily parser based on example daily.csv file --------- Co-authored-by: Naman Rusia <naman.rusia14@gmail.com>
* Implement initial API Gateway handler and response structure with `health` endpoint * Add README and OpenAPI documentation for internal-serving-lambda * Add tests for health endpoint and enable linting for internal serving lambda
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ℹ️ Issue
Closes #30
📝 Description
Adds a helper that creates a 5-minute presigned PUT URL so staff can upload CSVs straight to S3.
internal-serving-lambda/src/presigned-upload-url.ts: rejects non-.csv files, builds the key server-side (daily/<date>/<uuid>.csv), and requiresContent-Type: text/csv.docs/upload-keys.md: key format and decisions.@aws-sdk/s3-request-presigner(pinned to matchclient-s3).✔️ Verification
npx jest --config apps/backend/internal-serving-lambda/jest.config.js(18/18 pass), plus tsc andnx lint internal-serving-lambdawith no errors.🏕️ (Optional) Future Work / Notes
requestChecksumCalculation: 'WHEN_REQUIRED', or uploads fail. The helper checks this.Content-Type: text/csvitself.importsneeds them). That's for a later ticket.