Refactor supplier analysis and product handling
- Updated `SupplierAnalysisResult` to include a `product` field and modified related tests. - Refactored `addRowsSheet` to accommodate changes in the product structure. - Enhanced UPC file analysis to utilize a new `toSupplierInputRecord` function for cleaner record creation. - Introduced new types for supplier input records and product observations. - Updated frontend components to handle new product details and analysis history. - Improved database writing functions to streamline run completion and error handling. - Added new API endpoints for product details and adjusted routing in the frontend.
This commit is contained in:
@@ -19,4 +19,5 @@ GOOGLE_API_KEY=your_google_api_key
|
||||
GOOGLE_CSE_ID=your_google_programmable_search_engine_id
|
||||
SERPAPI_API_KEY=your_serpapi_api_key_for_google_shopping
|
||||
|
||||
DB_CONNECTION_STRING=your_database_connection_string
|
||||
# Matches the default PostgreSQL service in docker-compose.yaml.
|
||||
DB_CONNECTION_STRING=postgres://asin_check:asin_check@localhost:5432/asin_check
|
||||
|
||||
@@ -90,6 +90,7 @@ Tracks competitor sellers across ASINs. Fetches storefronts, checks sellability
|
||||
| `src/config.ts` | Env var loading via `Bun.env` |
|
||||
| `src/db/index.ts` | Drizzle Postgres connection (shared pool) |
|
||||
| `src/db/schema.ts` | Drizzle schema for all tables |
|
||||
| `src/db/persistence.ts` | Product, observation, unified run-item, UPC resolution, and revision persistence |
|
||||
| `src/integrations/keepa.ts` | Keepa API: batch ASIN fetch, UPC lookup, auto rate-limiting |
|
||||
| `src/integrations/sp-api.ts` | SP-API: sellability, pricing+fees, UPC catalog lookup |
|
||||
| `src/integrations/cache.ts` | Redis caching (24h TTL for lead-list; 12h for mid-range) |
|
||||
@@ -112,4 +113,6 @@ Tracks competitor sellers across ASINs. Fetches storefronts, checks sellability
|
||||
- The supplier UPC pipeline must not call LM Studio.
|
||||
- Supplier UPC files resolve UPC/EAN through SP-API catalog lookup first; Keepa UPC lookup is fallback only (no-match or request-failure cases).
|
||||
- Supplier workbook output must keep `Ranked Leads`, `Skipped`, and `Summary` sheets.
|
||||
- Treat `products.asin` as the canonical normalized product identity; UPC values belong only in identifier and resolution records.
|
||||
- Store time-varying data in observations or revisions and retain run history rather than overwriting prior analysis.
|
||||
- When changing UPC supplier behavior, cover SP-API UPC parsing, deterministic scoring, and workbook export with `bun test`.
|
||||
|
||||
28
README.md
28
README.md
@@ -155,7 +155,7 @@ ranked sourcing workbook:
|
||||
2. Resolves UPCs to ASINs with SP-API catalog lookup first, then falls back to Keepa for no-match/request-failure cases.
|
||||
3. Enriches resolved ASINs with Keepa demand/competition data and SP-API sellability + FBA fees.
|
||||
4. Scores products with deterministic BUY/WATCH/SKIP logic; this path does not call LM Studio.
|
||||
5. Writes a ranked Excel workbook and persists rows into the existing `runs` + `results` tables.
|
||||
5. Writes a ranked Excel workbook and persists rows through unified runs, UPC resolution, product observation, and scoring-history tables.
|
||||
|
||||
CLI usage:
|
||||
|
||||
@@ -244,20 +244,28 @@ Numeric parsing accepts plain numbers as well as formatted values like `$12.50`,
|
||||
4. **Keepa fetch** — batch the sellable (uncached) ASINs in a single API call (up to 100 per request)
|
||||
5. **Enrich** — fetch SP-API pricing + FBA/FBM fees for sellable ASINs; combine with Keepa data and spreadsheet data
|
||||
6. **LLM analysis** — send batches of 5 sellable products to LM Studio for FBA/FBM/SKIP verdict; skipped ASINs get auto-SKIP verdict (confidence 100) and bypass LLM entirely
|
||||
7. **Output** — print results table to console (includes all ASINs), optionally write CSV/XLSX, and **persist results to a SQLite database**.
|
||||
7. **Output** — print results table to console (includes all ASINs), optionally write CSV/XLSX, and persist products, observations, run items, and analysis revisions to PostgreSQL.
|
||||
|
||||
## Persistent Storage with SQLite
|
||||
## Persistent Storage
|
||||
|
||||
Results from each run are now stored in a SQLite database named `db/results.db` by default. The SQLite implementation details are handled in `src/database.ts`. This allows you to:
|
||||
PostgreSQL persistence is managed with Drizzle in `src/db/schema.ts` and `src/db/persistence.ts`. ASINs are canonical product identities: all inputs normalize to uppercase 10-character alphanumeric keys before any product reference is stored.
|
||||
|
||||
- Revisit past analysis results.
|
||||
- Query and analyze historical data.
|
||||
- Track product performance over time.
|
||||
Core tables:
|
||||
|
||||
The database will automatically be created if it doesn't exist. Two tables are created:
|
||||
- `products`: one canonical row per ASIN with latest descriptive metadata.
|
||||
- `product_observations`: append-only marketplace, pricing, fee, and sellability snapshots.
|
||||
- `runs` and `run_items`: unified lifecycle/history for lead, category, supplier UPC, and stalker workflows.
|
||||
- `analysis_revisions` and `supplier_scores`: append-only analysis results; reanalysis does not overwrite prior decisions.
|
||||
- `sourcing_inputs`, `upc_resolutions`, and `product_identifiers`: source-row and confirmed identifier data kept separate from catalog products.
|
||||
- `stalker_run_details`, `stalker_scans`, and `stalker_inventory_items`: seller workflow provenance linked back to products and observations.
|
||||
|
||||
- `runs`: Stores metadata about each analysis run (timestamp, input file, output file, and summary counts).
|
||||
- `results`: Stores detailed analysis results for each product from each run, linked to the `runs` table.
|
||||
Unresolved or ambiguous supplier UPCs stay on their run item and resolution records; a UPC is never stored as an ASIN.
|
||||
|
||||
Web endpoints use unified identifiers:
|
||||
|
||||
- `GET /api/runs`, `GET /api/runs/:runId`, `GET /api/runs/:runId/items`
|
||||
- `GET /api/products`, `GET /api/products/:asin`
|
||||
- `POST /api/run-items/:itemId/reanalyze`
|
||||
|
||||
## Output columns
|
||||
|
||||
|
||||
35
docker-compose.yaml
Normal file
35
docker-compose.yaml
Normal file
@@ -0,0 +1,35 @@
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: asin_check
|
||||
POSTGRES_USER: asin_check
|
||||
POSTGRES_PASSWORD: asin_check
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
command: ["redis-server", "--appendonly", "yes"]
|
||||
ports:
|
||||
- "6379:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
@@ -1,217 +0,0 @@
|
||||
CREATE TYPE "public"."run_status" AS ENUM('running', 'ok', 'empty', 'failed', 'completed');--> statement-breakpoint
|
||||
CREATE TYPE "public"."run_type" AS ENUM('lead_analysis', 'category_analysis', 'supplier_upc', 'stalker');--> statement-breakpoint
|
||||
CREATE TABLE "analysis_results" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"run_id" integer NOT NULL,
|
||||
"asin" text NOT NULL,
|
||||
"product_name" text,
|
||||
"brand" text,
|
||||
"category" text,
|
||||
"upc" text,
|
||||
"unit_cost" real,
|
||||
"avg_price_90d_sheet" real,
|
||||
"selling_price_sheet" real,
|
||||
"fba_net_sheet" real,
|
||||
"gross_profit_dollar" real,
|
||||
"gross_profit_pct" real,
|
||||
"net_profit_sheet" real,
|
||||
"roi_sheet" real,
|
||||
"moq" integer,
|
||||
"moq_cost" real,
|
||||
"qty_available" integer,
|
||||
"supplier" text,
|
||||
"source_url" text,
|
||||
"asin_link" text,
|
||||
"promo_coupon_code" text,
|
||||
"notes" text,
|
||||
"lead_date" text,
|
||||
"current_price" real,
|
||||
"avg_price_90d" real,
|
||||
"sales_rank" integer,
|
||||
"rank_avg_90d" integer,
|
||||
"monthly_sold" integer,
|
||||
"rank_drops_30d" integer,
|
||||
"rank_drops_90d" integer,
|
||||
"seller_count" integer,
|
||||
"amazon_is_seller" boolean,
|
||||
"amazon_buybox_share_pct_90d" real,
|
||||
"fba_fee" real,
|
||||
"fbm_fee" real,
|
||||
"referral_percent" real,
|
||||
"can_sell" text,
|
||||
"sellability_status" text,
|
||||
"sellability_reason" text,
|
||||
"supplier_score" real,
|
||||
"supplier_profit" real,
|
||||
"supplier_margin" real,
|
||||
"supplier_roi" real,
|
||||
"supplier_reason" text,
|
||||
"upc_lookup_status" text,
|
||||
"upc_lookup_reason" text,
|
||||
"candidate_asins" text,
|
||||
"verdict" text NOT NULL,
|
||||
"confidence" real,
|
||||
"reasoning" text,
|
||||
"fetched_at" timestamp with time zone NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "category_product_results" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"asin" text NOT NULL,
|
||||
"run_id" integer NOT NULL,
|
||||
"name" text NOT NULL,
|
||||
"brand" text,
|
||||
"category" text,
|
||||
"unit_cost" real,
|
||||
"current_price" real,
|
||||
"avg_price_90d" real,
|
||||
"avg_price_90d_sheet" real,
|
||||
"selling_price_sheet" real,
|
||||
"sales_rank" integer,
|
||||
"sales_rank_avg_90d" integer,
|
||||
"seller_count" integer,
|
||||
"amazon_is_seller" boolean,
|
||||
"amazon_buybox_share_pct_90d" real,
|
||||
"monthly_sold" integer,
|
||||
"rank_drops_30d" integer,
|
||||
"rank_drops_90d" integer,
|
||||
"fba_fee" real,
|
||||
"fbm_fee" real,
|
||||
"referral_percent" real,
|
||||
"can_sell" text,
|
||||
"sellability_status" text,
|
||||
"sellability_reason" text,
|
||||
"verdict" text NOT NULL,
|
||||
"confidence" real NOT NULL,
|
||||
"reasoning" text,
|
||||
"fetched_at" timestamp with time zone NOT NULL,
|
||||
CONSTRAINT "category_product_results_asin_unique" UNIQUE("asin")
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "runs" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"type" "run_type" NOT NULL,
|
||||
"input_file" text,
|
||||
"output_file" text,
|
||||
"status" "run_status" DEFAULT 'running' NOT NULL,
|
||||
"error_message" text,
|
||||
"total_products" integer,
|
||||
"fba_count" integer,
|
||||
"fbm_count" integer,
|
||||
"skip_count" integer,
|
||||
"category_id" integer,
|
||||
"category_label" text,
|
||||
"top_asins_checked" integer,
|
||||
"available_asins" integer,
|
||||
"started_at" timestamp with time zone DEFAULT now() NOT NULL,
|
||||
"completed_at" timestamp with time zone
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "sellers" (
|
||||
"seller_id" text PRIMARY KEY NOT NULL,
|
||||
"seller_name" text,
|
||||
"rating" real,
|
||||
"rating_count" integer,
|
||||
"storefront_asin_total" integer,
|
||||
"persisted_inventory_sample_count" integer,
|
||||
"last_updated_at" timestamp with time zone NOT NULL,
|
||||
"raw_seller_json" text
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "stalker_asin_scans" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"run_id" integer NOT NULL,
|
||||
"source_asin" text NOT NULL,
|
||||
"title" text,
|
||||
"offer_count" integer DEFAULT 0 NOT NULL,
|
||||
"candidate_seller_count" integer DEFAULT 0 NOT NULL,
|
||||
"matched_seller_count" integer DEFAULT 0 NOT NULL,
|
||||
"fetched_at" timestamp with time zone NOT NULL,
|
||||
"raw_product_json" text,
|
||||
CONSTRAINT "uq_stalker_scans_run_asin" UNIQUE("run_id","source_asin")
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "stalker_asin_sellers" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"scan_id" integer NOT NULL,
|
||||
"seller_id" text NOT NULL,
|
||||
"offer_price" real,
|
||||
"condition" text,
|
||||
"is_fba" boolean,
|
||||
"stock" integer,
|
||||
"seller_rating" real,
|
||||
"seller_rating_count" integer,
|
||||
"raw_offer_json" text,
|
||||
CONSTRAINT "uq_stalker_asin_sellers_scan_seller" UNIQUE("scan_id","seller_id")
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "stalker_runs" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"input_file" text NOT NULL,
|
||||
"started_at" timestamp with time zone NOT NULL,
|
||||
"completed_at" timestamp with time zone,
|
||||
"requested_asins" integer DEFAULT 0 NOT NULL,
|
||||
"skipped_asins" integer DEFAULT 0 NOT NULL,
|
||||
"scanned_asins" integer DEFAULT 0 NOT NULL,
|
||||
"source_asins_with_matches" integer DEFAULT 0 NOT NULL,
|
||||
"candidate_sellers" integer DEFAULT 0 NOT NULL,
|
||||
"qualifying_sellers" integer DEFAULT 0 NOT NULL,
|
||||
"matched_sellers" integer DEFAULT 0 NOT NULL,
|
||||
"seller_metadata_requests" integer DEFAULT 0 NOT NULL,
|
||||
"seller_storefront_requests" integer DEFAULT 0 NOT NULL,
|
||||
"inventory_sellability_checked_asins" integer DEFAULT 0 NOT NULL,
|
||||
"inventory_sellability_available_asins" integer DEFAULT 0 NOT NULL,
|
||||
"inventory_sellability_excluded_asins" integer DEFAULT 0 NOT NULL,
|
||||
"persisted_inventory_asins" integer DEFAULT 0 NOT NULL,
|
||||
"status" text NOT NULL,
|
||||
"error_message" text
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "stalker_seller_inventory" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"run_id" integer NOT NULL,
|
||||
"seller_id" text NOT NULL,
|
||||
"asin" text NOT NULL,
|
||||
"can_sell" boolean,
|
||||
"sellability_status" text,
|
||||
"sellability_reason" text,
|
||||
"product_title" text,
|
||||
"brand" text,
|
||||
"category_tree" text,
|
||||
"current_price" real,
|
||||
"avg_price_90d" real,
|
||||
"sales_rank" integer,
|
||||
"monthly_sold" integer,
|
||||
"seller_count" integer,
|
||||
"amazon_is_seller" boolean,
|
||||
"raw_product_json" text,
|
||||
"last_seen_at" timestamp with time zone NOT NULL,
|
||||
"raw_inventory_json" text,
|
||||
CONSTRAINT "uq_stalker_inventory_run_seller_asin" UNIQUE("run_id","seller_id","asin")
|
||||
);
|
||||
--> statement-breakpoint
|
||||
ALTER TABLE "analysis_results" ADD CONSTRAINT "analysis_results_run_id_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."runs"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "category_product_results" ADD CONSTRAINT "category_product_results_run_id_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."runs"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "stalker_asin_scans" ADD CONSTRAINT "stalker_asin_scans_run_id_stalker_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."stalker_runs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "stalker_asin_sellers" ADD CONSTRAINT "stalker_asin_sellers_scan_id_stalker_asin_scans_id_fk" FOREIGN KEY ("scan_id") REFERENCES "public"."stalker_asin_scans"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "stalker_asin_sellers" ADD CONSTRAINT "stalker_asin_sellers_seller_id_sellers_seller_id_fk" FOREIGN KEY ("seller_id") REFERENCES "public"."sellers"("seller_id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "stalker_seller_inventory" ADD CONSTRAINT "stalker_seller_inventory_run_id_stalker_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."stalker_runs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "stalker_seller_inventory" ADD CONSTRAINT "stalker_seller_inventory_seller_id_sellers_seller_id_fk" FOREIGN KEY ("seller_id") REFERENCES "public"."sellers"("seller_id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
|
||||
CREATE INDEX "idx_analysis_results_run_id" ON "analysis_results" USING btree ("run_id");--> statement-breakpoint
|
||||
CREATE INDEX "idx_analysis_results_asin" ON "analysis_results" USING btree ("asin");--> statement-breakpoint
|
||||
CREATE INDEX "idx_analysis_results_verdict" ON "analysis_results" USING btree ("verdict");--> statement-breakpoint
|
||||
CREATE INDEX "idx_analysis_results_sellability_status" ON "analysis_results" USING btree ("sellability_status");--> statement-breakpoint
|
||||
CREATE INDEX "idx_analysis_results_fetched_at" ON "analysis_results" USING btree ("fetched_at");--> statement-breakpoint
|
||||
CREATE INDEX "idx_category_results_run_id" ON "category_product_results" USING btree ("run_id");--> statement-breakpoint
|
||||
CREATE INDEX "idx_category_results_verdict" ON "category_product_results" USING btree ("verdict");--> statement-breakpoint
|
||||
CREATE INDEX "idx_category_results_sellability_status" ON "category_product_results" USING btree ("sellability_status");--> statement-breakpoint
|
||||
CREATE INDEX "idx_category_results_fetched_at" ON "category_product_results" USING btree ("fetched_at");--> statement-breakpoint
|
||||
CREATE INDEX "idx_runs_started_at" ON "runs" USING btree ("started_at");--> statement-breakpoint
|
||||
CREATE INDEX "idx_runs_type" ON "runs" USING btree ("type");--> statement-breakpoint
|
||||
CREATE INDEX "idx_runs_status" ON "runs" USING btree ("status");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_scans_run_id" ON "stalker_asin_scans" USING btree ("run_id");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_scans_source_asin" ON "stalker_asin_scans" USING btree ("source_asin");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_runs_started_at" ON "stalker_runs" USING btree ("started_at");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_inventory_seller_id" ON "stalker_seller_inventory" USING btree ("seller_id");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_inventory_asin" ON "stalker_seller_inventory" USING btree ("asin");--> statement-breakpoint
|
||||
CREATE INDEX "idx_stalker_inventory_product_title" ON "stalker_seller_inventory" USING btree ("product_title");
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,13 +0,0 @@
|
||||
{
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"entries": [
|
||||
{
|
||||
"idx": 0,
|
||||
"version": "7",
|
||||
"when": 1779683900467,
|
||||
"tag": "0000_gorgeous_william_stryker",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
114
src/analysis-pipeline.test.ts
Normal file
114
src/analysis-pipeline.test.ts
Normal file
@@ -0,0 +1,114 @@
|
||||
import { beforeEach, expect, mock, test } from "bun:test";
|
||||
import { processProductChunk } from "./analysis-pipeline.ts";
|
||||
import type { ProductRecord } from "./types.ts";
|
||||
|
||||
const fetchKeepaDataBatchMock = mock(async (asins: string[]) => {
|
||||
return new Map(
|
||||
asins.map((asin) => [
|
||||
asin,
|
||||
{
|
||||
currentPrice: 20,
|
||||
avgPrice90: 18,
|
||||
minPrice90: null,
|
||||
maxPrice90: null,
|
||||
salesRank: 100,
|
||||
salesRankAvg90: null,
|
||||
salesRankDrops30: null,
|
||||
salesRankDrops90: null,
|
||||
sellerCount: 3,
|
||||
amazonIsSeller: false,
|
||||
amazonBuyboxSharePct90d: null,
|
||||
buyBoxSeller: null,
|
||||
buyBoxPrice: null,
|
||||
monthlySold: 50,
|
||||
categoryTree: [],
|
||||
},
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
const fetchSellabilityBatchMock = mock(async (asins: string[]) => {
|
||||
return new Map(
|
||||
asins.map((asin) => [
|
||||
asin,
|
||||
asin === "B000000002"
|
||||
? {
|
||||
canSell: false,
|
||||
sellabilityStatus: "restricted" as const,
|
||||
sellabilityReason: "Approval required",
|
||||
}
|
||||
: {
|
||||
canSell: true,
|
||||
sellabilityStatus: "available" as const,
|
||||
sellabilityReason: "Available",
|
||||
},
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
const fetchSpApiPricingAndFeesMock = mock(async () => ({
|
||||
fbaFee: 4,
|
||||
fbmFee: 2,
|
||||
referralFeePercent: 15,
|
||||
estimatedSalePrice: 20,
|
||||
canSell: true,
|
||||
sellabilityStatus: "available" as const,
|
||||
sellabilityReason: "Available",
|
||||
}));
|
||||
|
||||
const analyzeProductsMock = mock(async (products: any[]) =>
|
||||
products.map((product) => ({
|
||||
asin: product.record.asin,
|
||||
verdict: "FBA" as const,
|
||||
confidence: 95,
|
||||
reasoning: "Analyzed",
|
||||
})),
|
||||
);
|
||||
|
||||
const getCacheMock = mock(async () => null);
|
||||
const setCacheMock = mock(async () => undefined);
|
||||
|
||||
beforeEach(() => {
|
||||
fetchKeepaDataBatchMock.mockClear();
|
||||
fetchSellabilityBatchMock.mockClear();
|
||||
fetchSpApiPricingAndFeesMock.mockClear();
|
||||
analyzeProductsMock.mockClear();
|
||||
getCacheMock.mockClear();
|
||||
setCacheMock.mockClear();
|
||||
});
|
||||
|
||||
test("lead analysis retains restricted input rows as SKIP without LLM analysis", async () => {
|
||||
const products: ProductRecord[] = [
|
||||
{ asin: "B000000001", name: "Available", unitCost: 5 },
|
||||
{ asin: "B000000002", name: "Restricted", unitCost: 6 },
|
||||
];
|
||||
|
||||
const results = await processProductChunk(products, {
|
||||
llmBatchDelayMs: 0,
|
||||
llmRetryDelayMs: 0,
|
||||
dependencies: {
|
||||
fetchKeepaDataBatch: fetchKeepaDataBatchMock,
|
||||
fetchSellabilityBatch: fetchSellabilityBatchMock,
|
||||
fetchSpApiPricingAndFees: fetchSpApiPricingAndFeesMock,
|
||||
analyzeProducts: analyzeProductsMock,
|
||||
getCache: getCacheMock,
|
||||
setCache: setCacheMock,
|
||||
},
|
||||
});
|
||||
|
||||
expect(results).toHaveLength(2);
|
||||
expect(results.map((result) => result.product.record.asin)).toEqual([
|
||||
"B000000001",
|
||||
"B000000002",
|
||||
]);
|
||||
expect(results.find((result) => result.product.record.asin === "B000000002")?.verdict)
|
||||
.toEqual({
|
||||
asin: "B000000002",
|
||||
verdict: "SKIP",
|
||||
confidence: 100,
|
||||
reasoning: "Approval required",
|
||||
});
|
||||
expect(fetchKeepaDataBatchMock.mock.calls[0]?.[0]).toEqual(["B000000001"]);
|
||||
expect(fetchSpApiPricingAndFeesMock.mock.calls).toHaveLength(1);
|
||||
expect(analyzeProductsMock.mock.calls[0]?.[0]).toHaveLength(1);
|
||||
});
|
||||
@@ -16,6 +16,15 @@ export const DEFAULT_PRICING_CONCURRENCY = 5;
|
||||
|
||||
export type SellabilityFilter = "available" | "all";
|
||||
|
||||
type AnalysisPipelineDependencies = {
|
||||
fetchKeepaDataBatch: typeof fetchKeepaDataBatch;
|
||||
fetchSellabilityBatch: typeof fetchSellabilityBatch;
|
||||
fetchSpApiPricingAndFees: typeof fetchSpApiPricingAndFees;
|
||||
getCache: typeof getCache;
|
||||
setCache: typeof setCache;
|
||||
analyzeProducts: typeof analyzeProducts;
|
||||
};
|
||||
|
||||
export type AnalysisPipelineOptions = {
|
||||
llmBatchSize?: number;
|
||||
pricingConcurrency?: number;
|
||||
@@ -23,6 +32,7 @@ export type AnalysisPipelineOptions = {
|
||||
llmRetryDelayMs?: number;
|
||||
sellability?: SellabilityFilter;
|
||||
useClaude?: boolean;
|
||||
dependencies?: Partial<AnalysisPipelineDependencies>;
|
||||
};
|
||||
|
||||
export function chunkArray<T>(items: T[], chunkSize: number): T[][] {
|
||||
@@ -62,23 +72,33 @@ export async function processProductChunk(
|
||||
const llmRetryDelayMs = Math.max(0, options.llmRetryDelayMs ?? 10_000);
|
||||
const sellabilityFilter = options.sellability ?? "available";
|
||||
const useClaude = options.useClaude === true;
|
||||
const dependencies: AnalysisPipelineDependencies = {
|
||||
fetchKeepaDataBatch,
|
||||
fetchSellabilityBatch,
|
||||
fetchSpApiPricingAndFees,
|
||||
getCache,
|
||||
setCache,
|
||||
analyzeProducts,
|
||||
...options.dependencies,
|
||||
};
|
||||
|
||||
console.log(`\nChecking cache for ${products.length} products...`);
|
||||
const cached = new Map<string, EnrichedProduct>();
|
||||
const excludedCachedAsins = new Set<string>();
|
||||
const excludedCached = new Map<string, EnrichedProduct>();
|
||||
const uncachedProducts: ProductRecord[] = [];
|
||||
|
||||
for (const p of products) {
|
||||
const hit = await getCache(p.asin);
|
||||
const hit = await dependencies.getCache(p.asin);
|
||||
if (hit) {
|
||||
const currentSourceProduct = { ...hit, record: p };
|
||||
if (
|
||||
sellabilityFilter === "all" ||
|
||||
hit.spApi.sellabilityStatus === "available"
|
||||
) {
|
||||
console.log(` [cache hit] ${p.asin}`);
|
||||
cached.set(p.asin, hit);
|
||||
cached.set(p.asin, currentSourceProduct);
|
||||
} else {
|
||||
excludedCachedAsins.add(p.asin);
|
||||
excludedCached.set(p.asin, currentSourceProduct);
|
||||
console.log(
|
||||
` [exclude cached] ${p.asin} - status=${hit.spApi.sellabilityStatus}`,
|
||||
);
|
||||
@@ -89,7 +109,7 @@ export async function processProductChunk(
|
||||
}
|
||||
|
||||
console.log(
|
||||
`${cached.size} cached available, ${excludedCachedAsins.size} cached excluded, ${uncachedProducts.length} to fetch`,
|
||||
`${cached.size} cached available, ${excludedCached.size} cached excluded, ${uncachedProducts.length} to fetch`,
|
||||
);
|
||||
|
||||
const sellabilityMap = new Map<string, SellabilityInfo>();
|
||||
@@ -100,7 +120,7 @@ export async function processProductChunk(
|
||||
console.log(
|
||||
`\nChecking sellability for ${uncachedProducts.length} ASINs...`,
|
||||
);
|
||||
const sellResults = await fetchSellabilityBatch(
|
||||
const sellResults = await dependencies.fetchSellabilityBatch(
|
||||
uncachedProducts.map((p) => p.asin),
|
||||
);
|
||||
|
||||
@@ -143,7 +163,7 @@ export async function processProductChunk(
|
||||
if (availableProducts.length > 0) {
|
||||
console.log(`\nFetching ${availableProducts.length} ASINs from Keepa...`);
|
||||
try {
|
||||
keepaResults = await fetchKeepaDataBatch(
|
||||
keepaResults = await dependencies.fetchKeepaDataBatch(
|
||||
availableProducts.map((p) => p.asin),
|
||||
);
|
||||
} catch (err) {
|
||||
@@ -168,7 +188,10 @@ export async function processProductChunk(
|
||||
sellabilityStatus: "unknown" as const,
|
||||
sellabilityReason: "Sellability check returned no result",
|
||||
};
|
||||
const spApi = await fetchSpApiPricingAndFees(p.asin, sellability);
|
||||
const spApi = await dependencies.fetchSpApiPricingAndFees(
|
||||
p.asin,
|
||||
sellability,
|
||||
);
|
||||
|
||||
const keepa = keepaResults.get(p.asin);
|
||||
if (keepa?.currentPrice && spApi.estimatedSalePrice === 0) {
|
||||
@@ -196,17 +219,33 @@ export async function processProductChunk(
|
||||
const availableAsins = new Set(availableProducts.map((ap) => ap.asin));
|
||||
|
||||
for (const p of products) {
|
||||
if (excludedCachedAsins.has(p.asin)) {
|
||||
const excludedCachedProduct = excludedCached.get(p.asin);
|
||||
if (excludedCachedProduct) {
|
||||
enriched.push({ ...excludedCachedProduct, record: p });
|
||||
continue;
|
||||
}
|
||||
|
||||
const cachedProduct = cached.get(p.asin);
|
||||
if (cachedProduct) {
|
||||
enriched.push(cachedProduct);
|
||||
enriched.push({ ...cachedProduct, record: p });
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!availableAsins.has(p.asin)) {
|
||||
const sellability = sellabilityMap.get(p.asin);
|
||||
if (sellability) {
|
||||
enriched.push({
|
||||
record: p,
|
||||
keepa: null,
|
||||
spApi: {
|
||||
...unknownSpApiData(
|
||||
sellability.sellabilityReason ?? "Product is not available",
|
||||
),
|
||||
...sellability,
|
||||
},
|
||||
fetchedAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -221,19 +260,41 @@ export async function processProductChunk(
|
||||
fetchedAt: new Date().toISOString(),
|
||||
};
|
||||
|
||||
await setCache(p.asin, product);
|
||||
await dependencies.setCache(p.asin, product);
|
||||
enriched.push(product);
|
||||
}
|
||||
|
||||
const resultsByProduct = new Map<EnrichedProduct, AnalysisResult>();
|
||||
const llmProducts: EnrichedProduct[] = [];
|
||||
for (const product of enriched) {
|
||||
if (
|
||||
sellabilityFilter !== "all" &&
|
||||
product.spApi.sellabilityStatus !== "available"
|
||||
) {
|
||||
resultsByProduct.set(product, {
|
||||
product,
|
||||
verdict: {
|
||||
asin: product.record.asin,
|
||||
verdict: "SKIP",
|
||||
confidence: 100,
|
||||
reasoning:
|
||||
product.spApi.sellabilityReason ??
|
||||
`Sellability status: ${product.spApi.sellabilityStatus}`,
|
||||
},
|
||||
});
|
||||
} else {
|
||||
llmProducts.push(product);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(
|
||||
`\nAnalyzing ${enriched.length} products via LLM (batch size: ${llmBatchSize})...\n`,
|
||||
`\nAnalyzing ${llmProducts.length} products via LLM (batch size: ${llmBatchSize})...\n`,
|
||||
);
|
||||
|
||||
const results: AnalysisResult[] = [];
|
||||
for (let i = 0; i < enriched.length; i += llmBatchSize) {
|
||||
const batch = enriched.slice(i, i + llmBatchSize);
|
||||
for (let i = 0; i < llmProducts.length; i += llmBatchSize) {
|
||||
const batch = llmProducts.slice(i, i + llmBatchSize);
|
||||
const batchNum = Math.floor(i / llmBatchSize) + 1;
|
||||
const totalBatches = Math.ceil(enriched.length / llmBatchSize);
|
||||
const totalBatches = Math.ceil(llmProducts.length / llmBatchSize);
|
||||
console.log(` LLM batch ${batchNum}/${totalBatches}...`);
|
||||
|
||||
if (i > 0 && llmBatchDelayMs > 0) {
|
||||
@@ -242,7 +303,7 @@ export async function processProductChunk(
|
||||
|
||||
let verdicts;
|
||||
try {
|
||||
verdicts = await analyzeProducts(batch, {
|
||||
verdicts = await dependencies.analyzeProducts(batch, {
|
||||
ignoreSellability: sellabilityFilter === "all",
|
||||
useClaude,
|
||||
});
|
||||
@@ -251,7 +312,7 @@ export async function processProductChunk(
|
||||
await wait(llmRetryDelayMs);
|
||||
}
|
||||
try {
|
||||
verdicts = await analyzeProducts(batch, {
|
||||
verdicts = await dependencies.analyzeProducts(batch, {
|
||||
ignoreSellability: sellabilityFilter === "all",
|
||||
useClaude,
|
||||
});
|
||||
@@ -264,7 +325,7 @@ export async function processProductChunk(
|
||||
const enrichedProduct = batch[j];
|
||||
if (!enrichedProduct) continue;
|
||||
|
||||
results.push({
|
||||
resultsByProduct.set(enrichedProduct, {
|
||||
product: enrichedProduct,
|
||||
verdict: verdicts?.[j] ?? {
|
||||
asin: enrichedProduct.record.asin,
|
||||
@@ -276,5 +337,7 @@ export async function processProductChunk(
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
return enriched
|
||||
.map((product) => resultsByProduct.get(product))
|
||||
.filter((result): result is AnalysisResult => result !== undefined);
|
||||
}
|
||||
|
||||
13
src/asin.test.ts
Normal file
13
src/asin.test.ts
Normal file
@@ -0,0 +1,13 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { normalizeAsin, requireAsin } from "./asin.ts";
|
||||
|
||||
test("normalizes any valid ten-character ASIN including ISBN-style values", () => {
|
||||
expect(normalizeAsin(" b07sn9bhvv ")).toBe("B07SN9BHVV");
|
||||
expect(normalizeAsin("0306406152")).toBe("0306406152");
|
||||
});
|
||||
|
||||
test("rejects values that cannot be canonical product ASIN keys", () => {
|
||||
expect(normalizeAsin("short")).toBeNull();
|
||||
expect(normalizeAsin("B07SN9BHV!")).toBeNull();
|
||||
expect(() => requireAsin("012345678901")).toThrow("Invalid ASIN");
|
||||
});
|
||||
14
src/asin.ts
Normal file
14
src/asin.ts
Normal file
@@ -0,0 +1,14 @@
|
||||
export const ASIN_PATTERN = /^[A-Z0-9]{10}$/;
|
||||
|
||||
export function normalizeAsin(value: unknown): string | null {
|
||||
const asin = String(value ?? "").trim().toUpperCase();
|
||||
return ASIN_PATTERN.test(asin) ? asin : null;
|
||||
}
|
||||
|
||||
export function requireAsin(value: unknown): string {
|
||||
const asin = normalizeAsin(value);
|
||||
if (!asin) {
|
||||
throw new Error(`Invalid ASIN: "${String(value ?? "").trim()}"`);
|
||||
}
|
||||
return asin;
|
||||
}
|
||||
@@ -1,8 +1,11 @@
|
||||
import { existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { db } from "../db/index.ts";
|
||||
import { runs, categoryProductResults } from "../db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import {
|
||||
createCategoryRun,
|
||||
persistLlmResults,
|
||||
updateCategoryRun,
|
||||
} from "../db/persistence.ts";
|
||||
import { config } from "../config.ts";
|
||||
import { analyzeProducts } from "../integrations/llm.ts";
|
||||
import { fetchSellabilityBatch, fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
|
||||
@@ -144,26 +147,7 @@ export async function insertCategoryRunSummary(
|
||||
summary: CategoryRunSummary,
|
||||
runTimestamp: string,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "category_analysis",
|
||||
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
|
||||
categoryId: summary.categoryId,
|
||||
categoryLabel: summary.categoryLabel,
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
errorMessage: summary.error || null,
|
||||
startedAt: new Date(runTimestamp),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
|
||||
if (!row) throw new Error("Failed to insert category run.");
|
||||
return row.id;
|
||||
return createCategoryRun(summary, runTimestamp);
|
||||
}
|
||||
|
||||
export async function updateCategoryRunSummary(
|
||||
@@ -179,20 +163,7 @@ export async function updateCategoryRunSummary(
|
||||
| "error"
|
||||
>,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
status: summary.status as typeof runs.$inferInsert.status,
|
||||
errorMessage: summary.error || null,
|
||||
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
|
||||
})
|
||||
.where(eq(runs.id, runId));
|
||||
await updateCategoryRun(runId, summary);
|
||||
}
|
||||
|
||||
export async function insertProductAnalysisResults(
|
||||
@@ -200,89 +171,10 @@ export async function insertProductAnalysisResults(
|
||||
results: AnalysisResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((r) => {
|
||||
const price =
|
||||
r.product.keepa?.currentPrice ??
|
||||
r.product.record.sellingPriceFromSheet ??
|
||||
r.product.spApi.estimatedSalePrice;
|
||||
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
|
||||
|
||||
return {
|
||||
asin: r.product.record.asin,
|
||||
runId,
|
||||
name: r.product.record.name,
|
||||
brand: r.product.record.brand ?? null,
|
||||
category:
|
||||
r.product.record.category ??
|
||||
r.product.keepa?.categoryTree?.join(" > ") ??
|
||||
null,
|
||||
unitCost: r.product.record.unitCost ?? null,
|
||||
currentPrice: price ?? null,
|
||||
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
|
||||
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
|
||||
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
|
||||
salesRank: rank ?? null,
|
||||
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
|
||||
sellerCount: r.product.keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
monthlySold: r.product.keepa?.monthlySold ?? null,
|
||||
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
|
||||
fbaFee: r.product.spApi.fbaFee ?? null,
|
||||
fbmFee: r.product.spApi.fbmFee ?? null,
|
||||
referralPercent: r.product.spApi.referralFeePercent ?? null,
|
||||
canSell:
|
||||
r.product.spApi.canSell == null
|
||||
? "unknown"
|
||||
: r.product.spApi.canSell
|
||||
? "yes"
|
||||
: "no",
|
||||
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
|
||||
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
|
||||
verdict: r.verdict.verdict,
|
||||
confidence: r.verdict.confidence,
|
||||
reasoning: r.verdict.reasoning ?? null,
|
||||
fetchedAt: new Date(r.product.fetchedAt),
|
||||
};
|
||||
await persistLlmResults(runId, results, {
|
||||
source: "category_analysis",
|
||||
metadataSource: "catalog",
|
||||
});
|
||||
|
||||
await db
|
||||
.insert(categoryProductResults)
|
||||
.values(rows)
|
||||
.onConflictDoUpdate({
|
||||
target: categoryProductResults.asin,
|
||||
set: {
|
||||
runId: sql`EXCLUDED.run_id`,
|
||||
name: sql`EXCLUDED.name`,
|
||||
brand: sql`EXCLUDED.brand`,
|
||||
category: sql`EXCLUDED.category`,
|
||||
unitCost: sql`EXCLUDED.unit_cost`,
|
||||
currentPrice: sql`EXCLUDED.current_price`,
|
||||
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
|
||||
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
|
||||
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
|
||||
salesRank: sql`EXCLUDED.sales_rank`,
|
||||
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
|
||||
sellerCount: sql`EXCLUDED.seller_count`,
|
||||
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
|
||||
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
|
||||
monthlySold: sql`EXCLUDED.monthly_sold`,
|
||||
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
|
||||
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
|
||||
fbaFee: sql`EXCLUDED.fba_fee`,
|
||||
fbmFee: sql`EXCLUDED.fbm_fee`,
|
||||
referralPercent: sql`EXCLUDED.referral_percent`,
|
||||
canSell: sql`EXCLUDED.can_sell`,
|
||||
sellabilityStatus: sql`EXCLUDED.sellability_status`,
|
||||
sellabilityReason: sql`EXCLUDED.sellability_reason`,
|
||||
verdict: sql`EXCLUDED.verdict`,
|
||||
confidence: sql`EXCLUDED.confidence`,
|
||||
reasoning: sql`EXCLUDED.reasoning`,
|
||||
fetchedAt: sql`EXCLUDED.fetched_at`,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function loadCategoryBlacklist(filePath: string): Set<number> {
|
||||
@@ -664,7 +556,11 @@ async function fetchCategoryBestSellerAsins(
|
||||
for (const value of candidates) {
|
||||
if (Array.isArray(value)) {
|
||||
return [
|
||||
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
|
||||
...new Set(
|
||||
value
|
||||
.map((v) => normalizeAsin(v))
|
||||
.filter((asin): asin is string => asin !== null),
|
||||
),
|
||||
].slice(0, limit);
|
||||
}
|
||||
}
|
||||
@@ -919,7 +815,7 @@ async function fetchKeepaEnrichmentMap(
|
||||
|
||||
const products = Array.isArray(data?.products) ? data.products : [];
|
||||
for (const product of products) {
|
||||
const asin = String(product?.asin ?? "").trim();
|
||||
const asin = normalizeAsin(product?.asin);
|
||||
if (!asin) continue;
|
||||
out.set(asin, {
|
||||
keepa: parseKeepaProduct(product),
|
||||
|
||||
@@ -2,9 +2,12 @@ import { existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createInterface } from "node:readline/promises";
|
||||
import { stdin as input, stdout as output } from "node:process";
|
||||
import { db } from "../db/index.ts";
|
||||
import { runs, categoryProductResults } from "../db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import {
|
||||
createCategoryRun,
|
||||
persistLlmResults,
|
||||
updateCategoryRun,
|
||||
} from "../db/persistence.ts";
|
||||
import { config } from "../config.ts";
|
||||
import {
|
||||
connectCache,
|
||||
@@ -479,26 +482,7 @@ export async function insertCategoryRunSummary(
|
||||
summary: CategoryRunSummary,
|
||||
runTimestamp: string,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "category_analysis",
|
||||
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
|
||||
categoryId: summary.categoryId,
|
||||
categoryLabel: summary.categoryLabel,
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
errorMessage: summary.error || null,
|
||||
startedAt: new Date(runTimestamp),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
|
||||
if (!row) throw new Error("Failed to insert category run.");
|
||||
return row.id;
|
||||
return createCategoryRun(summary, runTimestamp);
|
||||
}
|
||||
|
||||
export async function updateCategoryRunSummary(
|
||||
@@ -514,20 +498,7 @@ export async function updateCategoryRunSummary(
|
||||
| "error"
|
||||
>,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
status: summary.status as typeof runs.$inferInsert.status,
|
||||
errorMessage: summary.error || null,
|
||||
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
|
||||
})
|
||||
.where(eq(runs.id, runId));
|
||||
await updateCategoryRun(runId, summary);
|
||||
}
|
||||
|
||||
export async function insertProductAnalysisResults(
|
||||
@@ -535,89 +506,10 @@ export async function insertProductAnalysisResults(
|
||||
results: AnalysisResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((r) => {
|
||||
const price =
|
||||
r.product.keepa?.currentPrice ??
|
||||
r.product.record.sellingPriceFromSheet ??
|
||||
r.product.spApi.estimatedSalePrice;
|
||||
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
|
||||
|
||||
return {
|
||||
asin: r.product.record.asin,
|
||||
runId,
|
||||
name: r.product.record.name,
|
||||
brand: r.product.record.brand ?? null,
|
||||
category:
|
||||
r.product.record.category ??
|
||||
r.product.keepa?.categoryTree?.join(" > ") ??
|
||||
null,
|
||||
unitCost: r.product.record.unitCost ?? null,
|
||||
currentPrice: price ?? null,
|
||||
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
|
||||
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
|
||||
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
|
||||
salesRank: rank ?? null,
|
||||
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
|
||||
sellerCount: r.product.keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
monthlySold: r.product.keepa?.monthlySold ?? null,
|
||||
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
|
||||
fbaFee: r.product.spApi.fbaFee ?? null,
|
||||
fbmFee: r.product.spApi.fbmFee ?? null,
|
||||
referralPercent: r.product.spApi.referralFeePercent ?? null,
|
||||
canSell:
|
||||
r.product.spApi.canSell == null
|
||||
? "unknown"
|
||||
: r.product.spApi.canSell
|
||||
? "yes"
|
||||
: "no",
|
||||
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
|
||||
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
|
||||
verdict: r.verdict.verdict,
|
||||
confidence: r.verdict.confidence,
|
||||
reasoning: r.verdict.reasoning ?? null,
|
||||
fetchedAt: new Date(r.product.fetchedAt),
|
||||
};
|
||||
await persistLlmResults(runId, results, {
|
||||
source: "category_analysis",
|
||||
metadataSource: "catalog",
|
||||
});
|
||||
|
||||
await db
|
||||
.insert(categoryProductResults)
|
||||
.values(rows)
|
||||
.onConflictDoUpdate({
|
||||
target: categoryProductResults.asin,
|
||||
set: {
|
||||
runId: sql`EXCLUDED.run_id`,
|
||||
name: sql`EXCLUDED.name`,
|
||||
brand: sql`EXCLUDED.brand`,
|
||||
category: sql`EXCLUDED.category`,
|
||||
unitCost: sql`EXCLUDED.unit_cost`,
|
||||
currentPrice: sql`EXCLUDED.current_price`,
|
||||
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
|
||||
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
|
||||
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
|
||||
salesRank: sql`EXCLUDED.sales_rank`,
|
||||
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
|
||||
sellerCount: sql`EXCLUDED.seller_count`,
|
||||
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
|
||||
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
|
||||
monthlySold: sql`EXCLUDED.monthly_sold`,
|
||||
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
|
||||
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
|
||||
fbaFee: sql`EXCLUDED.fba_fee`,
|
||||
fbmFee: sql`EXCLUDED.fbm_fee`,
|
||||
referralPercent: sql`EXCLUDED.referral_percent`,
|
||||
canSell: sql`EXCLUDED.can_sell`,
|
||||
sellabilityStatus: sql`EXCLUDED.sellability_status`,
|
||||
sellabilityReason: sql`EXCLUDED.sellability_reason`,
|
||||
verdict: sql`EXCLUDED.verdict`,
|
||||
confidence: sql`EXCLUDED.confidence`,
|
||||
reasoning: sql`EXCLUDED.reasoning`,
|
||||
fetchedAt: sql`EXCLUDED.fetched_at`,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function loadCategoryBlacklist(filePath: string): Set<number> {
|
||||
@@ -999,7 +891,11 @@ async function fetchCategoryBestSellerAsins(
|
||||
for (const value of candidates) {
|
||||
if (Array.isArray(value)) {
|
||||
return [
|
||||
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
|
||||
...new Set(
|
||||
value
|
||||
.map((v) => normalizeAsin(v))
|
||||
.filter((asin): asin is string => asin !== null),
|
||||
),
|
||||
].slice(0, limit);
|
||||
}
|
||||
}
|
||||
@@ -1258,7 +1154,7 @@ async function fetchKeepaEnrichmentMap(
|
||||
|
||||
const products = Array.isArray(data?.products) ? data.products : [];
|
||||
for (const product of products) {
|
||||
const asin = String(product?.asin ?? "").trim();
|
||||
const asin = normalizeAsin(product?.asin);
|
||||
if (!asin) continue;
|
||||
const parsed = {
|
||||
keepa: parseKeepaProduct(product),
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
import { existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { db } from "../db/index.ts";
|
||||
import { runs, categoryProductResults } from "../db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import {
|
||||
createCategoryRun,
|
||||
persistLlmResults,
|
||||
updateCategoryRun,
|
||||
} from "../db/persistence.ts";
|
||||
import { config } from "../config.ts";
|
||||
import { analyzeProducts } from "../integrations/llm.ts";
|
||||
import { fetchSellabilityBatch, fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
|
||||
@@ -176,26 +179,7 @@ export async function insertCategoryRunSummary(
|
||||
summary: CategoryRunSummary,
|
||||
runTimestamp: string,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "category_analysis",
|
||||
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
|
||||
categoryId: summary.categoryId,
|
||||
categoryLabel: summary.categoryLabel,
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
errorMessage: summary.error || null,
|
||||
startedAt: new Date(runTimestamp),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
|
||||
if (!row) throw new Error("Failed to insert category run.");
|
||||
return row.id;
|
||||
return createCategoryRun(summary, runTimestamp);
|
||||
}
|
||||
|
||||
export async function updateCategoryRunSummary(
|
||||
@@ -211,20 +195,7 @@ export async function updateCategoryRunSummary(
|
||||
| "error"
|
||||
>,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
topAsinsChecked: summary.topAsinsChecked,
|
||||
availableAsins: summary.availableAsins,
|
||||
totalProducts: summary.topAsinsChecked,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
status: summary.status as typeof runs.$inferInsert.status,
|
||||
errorMessage: summary.error || null,
|
||||
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
|
||||
})
|
||||
.where(eq(runs.id, runId));
|
||||
await updateCategoryRun(runId, summary);
|
||||
}
|
||||
|
||||
export async function insertProductAnalysisResults(
|
||||
@@ -232,89 +203,10 @@ export async function insertProductAnalysisResults(
|
||||
results: AnalysisResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((r) => {
|
||||
const price =
|
||||
r.product.keepa?.currentPrice ??
|
||||
r.product.record.sellingPriceFromSheet ??
|
||||
r.product.spApi.estimatedSalePrice;
|
||||
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
|
||||
|
||||
return {
|
||||
asin: r.product.record.asin,
|
||||
runId,
|
||||
name: r.product.record.name,
|
||||
brand: r.product.record.brand ?? null,
|
||||
category:
|
||||
r.product.record.category ??
|
||||
r.product.keepa?.categoryTree?.join(" > ") ??
|
||||
null,
|
||||
unitCost: r.product.record.unitCost ?? null,
|
||||
currentPrice: price ?? null,
|
||||
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
|
||||
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
|
||||
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
|
||||
salesRank: rank ?? null,
|
||||
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
|
||||
sellerCount: r.product.keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
monthlySold: r.product.keepa?.monthlySold ?? null,
|
||||
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
|
||||
fbaFee: r.product.spApi.fbaFee ?? null,
|
||||
fbmFee: r.product.spApi.fbmFee ?? null,
|
||||
referralPercent: r.product.spApi.referralFeePercent ?? null,
|
||||
canSell:
|
||||
r.product.spApi.canSell == null
|
||||
? "unknown"
|
||||
: r.product.spApi.canSell
|
||||
? "yes"
|
||||
: "no",
|
||||
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
|
||||
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
|
||||
verdict: r.verdict.verdict,
|
||||
confidence: r.verdict.confidence,
|
||||
reasoning: r.verdict.reasoning ?? null,
|
||||
fetchedAt: new Date(r.product.fetchedAt),
|
||||
};
|
||||
await persistLlmResults(runId, results, {
|
||||
source: "category_analysis",
|
||||
metadataSource: "catalog",
|
||||
});
|
||||
|
||||
await db
|
||||
.insert(categoryProductResults)
|
||||
.values(rows)
|
||||
.onConflictDoUpdate({
|
||||
target: categoryProductResults.asin,
|
||||
set: {
|
||||
runId: sql`EXCLUDED.run_id`,
|
||||
name: sql`EXCLUDED.name`,
|
||||
brand: sql`EXCLUDED.brand`,
|
||||
category: sql`EXCLUDED.category`,
|
||||
unitCost: sql`EXCLUDED.unit_cost`,
|
||||
currentPrice: sql`EXCLUDED.current_price`,
|
||||
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
|
||||
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
|
||||
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
|
||||
salesRank: sql`EXCLUDED.sales_rank`,
|
||||
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
|
||||
sellerCount: sql`EXCLUDED.seller_count`,
|
||||
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
|
||||
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
|
||||
monthlySold: sql`EXCLUDED.monthly_sold`,
|
||||
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
|
||||
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
|
||||
fbaFee: sql`EXCLUDED.fba_fee`,
|
||||
fbmFee: sql`EXCLUDED.fbm_fee`,
|
||||
referralPercent: sql`EXCLUDED.referral_percent`,
|
||||
canSell: sql`EXCLUDED.can_sell`,
|
||||
sellabilityStatus: sql`EXCLUDED.sellability_status`,
|
||||
sellabilityReason: sql`EXCLUDED.sellability_reason`,
|
||||
verdict: sql`EXCLUDED.verdict`,
|
||||
confidence: sql`EXCLUDED.confidence`,
|
||||
reasoning: sql`EXCLUDED.reasoning`,
|
||||
fetchedAt: sql`EXCLUDED.fetched_at`,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function loadCategoryBlacklist(filePath: string): Set<number> {
|
||||
@@ -696,7 +588,11 @@ async function fetchCategoryBestSellerAsins(
|
||||
for (const value of candidates) {
|
||||
if (Array.isArray(value)) {
|
||||
return [
|
||||
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
|
||||
...new Set(
|
||||
value
|
||||
.map((v) => normalizeAsin(v))
|
||||
.filter((asin): asin is string => asin !== null),
|
||||
),
|
||||
].slice(0, limit);
|
||||
}
|
||||
}
|
||||
@@ -951,7 +847,7 @@ async function fetchKeepaEnrichmentMap(
|
||||
|
||||
const products = Array.isArray(data?.products) ? data.products : [];
|
||||
for (const product of products) {
|
||||
const asin = String(product?.asin ?? "").trim();
|
||||
const asin = normalizeAsin(product?.asin);
|
||||
if (!asin) continue;
|
||||
out.set(asin, {
|
||||
keepa: parseKeepaProduct(product),
|
||||
|
||||
541
src/db/persistence.ts
Normal file
541
src/db/persistence.ts
Normal file
@@ -0,0 +1,541 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { requireAsin, normalizeAsin } from "../asin.ts";
|
||||
import type {
|
||||
AnalysisResult,
|
||||
ProductRecord,
|
||||
SupplierAnalysisResult,
|
||||
} from "../types.ts";
|
||||
import { db } from "./index.ts";
|
||||
import {
|
||||
analysisRevisions,
|
||||
analysisRunStats,
|
||||
categoryRunDetails,
|
||||
productIdentifiers,
|
||||
productObservations,
|
||||
products,
|
||||
runItems,
|
||||
runs,
|
||||
sourcingInputs,
|
||||
supplierScores,
|
||||
upcResolutionCandidates,
|
||||
upcResolutions,
|
||||
} from "./schema.ts";
|
||||
|
||||
type Executor = any;
|
||||
type MetadataSource = "input" | "catalog";
|
||||
|
||||
type ProductSeed = {
|
||||
asin: string;
|
||||
name?: string | null;
|
||||
brand?: string | null;
|
||||
category?: string | null;
|
||||
metadataSource?: MetadataSource;
|
||||
fetchedAt?: Date;
|
||||
};
|
||||
|
||||
export type CategoryRunSummaryInput = {
|
||||
categoryId: number;
|
||||
categoryLabel: string;
|
||||
topAsinsChecked: number;
|
||||
availableAsins: number;
|
||||
fba: number;
|
||||
fbm: number;
|
||||
skip: number;
|
||||
status: "running" | "ok" | "empty" | "failed";
|
||||
error: string;
|
||||
};
|
||||
|
||||
export type RunCounts = {
|
||||
totalProducts: number;
|
||||
fbaCount: number;
|
||||
fbmCount: number;
|
||||
skipCount: number;
|
||||
};
|
||||
|
||||
function emptyToNull(value: string | undefined | null): string | null {
|
||||
const trimmed = value?.trim();
|
||||
return trimmed ? trimmed : null;
|
||||
}
|
||||
|
||||
function productCategory(record: ProductRecord, result: AnalysisResult): string | null {
|
||||
return emptyToNull(
|
||||
record.category ?? result.product.keepa?.categoryTree?.join(" > "),
|
||||
);
|
||||
}
|
||||
|
||||
export async function upsertProduct(
|
||||
seed: ProductSeed,
|
||||
executor: Executor = db,
|
||||
): Promise<string> {
|
||||
const asin = requireAsin(seed.asin);
|
||||
const now = seed.fetchedAt ?? new Date();
|
||||
const isCatalog = seed.metadataSource === "catalog";
|
||||
|
||||
await executor
|
||||
.insert(products)
|
||||
.values({
|
||||
asin,
|
||||
name: emptyToNull(seed.name),
|
||||
brand: emptyToNull(seed.brand),
|
||||
category: emptyToNull(seed.category),
|
||||
metadataFetchedAt: isCatalog ? now : null,
|
||||
firstSeenAt: now,
|
||||
lastSeenAt: now,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: products.asin,
|
||||
set: {
|
||||
lastSeenAt: sql`GREATEST(${products.lastSeenAt}, EXCLUDED.last_seen_at)`,
|
||||
name: isCatalog
|
||||
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.name, '') IS NOT NULL THEN EXCLUDED.name ELSE ${products.name} END`
|
||||
: sql`COALESCE(${products.name}, NULLIF(EXCLUDED.name, ''))`,
|
||||
brand: isCatalog
|
||||
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.brand, '') IS NOT NULL THEN EXCLUDED.brand ELSE ${products.brand} END`
|
||||
: sql`COALESCE(${products.brand}, NULLIF(EXCLUDED.brand, ''))`,
|
||||
category: isCatalog
|
||||
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.category, '') IS NOT NULL THEN EXCLUDED.category ELSE ${products.category} END`
|
||||
: sql`COALESCE(${products.category}, NULLIF(EXCLUDED.category, ''))`,
|
||||
metadataFetchedAt: isCatalog
|
||||
? sql`GREATEST(COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz), EXCLUDED.metadata_fetched_at)`
|
||||
: products.metadataFetchedAt,
|
||||
},
|
||||
});
|
||||
|
||||
return asin;
|
||||
}
|
||||
|
||||
export async function insertObservation(
|
||||
runId: number,
|
||||
result: AnalysisResult,
|
||||
source: string,
|
||||
executor: Executor = db,
|
||||
): Promise<number> {
|
||||
const fetchedAt = new Date(result.product.fetchedAt);
|
||||
const record = result.product.record;
|
||||
const keepa = result.product.keepa;
|
||||
const spApi = result.product.spApi;
|
||||
const asin = requireAsin(record.asin);
|
||||
const [observation] = await executor
|
||||
.insert(productObservations)
|
||||
.values({
|
||||
productAsin: asin,
|
||||
runId,
|
||||
source,
|
||||
currentPrice:
|
||||
keepa?.currentPrice ??
|
||||
record.sellingPriceFromSheet ??
|
||||
spApi.estimatedSalePrice ??
|
||||
null,
|
||||
avgPrice90d: keepa?.avgPrice90 ?? null,
|
||||
salesRank: keepa?.salesRank ?? record.amazonRank ?? null,
|
||||
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
|
||||
monthlySold: keepa?.monthlySold ?? null,
|
||||
rankDrops30d: keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: keepa?.salesRankDrops90 ?? null,
|
||||
sellerCount: keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
fbaFee: spApi.fbaFee ?? null,
|
||||
fbmFee: spApi.fbmFee ?? null,
|
||||
referralPercent: spApi.referralFeePercent ?? null,
|
||||
canSell: spApi.canSell,
|
||||
sellabilityStatus: spApi.sellabilityStatus,
|
||||
sellabilityReason: spApi.sellabilityReason ?? null,
|
||||
fetchedAt,
|
||||
})
|
||||
.returning({ id: productObservations.id });
|
||||
|
||||
if (!observation) throw new Error(`Failed to insert observation for ${asin}`);
|
||||
return observation.id;
|
||||
}
|
||||
|
||||
function sourcingInputValues(runItemId: number, record: ProductRecord) {
|
||||
return {
|
||||
runItemId,
|
||||
suppliedName: emptyToNull(record.name),
|
||||
suppliedBrand: emptyToNull(record.brand),
|
||||
suppliedCategory: emptyToNull(record.category),
|
||||
unitCost: record.unitCost ?? null,
|
||||
avgPrice90dSheet: record.avgPrice90FromSheet ?? null,
|
||||
sellingPriceSheet: record.sellingPriceFromSheet ?? null,
|
||||
fbaNetSheet: record.fbaNet ?? null,
|
||||
grossProfitDollar: record.grossProfit ?? null,
|
||||
grossProfitPct: record.grossProfitPct ?? null,
|
||||
netProfitSheet: record.netProfitFromSheet ?? null,
|
||||
roiSheet: record.roiFromSheet ?? null,
|
||||
moq: record.moq ?? null,
|
||||
moqCost: record.moqCost ?? null,
|
||||
qtyAvailable: record.totalQtyAvail ?? null,
|
||||
supplier: emptyToNull(record.supplier),
|
||||
sourceUrl: emptyToNull(record.sourceUrl),
|
||||
asinLink: emptyToNull(record.asinLink),
|
||||
promoCouponCode: emptyToNull(record.promoCouponCode),
|
||||
notes: emptyToNull(record.notes),
|
||||
leadDate: emptyToNull(record.leadDate),
|
||||
};
|
||||
}
|
||||
|
||||
export async function persistLlmResults(
|
||||
runId: number,
|
||||
results: AnalysisResult[],
|
||||
options: {
|
||||
source: string;
|
||||
metadataSource?: MetadataSource;
|
||||
preserveSourcingInput?: boolean;
|
||||
sourceInventoryIds?: Map<string, number>;
|
||||
},
|
||||
): Promise<void> {
|
||||
for (const result of results) {
|
||||
const record = result.product.record;
|
||||
const fetchedAt = new Date(result.product.fetchedAt);
|
||||
const asin = await upsertProduct({
|
||||
asin: record.asin,
|
||||
name: record.name,
|
||||
brand: record.brand,
|
||||
category: productCategory(record, result),
|
||||
metadataSource: options.metadataSource ?? "input",
|
||||
fetchedAt,
|
||||
});
|
||||
const [item] = await db
|
||||
.insert(runItems)
|
||||
.values({
|
||||
runId,
|
||||
productAsin: asin,
|
||||
sourceInventoryItemId: options.sourceInventoryIds?.get(asin) ?? null,
|
||||
})
|
||||
.returning({ id: runItems.id });
|
||||
if (!item) throw new Error(`Failed to insert run item for ${asin}`);
|
||||
|
||||
if (options.preserveSourcingInput) {
|
||||
await db.insert(sourcingInputs).values(sourcingInputValues(item.id, record));
|
||||
}
|
||||
|
||||
const observationId = await insertObservation(runId, result, options.source);
|
||||
await db.insert(analysisRevisions).values({
|
||||
runItemId: item.id,
|
||||
observationId,
|
||||
method: "llm",
|
||||
decision: result.verdict.verdict,
|
||||
confidence: result.verdict.confidence,
|
||||
reasoning: result.verdict.reasoning ?? null,
|
||||
analyzedAt: fetchedAt,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function supplierSourcingValues(runItemId: number, result: SupplierAnalysisResult) {
|
||||
return {
|
||||
runItemId,
|
||||
suppliedName: emptyToNull(result.record.name),
|
||||
suppliedBrand: emptyToNull(result.record.brand),
|
||||
suppliedCategory: emptyToNull(result.record.category),
|
||||
unitCost: result.record.unitCost ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
async function insertSupplierObservation(
|
||||
runId: number,
|
||||
productAsin: string,
|
||||
result: SupplierAnalysisResult,
|
||||
): Promise<number | null> {
|
||||
const keepa = result.keepa;
|
||||
const spApi = result.spApi;
|
||||
if (!spApi && !keepa) return null;
|
||||
|
||||
const [row] = await db
|
||||
.insert(productObservations)
|
||||
.values({
|
||||
productAsin,
|
||||
runId,
|
||||
source: "supplier_upc",
|
||||
currentPrice: result.score.salePrice,
|
||||
avgPrice90d: keepa?.avgPrice90 ?? null,
|
||||
salesRank: keepa?.salesRank ?? null,
|
||||
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
|
||||
monthlySold: keepa?.monthlySold ?? null,
|
||||
rankDrops30d: keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: keepa?.salesRankDrops90 ?? null,
|
||||
sellerCount: keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
fbaFee: spApi?.fbaFee ?? null,
|
||||
fbmFee: spApi?.fbmFee ?? null,
|
||||
referralPercent: spApi?.referralFeePercent ?? null,
|
||||
canSell: spApi?.canSell ?? null,
|
||||
sellabilityStatus: spApi?.sellabilityStatus ?? null,
|
||||
sellabilityReason: spApi?.sellabilityReason ?? null,
|
||||
fetchedAt: new Date(result.fetchedAt),
|
||||
})
|
||||
.returning({ id: productObservations.id });
|
||||
return row?.id ?? null;
|
||||
}
|
||||
|
||||
export async function persistSupplierResults(
|
||||
runId: number,
|
||||
results: SupplierAnalysisResult[],
|
||||
): Promise<void> {
|
||||
for (const result of results) {
|
||||
const resolvedAsin = normalizeAsin(result.lookup.asin);
|
||||
if (resolvedAsin) {
|
||||
await upsertProduct({
|
||||
asin: resolvedAsin,
|
||||
name: result.record.name,
|
||||
brand: result.record.brand,
|
||||
category: result.record.category,
|
||||
metadataSource: "input",
|
||||
fetchedAt: new Date(result.fetchedAt),
|
||||
});
|
||||
if (result.keepa?.categoryTree?.length) {
|
||||
await upsertProduct({
|
||||
asin: resolvedAsin,
|
||||
category: result.keepa.categoryTree.join(" > "),
|
||||
metadataSource: "catalog",
|
||||
fetchedAt: new Date(result.fetchedAt),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const [item] = await db
|
||||
.insert(runItems)
|
||||
.values({
|
||||
runId,
|
||||
productAsin: resolvedAsin,
|
||||
sourceRow: result.rowNumber ?? null,
|
||||
})
|
||||
.returning({ id: runItems.id });
|
||||
if (!item) throw new Error("Failed to insert supplier run item");
|
||||
|
||||
await db.insert(sourcingInputs).values(supplierSourcingValues(item.id, result));
|
||||
await db.insert(upcResolutions).values({
|
||||
runItemId: item.id,
|
||||
requestedUpc: result.upc,
|
||||
normalizedUpc: result.lookup.normalizedUpc,
|
||||
provider: result.lookup.provider ?? "unknown",
|
||||
status: result.lookup.status,
|
||||
reason: result.lookup.reason ?? null,
|
||||
resolvedProductAsin: resolvedAsin,
|
||||
resolvedAt: new Date(result.fetchedAt),
|
||||
});
|
||||
|
||||
for (const candidate of result.lookup.candidateAsins) {
|
||||
const candidateAsin = normalizeAsin(candidate);
|
||||
if (!candidateAsin) continue;
|
||||
await upsertProduct({ asin: candidateAsin, fetchedAt: new Date(result.fetchedAt) });
|
||||
await db
|
||||
.insert(upcResolutionCandidates)
|
||||
.values({ runItemId: item.id, productAsin: candidateAsin })
|
||||
.onConflictDoUpdate({
|
||||
target: [
|
||||
upcResolutionCandidates.runItemId,
|
||||
upcResolutionCandidates.productAsin,
|
||||
],
|
||||
set: { productAsin: sql`EXCLUDED.product_asin` },
|
||||
});
|
||||
}
|
||||
|
||||
if (resolvedAsin) {
|
||||
await db
|
||||
.insert(productIdentifiers)
|
||||
.values({
|
||||
productAsin: resolvedAsin,
|
||||
identifierType:
|
||||
result.lookup.normalizedUpc.length === 12
|
||||
? "upc"
|
||||
: result.lookup.normalizedUpc.length === 13
|
||||
? "ean"
|
||||
: "gtin",
|
||||
identifierValue: result.lookup.normalizedUpc,
|
||||
source: "supplier_upc",
|
||||
confirmedAt: new Date(result.fetchedAt),
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: [
|
||||
productIdentifiers.identifierType,
|
||||
productIdentifiers.identifierValue,
|
||||
],
|
||||
set: {
|
||||
productAsin: resolvedAsin,
|
||||
source: "supplier_upc",
|
||||
confirmedAt: new Date(result.fetchedAt),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
const observationId = resolvedAsin
|
||||
? await insertSupplierObservation(runId, resolvedAsin, result)
|
||||
: null;
|
||||
const [revision] = await db
|
||||
.insert(analysisRevisions)
|
||||
.values({
|
||||
runItemId: item.id,
|
||||
observationId,
|
||||
method: "supplier_scoring",
|
||||
decision: result.score.verdict,
|
||||
confidence: result.score.score,
|
||||
reasoning: result.score.reason,
|
||||
analyzedAt: new Date(result.fetchedAt),
|
||||
})
|
||||
.returning({ id: analysisRevisions.id });
|
||||
if (!revision) throw new Error("Failed to insert supplier analysis revision");
|
||||
|
||||
await db.insert(supplierScores).values({
|
||||
revisionId: revision.id,
|
||||
score: result.score.score,
|
||||
salePrice: result.score.salePrice,
|
||||
fbaFee: result.score.fbaFee,
|
||||
profit: result.score.profit,
|
||||
margin: result.score.margin,
|
||||
roi: result.score.roi,
|
||||
reason: result.score.reason,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export async function createCategoryRun(
|
||||
summary: CategoryRunSummaryInput,
|
||||
runTimestamp: string,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "category_analysis",
|
||||
status: summary.status,
|
||||
errorMessage: summary.error || null,
|
||||
startedAt: new Date(runTimestamp),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
if (!row) throw new Error("Failed to insert category run.");
|
||||
|
||||
await db.insert(categoryRunDetails).values({
|
||||
runId: row.id,
|
||||
categoryId: summary.categoryId,
|
||||
categoryLabel: summary.categoryLabel,
|
||||
checkedAsinCount: summary.topAsinsChecked,
|
||||
});
|
||||
await db.insert(analysisRunStats).values({
|
||||
runId: row.id,
|
||||
processedCount: summary.topAsinsChecked,
|
||||
availableCount: summary.availableAsins,
|
||||
analyzedCount: summary.fba + summary.fbm + summary.skip,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
});
|
||||
return row.id;
|
||||
}
|
||||
|
||||
export async function updateCategoryRun(
|
||||
runId: number,
|
||||
summary: Pick<
|
||||
CategoryRunSummaryInput,
|
||||
| "topAsinsChecked"
|
||||
| "availableAsins"
|
||||
| "fba"
|
||||
| "fbm"
|
||||
| "skip"
|
||||
| "status"
|
||||
| "error"
|
||||
>,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
status: summary.status,
|
||||
errorMessage: summary.error || null,
|
||||
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
|
||||
})
|
||||
.where(sql`${runs.id} = ${runId}`);
|
||||
await db
|
||||
.insert(categoryRunDetails)
|
||||
.values({
|
||||
runId,
|
||||
categoryId: 0,
|
||||
categoryLabel: "",
|
||||
checkedAsinCount: summary.topAsinsChecked,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: categoryRunDetails.runId,
|
||||
set: { checkedAsinCount: summary.topAsinsChecked },
|
||||
});
|
||||
await db
|
||||
.insert(analysisRunStats)
|
||||
.values({
|
||||
runId,
|
||||
processedCount: summary.topAsinsChecked,
|
||||
availableCount: summary.availableAsins,
|
||||
analyzedCount: summary.fba + summary.fbm + summary.skip,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: analysisRunStats.runId,
|
||||
set: {
|
||||
processedCount: summary.topAsinsChecked,
|
||||
availableCount: summary.availableAsins,
|
||||
analyzedCount: summary.fba + summary.fbm + summary.skip,
|
||||
fbaCount: summary.fba,
|
||||
fbmCount: summary.fbm,
|
||||
skipCount: summary.skip,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export async function refreshRunStats(runId: number): Promise<RunCounts> {
|
||||
const [stats] = await db.execute(
|
||||
sql<{
|
||||
total: string;
|
||||
fba: string | null;
|
||||
fbm: string | null;
|
||||
buy: string | null;
|
||||
watch: string | null;
|
||||
skip: string | null;
|
||||
}>`WITH latest AS (
|
||||
SELECT DISTINCT ON (ri.id) ar.decision
|
||||
FROM run_items ri
|
||||
JOIN analysis_revisions ar ON ar.run_item_id = ri.id
|
||||
WHERE ri.run_id = ${runId}
|
||||
ORDER BY ri.id, ar.analyzed_at DESC, ar.id DESC
|
||||
)
|
||||
SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN decision = 'FBA' THEN 1 ELSE 0 END) AS fba,
|
||||
SUM(CASE WHEN decision = 'FBM' THEN 1 ELSE 0 END) AS fbm,
|
||||
SUM(CASE WHEN decision = 'BUY' THEN 1 ELSE 0 END) AS buy,
|
||||
SUM(CASE WHEN decision = 'WATCH' THEN 1 ELSE 0 END) AS watch,
|
||||
SUM(CASE WHEN decision = 'SKIP' THEN 1 ELSE 0 END) AS skip
|
||||
FROM latest`,
|
||||
);
|
||||
|
||||
const counts = {
|
||||
totalProducts: Number(stats?.total ?? 0),
|
||||
fbaCount: Number(stats?.fba ?? 0),
|
||||
fbmCount: Number(stats?.fbm ?? 0),
|
||||
skipCount: Number(stats?.skip ?? 0),
|
||||
};
|
||||
await db
|
||||
.insert(analysisRunStats)
|
||||
.values({
|
||||
runId,
|
||||
processedCount: counts.totalProducts,
|
||||
analyzedCount: counts.totalProducts,
|
||||
fbaCount: counts.fbaCount,
|
||||
fbmCount: counts.fbmCount,
|
||||
buyCount: Number(stats?.buy ?? 0),
|
||||
watchCount: Number(stats?.watch ?? 0),
|
||||
skipCount: counts.skipCount,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: analysisRunStats.runId,
|
||||
set: {
|
||||
processedCount: counts.totalProducts,
|
||||
analyzedCount: counts.totalProducts,
|
||||
fbaCount: counts.fbaCount,
|
||||
fbmCount: counts.fbmCount,
|
||||
buyCount: Number(stats?.buy ?? 0),
|
||||
watchCount: Number(stats?.watch ?? 0),
|
||||
skipCount: counts.skipCount,
|
||||
},
|
||||
});
|
||||
return counts;
|
||||
}
|
||||
504
src/db/schema.ts
504
src/db/schema.ts
@@ -1,9 +1,13 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import {
|
||||
type AnyPgColumn,
|
||||
boolean,
|
||||
check,
|
||||
index,
|
||||
integer,
|
||||
pgEnum,
|
||||
pgTable,
|
||||
primaryKey,
|
||||
real,
|
||||
serial,
|
||||
text,
|
||||
@@ -11,13 +15,12 @@ import {
|
||||
unique,
|
||||
} from "drizzle-orm/pg-core";
|
||||
|
||||
// ─── Enums ───────────────────────────────────────────────────────────────────
|
||||
|
||||
export const runTypeEnum = pgEnum("run_type", [
|
||||
"lead_analysis",
|
||||
"category_analysis",
|
||||
"supplier_upc",
|
||||
"stalker",
|
||||
"stalker_analysis",
|
||||
]);
|
||||
|
||||
export const runStatusEnum = pgEnum("run_status", [
|
||||
@@ -28,29 +31,54 @@ export const runStatusEnum = pgEnum("run_status", [
|
||||
"completed",
|
||||
]);
|
||||
|
||||
// ─── Runs ─────────────────────────────────────────────────────────────────────
|
||||
// Unified run log; replaces the old `runs` and `category_analysis_runs` tables.
|
||||
// Category-specific columns (categoryId, categoryLabel, …) are null for
|
||||
// lead_analysis / supplier_upc runs.
|
||||
export const analysisMethodEnum = pgEnum("analysis_method", [
|
||||
"llm",
|
||||
"supplier_scoring",
|
||||
]);
|
||||
|
||||
export const analysisDecisionEnum = pgEnum("analysis_decision", [
|
||||
"FBA",
|
||||
"FBM",
|
||||
"BUY",
|
||||
"WATCH",
|
||||
"SKIP",
|
||||
]);
|
||||
|
||||
export const products = pgTable(
|
||||
"products",
|
||||
{
|
||||
asin: text("asin").primaryKey(),
|
||||
name: text("name"),
|
||||
brand: text("brand"),
|
||||
category: text("category"),
|
||||
metadataFetchedAt: timestamp("metadata_fetched_at", { withTimezone: true }),
|
||||
firstSeenAt: timestamp("first_seen_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
lastSeenAt: timestamp("last_seen_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => [
|
||||
check("ck_products_asin", sql`${t.asin} ~ '^[A-Z0-9]{10}$'`),
|
||||
index("idx_products_name").on(t.name),
|
||||
index("idx_products_last_seen_at").on(t.lastSeenAt),
|
||||
],
|
||||
);
|
||||
|
||||
export const runs = pgTable(
|
||||
"runs",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
type: runTypeEnum("type").notNull(),
|
||||
parentRunId: integer("parent_run_id").references(
|
||||
(): AnyPgColumn => runs.id,
|
||||
{ onDelete: "cascade" },
|
||||
),
|
||||
inputFile: text("input_file"),
|
||||
outputFile: text("output_file"),
|
||||
status: runStatusEnum("status").notNull().default("running"),
|
||||
errorMessage: text("error_message"),
|
||||
totalProducts: integer("total_products"),
|
||||
fbaCount: integer("fba_count"),
|
||||
fbmCount: integer("fbm_count"),
|
||||
skipCount: integer("skip_count"),
|
||||
// Category-pipeline only
|
||||
categoryId: integer("category_id"),
|
||||
categoryLabel: text("category_label"),
|
||||
topAsinsChecked: integer("top_asins_checked"),
|
||||
availableAsins: integer("available_asins"),
|
||||
startedAt: timestamp("started_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
@@ -60,210 +88,262 @@ export const runs = pgTable(
|
||||
index("idx_runs_started_at").on(t.startedAt),
|
||||
index("idx_runs_type").on(t.type),
|
||||
index("idx_runs_status").on(t.status),
|
||||
index("idx_runs_parent_run_id").on(t.parentRunId),
|
||||
],
|
||||
);
|
||||
|
||||
// ─── Analysis results ─────────────────────────────────────────────────────────
|
||||
// Archival table: one row per product per run for the lead-list and supplier
|
||||
// UPC pipelines. Multiple rows for the same ASIN across different runs is fine.
|
||||
export const analysisRunStats = pgTable("analysis_run_stats", {
|
||||
runId: integer("run_id")
|
||||
.primaryKey()
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
processedCount: integer("processed_count").notNull().default(0),
|
||||
analyzedCount: integer("analyzed_count").notNull().default(0),
|
||||
availableCount: integer("available_count").notNull().default(0),
|
||||
fbaCount: integer("fba_count").notNull().default(0),
|
||||
fbmCount: integer("fbm_count").notNull().default(0),
|
||||
buyCount: integer("buy_count").notNull().default(0),
|
||||
watchCount: integer("watch_count").notNull().default(0),
|
||||
skipCount: integer("skip_count").notNull().default(0),
|
||||
});
|
||||
|
||||
export const analysisResults = pgTable(
|
||||
"analysis_results",
|
||||
export const categoryRunDetails = pgTable("category_run_details", {
|
||||
runId: integer("run_id")
|
||||
.primaryKey()
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
categoryId: integer("category_id").notNull(),
|
||||
categoryLabel: text("category_label").notNull(),
|
||||
checkedAsinCount: integer("checked_asin_count").notNull().default(0),
|
||||
selectionParametersJson: text("selection_parameters_json"),
|
||||
});
|
||||
|
||||
export const stalkerRunDetails = pgTable("stalker_run_details", {
|
||||
runId: integer("run_id")
|
||||
.primaryKey()
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
requestedAsins: integer("requested_asins").notNull().default(0),
|
||||
skippedAsins: integer("skipped_asins").notNull().default(0),
|
||||
scannedAsins: integer("scanned_asins").notNull().default(0),
|
||||
sourceAsinsWithMatches: integer("source_asins_with_matches")
|
||||
.notNull()
|
||||
.default(0),
|
||||
candidateSellers: integer("candidate_sellers").notNull().default(0),
|
||||
qualifyingSellers: integer("qualifying_sellers").notNull().default(0),
|
||||
matchedSellers: integer("matched_sellers").notNull().default(0),
|
||||
sellerMetadataRequests: integer("seller_metadata_requests")
|
||||
.notNull()
|
||||
.default(0),
|
||||
sellerStorefrontRequests: integer("seller_storefront_requests")
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityCheckedAsins: integer(
|
||||
"inventory_sellability_checked_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityAvailableAsins: integer(
|
||||
"inventory_sellability_available_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityExcludedAsins: integer(
|
||||
"inventory_sellability_excluded_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
persistedInventoryAsins: integer("persisted_inventory_asins")
|
||||
.notNull()
|
||||
.default(0),
|
||||
});
|
||||
|
||||
export const productIdentifiers = pgTable(
|
||||
"product_identifiers",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
runId: integer("run_id")
|
||||
productAsin: text("product_asin")
|
||||
.notNull()
|
||||
.references(() => runs.id),
|
||||
asin: text("asin").notNull(),
|
||||
// Product identity
|
||||
productName: text("product_name"),
|
||||
brand: text("brand"),
|
||||
category: text("category"),
|
||||
upc: text("upc"),
|
||||
// Supplier sheet data (lead_analysis only)
|
||||
unitCost: real("unit_cost"),
|
||||
avgPrice90dSheet: real("avg_price_90d_sheet"),
|
||||
sellingPriceSheet: real("selling_price_sheet"),
|
||||
fbaNetSheet: real("fba_net_sheet"),
|
||||
grossProfitDollar: real("gross_profit_dollar"),
|
||||
grossProfitPct: real("gross_profit_pct"),
|
||||
netProfitSheet: real("net_profit_sheet"),
|
||||
roiSheet: real("roi_sheet"),
|
||||
moq: integer("moq"),
|
||||
moqCost: real("moq_cost"),
|
||||
qtyAvailable: integer("qty_available"),
|
||||
supplier: text("supplier"),
|
||||
sourceUrl: text("source_url"),
|
||||
asinLink: text("asin_link"),
|
||||
promoCouponCode: text("promo_coupon_code"),
|
||||
notes: text("notes"),
|
||||
leadDate: text("lead_date"),
|
||||
// Market data
|
||||
currentPrice: real("current_price"),
|
||||
avgPrice90d: real("avg_price_90d"),
|
||||
salesRank: integer("sales_rank"),
|
||||
rankAvg90d: integer("rank_avg_90d"),
|
||||
monthlySold: integer("monthly_sold"),
|
||||
rankDrops30d: integer("rank_drops_30d"),
|
||||
rankDrops90d: integer("rank_drops_90d"),
|
||||
sellerCount: integer("seller_count"),
|
||||
amazonIsSeller: boolean("amazon_is_seller"),
|
||||
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
|
||||
// Fees
|
||||
fbaFee: real("fba_fee"),
|
||||
fbmFee: real("fbm_fee"),
|
||||
referralPercent: real("referral_percent"),
|
||||
// Sellability
|
||||
canSell: text("can_sell"),
|
||||
sellabilityStatus: text("sellability_status"),
|
||||
sellabilityReason: text("sellability_reason"),
|
||||
// Supplier-UPC scoring
|
||||
supplierScore: real("supplier_score"),
|
||||
supplierProfit: real("supplier_profit"),
|
||||
supplierMargin: real("supplier_margin"),
|
||||
supplierRoi: real("supplier_roi"),
|
||||
supplierReason: text("supplier_reason"),
|
||||
upcLookupStatus: text("upc_lookup_status"),
|
||||
upcLookupReason: text("upc_lookup_reason"),
|
||||
candidateAsins: text("candidate_asins"),
|
||||
// Verdict
|
||||
verdict: text("verdict").notNull(),
|
||||
confidence: real("confidence"),
|
||||
reasoning: text("reasoning"),
|
||||
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
|
||||
.references(() => products.asin),
|
||||
identifierType: text("identifier_type").notNull(),
|
||||
identifierValue: text("identifier_value").notNull(),
|
||||
source: text("source").notNull(),
|
||||
confirmedAt: timestamp("confirmed_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => [
|
||||
index("idx_analysis_results_run_id").on(t.runId),
|
||||
index("idx_analysis_results_asin").on(t.asin),
|
||||
index("idx_analysis_results_verdict").on(t.verdict),
|
||||
index("idx_analysis_results_sellability_status").on(t.sellabilityStatus),
|
||||
index("idx_analysis_results_fetched_at").on(t.fetchedAt),
|
||||
unique("uq_product_identifier_type_value").on(
|
||||
t.identifierType,
|
||||
t.identifierValue,
|
||||
),
|
||||
index("idx_product_identifiers_asin").on(t.productAsin),
|
||||
],
|
||||
);
|
||||
|
||||
// ─── Category product results ──────────────────────────────────────────────────
|
||||
// Latest-per-ASIN snapshot for the category pipelines (bestsellers, monthly-sold,
|
||||
// mid-range, stalker analysis). Upserted on conflict so each ASIN has one row.
|
||||
|
||||
export const categoryProductResults = pgTable(
|
||||
"category_product_results",
|
||||
export const productObservations = pgTable(
|
||||
"product_observations",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
asin: text("asin").notNull().unique(),
|
||||
productAsin: text("product_asin")
|
||||
.notNull()
|
||||
.references(() => products.asin),
|
||||
runId: integer("run_id")
|
||||
.notNull()
|
||||
.references(() => runs.id),
|
||||
name: text("name").notNull(),
|
||||
brand: text("brand"),
|
||||
category: text("category"),
|
||||
unitCost: real("unit_cost"),
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
source: text("source").notNull(),
|
||||
marketplace: text("marketplace").notNull().default("US"),
|
||||
currentPrice: real("current_price"),
|
||||
avgPrice90d: real("avg_price_90d"),
|
||||
avgPrice90dSheet: real("avg_price_90d_sheet"),
|
||||
sellingPriceSheet: real("selling_price_sheet"),
|
||||
salesRank: integer("sales_rank"),
|
||||
salesRankAvg90d: integer("sales_rank_avg_90d"),
|
||||
sellerCount: integer("seller_count"),
|
||||
amazonIsSeller: boolean("amazon_is_seller"),
|
||||
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
|
||||
monthlySold: integer("monthly_sold"),
|
||||
rankDrops30d: integer("rank_drops_30d"),
|
||||
rankDrops90d: integer("rank_drops_90d"),
|
||||
sellerCount: integer("seller_count"),
|
||||
amazonIsSeller: boolean("amazon_is_seller"),
|
||||
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
|
||||
fbaFee: real("fba_fee"),
|
||||
fbmFee: real("fbm_fee"),
|
||||
referralPercent: real("referral_percent"),
|
||||
canSell: text("can_sell"),
|
||||
canSell: boolean("can_sell"),
|
||||
sellabilityStatus: text("sellability_status"),
|
||||
sellabilityReason: text("sellability_reason"),
|
||||
verdict: text("verdict").notNull(),
|
||||
confidence: real("confidence").notNull(),
|
||||
reasoning: text("reasoning"),
|
||||
rawProductJson: text("raw_product_json"),
|
||||
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
|
||||
},
|
||||
(t) => [
|
||||
index("idx_category_results_run_id").on(t.runId),
|
||||
index("idx_category_results_verdict").on(t.verdict),
|
||||
index("idx_category_results_sellability_status").on(t.sellabilityStatus),
|
||||
index("idx_category_results_fetched_at").on(t.fetchedAt),
|
||||
index("idx_product_observations_product_time").on(
|
||||
t.productAsin,
|
||||
t.fetchedAt.desc(),
|
||||
),
|
||||
index("idx_product_observations_run_id").on(t.runId),
|
||||
index("idx_product_observations_sellability").on(t.sellabilityStatus),
|
||||
],
|
||||
);
|
||||
|
||||
// ─── Stalker runs ─────────────────────────────────────────────────────────────
|
||||
|
||||
export const stalkerRuns = pgTable(
|
||||
"stalker_runs",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
inputFile: text("input_file").notNull(),
|
||||
startedAt: timestamp("started_at", { withTimezone: true }).notNull(),
|
||||
completedAt: timestamp("completed_at", { withTimezone: true }),
|
||||
requestedAsins: integer("requested_asins").notNull().default(0),
|
||||
skippedAsins: integer("skipped_asins").notNull().default(0),
|
||||
scannedAsins: integer("scanned_asins").notNull().default(0),
|
||||
sourceAsinsWithMatches: integer("source_asins_with_matches")
|
||||
.notNull()
|
||||
.default(0),
|
||||
candidateSellers: integer("candidate_sellers").notNull().default(0),
|
||||
qualifyingSellers: integer("qualifying_sellers").notNull().default(0),
|
||||
matchedSellers: integer("matched_sellers").notNull().default(0),
|
||||
sellerMetadataRequests: integer("seller_metadata_requests")
|
||||
.notNull()
|
||||
.default(0),
|
||||
sellerStorefrontRequests: integer("seller_storefront_requests")
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityCheckedAsins: integer(
|
||||
"inventory_sellability_checked_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityAvailableAsins: integer(
|
||||
"inventory_sellability_available_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
inventorySellabilityExcludedAsins: integer(
|
||||
"inventory_sellability_excluded_asins",
|
||||
)
|
||||
.notNull()
|
||||
.default(0),
|
||||
persistedInventoryAsins: integer("persisted_inventory_asins")
|
||||
.notNull()
|
||||
.default(0),
|
||||
status: text("status").notNull(),
|
||||
errorMessage: text("error_message"),
|
||||
},
|
||||
(t) => [index("idx_stalker_runs_started_at").on(t.startedAt)],
|
||||
);
|
||||
|
||||
// ─── Stalker ASIN scans ───────────────────────────────────────────────────────
|
||||
|
||||
export const stalkerAsinScans = pgTable(
|
||||
"stalker_asin_scans",
|
||||
export const runItems = pgTable(
|
||||
"run_items",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
runId: integer("run_id")
|
||||
.notNull()
|
||||
.references(() => stalkerRuns.id, { onDelete: "cascade" }),
|
||||
sourceAsin: text("source_asin").notNull(),
|
||||
title: text("title"),
|
||||
offerCount: integer("offer_count").notNull().default(0),
|
||||
candidateSellerCount: integer("candidate_seller_count")
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
productAsin: text("product_asin").references(() => products.asin),
|
||||
sourceInventoryItemId: integer("source_inventory_item_id").references(
|
||||
(): AnyPgColumn => stalkerInventoryItems.id,
|
||||
{ onDelete: "set null" },
|
||||
),
|
||||
ordinal: integer("ordinal"),
|
||||
sourceRow: integer("source_row"),
|
||||
status: text("status").notNull().default("completed"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.default(0),
|
||||
matchedSellerCount: integer("matched_seller_count").notNull().default(0),
|
||||
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
|
||||
rawProductJson: text("raw_product_json"),
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => [
|
||||
unique("uq_stalker_scans_run_asin").on(t.runId, t.sourceAsin),
|
||||
index("idx_stalker_scans_run_id").on(t.runId),
|
||||
index("idx_stalker_scans_source_asin").on(t.sourceAsin),
|
||||
index("idx_run_items_run_id").on(t.runId),
|
||||
index("idx_run_items_product_asin").on(t.productAsin),
|
||||
],
|
||||
);
|
||||
|
||||
// ─── Sellers ──────────────────────────────────────────────────────────────────
|
||||
// General seller registry (was stalker_sellers).
|
||||
export const sourcingInputs = pgTable("sourcing_inputs", {
|
||||
runItemId: integer("run_item_id")
|
||||
.primaryKey()
|
||||
.references(() => runItems.id, { onDelete: "cascade" }),
|
||||
suppliedName: text("supplied_name"),
|
||||
suppliedBrand: text("supplied_brand"),
|
||||
suppliedCategory: text("supplied_category"),
|
||||
unitCost: real("unit_cost"),
|
||||
avgPrice90dSheet: real("avg_price_90d_sheet"),
|
||||
sellingPriceSheet: real("selling_price_sheet"),
|
||||
fbaNetSheet: real("fba_net_sheet"),
|
||||
grossProfitDollar: real("gross_profit_dollar"),
|
||||
grossProfitPct: real("gross_profit_pct"),
|
||||
netProfitSheet: real("net_profit_sheet"),
|
||||
roiSheet: real("roi_sheet"),
|
||||
moq: integer("moq"),
|
||||
moqCost: real("moq_cost"),
|
||||
qtyAvailable: integer("qty_available"),
|
||||
supplier: text("supplier"),
|
||||
sourceUrl: text("source_url"),
|
||||
asinLink: text("asin_link"),
|
||||
promoCouponCode: text("promo_coupon_code"),
|
||||
notes: text("notes"),
|
||||
leadDate: text("lead_date"),
|
||||
});
|
||||
|
||||
export const upcResolutions = pgTable(
|
||||
"upc_resolutions",
|
||||
{
|
||||
runItemId: integer("run_item_id")
|
||||
.primaryKey()
|
||||
.references(() => runItems.id, { onDelete: "cascade" }),
|
||||
requestedUpc: text("requested_upc").notNull(),
|
||||
normalizedUpc: text("normalized_upc").notNull(),
|
||||
provider: text("provider").notNull(),
|
||||
status: text("status").notNull(),
|
||||
reason: text("reason"),
|
||||
resolvedProductAsin: text("resolved_product_asin").references(
|
||||
() => products.asin,
|
||||
),
|
||||
resolvedAt: timestamp("resolved_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => [index("idx_upc_resolutions_normalized_upc").on(t.normalizedUpc)],
|
||||
);
|
||||
|
||||
export const upcResolutionCandidates = pgTable(
|
||||
"upc_resolution_candidates",
|
||||
{
|
||||
runItemId: integer("run_item_id")
|
||||
.notNull()
|
||||
.references(() => upcResolutions.runItemId, { onDelete: "cascade" }),
|
||||
productAsin: text("product_asin")
|
||||
.notNull()
|
||||
.references(() => products.asin),
|
||||
},
|
||||
(t) => [
|
||||
primaryKey({ columns: [t.runItemId, t.productAsin] }),
|
||||
index("idx_upc_candidates_product_asin").on(t.productAsin),
|
||||
],
|
||||
);
|
||||
|
||||
export const analysisRevisions = pgTable(
|
||||
"analysis_revisions",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
runItemId: integer("run_item_id")
|
||||
.notNull()
|
||||
.references(() => runItems.id, { onDelete: "cascade" }),
|
||||
observationId: integer("observation_id").references(
|
||||
() => productObservations.id,
|
||||
{ onDelete: "set null" },
|
||||
),
|
||||
method: analysisMethodEnum("method").notNull(),
|
||||
decision: analysisDecisionEnum("decision").notNull(),
|
||||
confidence: real("confidence"),
|
||||
reasoning: text("reasoning"),
|
||||
analyzedAt: timestamp("analyzed_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => [
|
||||
index("idx_analysis_revisions_run_item_time").on(t.runItemId, t.analyzedAt),
|
||||
index("idx_analysis_revisions_decision").on(t.decision),
|
||||
],
|
||||
);
|
||||
|
||||
export const supplierScores = pgTable("supplier_scores", {
|
||||
revisionId: integer("revision_id")
|
||||
.primaryKey()
|
||||
.references(() => analysisRevisions.id, { onDelete: "cascade" }),
|
||||
score: real("score"),
|
||||
salePrice: real("sale_price"),
|
||||
fbaFee: real("fba_fee"),
|
||||
profit: real("profit"),
|
||||
margin: real("margin"),
|
||||
roi: real("roi"),
|
||||
reason: text("reason"),
|
||||
});
|
||||
|
||||
export const sellers = pgTable("sellers", {
|
||||
sellerId: text("seller_id").primaryKey(),
|
||||
@@ -276,15 +356,44 @@ export const sellers = pgTable("sellers", {
|
||||
rawSellerJson: text("raw_seller_json"),
|
||||
});
|
||||
|
||||
// ─── Stalker ASIN sellers ─────────────────────────────────────────────────────
|
||||
export const stalkerScans = pgTable(
|
||||
"stalker_scans",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
runId: integer("run_id")
|
||||
.notNull()
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
sourceProductAsin: text("source_product_asin")
|
||||
.notNull()
|
||||
.references(() => products.asin),
|
||||
observationId: integer("observation_id").references(
|
||||
() => productObservations.id,
|
||||
{ onDelete: "set null" },
|
||||
),
|
||||
offerCount: integer("offer_count").notNull().default(0),
|
||||
candidateSellerCount: integer("candidate_seller_count")
|
||||
.notNull()
|
||||
.default(0),
|
||||
matchedSellerCount: integer("matched_seller_count").notNull().default(0),
|
||||
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
|
||||
},
|
||||
(t) => [
|
||||
unique("uq_stalker_scans_run_source_product").on(
|
||||
t.runId,
|
||||
t.sourceProductAsin,
|
||||
),
|
||||
index("idx_stalker_scans_run_id").on(t.runId),
|
||||
index("idx_stalker_scans_source_asin").on(t.sourceProductAsin),
|
||||
],
|
||||
);
|
||||
|
||||
export const stalkerAsinSellers = pgTable(
|
||||
"stalker_asin_sellers",
|
||||
export const stalkerScanSellers = pgTable(
|
||||
"stalker_scan_sellers",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
scanId: integer("scan_id")
|
||||
.notNull()
|
||||
.references(() => stalkerAsinScans.id, { onDelete: "cascade" }),
|
||||
.references(() => stalkerScans.id, { onDelete: "cascade" }),
|
||||
sellerId: text("seller_id")
|
||||
.notNull()
|
||||
.references(() => sellers.sellerId),
|
||||
@@ -297,47 +406,36 @@ export const stalkerAsinSellers = pgTable(
|
||||
rawOfferJson: text("raw_offer_json"),
|
||||
},
|
||||
(t) => [
|
||||
unique("uq_stalker_asin_sellers_scan_seller").on(t.scanId, t.sellerId),
|
||||
unique("uq_stalker_scan_sellers_scan_seller").on(t.scanId, t.sellerId),
|
||||
],
|
||||
);
|
||||
|
||||
// ─── Stalker seller inventory ─────────────────────────────────────────────────
|
||||
|
||||
export const stalkerSellerInventory = pgTable(
|
||||
"stalker_seller_inventory",
|
||||
export const stalkerInventoryItems = pgTable(
|
||||
"stalker_inventory_items",
|
||||
{
|
||||
id: serial("id").primaryKey(),
|
||||
runId: integer("run_id")
|
||||
.notNull()
|
||||
.references(() => stalkerRuns.id, { onDelete: "cascade" }),
|
||||
.references(() => runs.id, { onDelete: "cascade" }),
|
||||
sellerId: text("seller_id")
|
||||
.notNull()
|
||||
.references(() => sellers.sellerId),
|
||||
asin: text("asin").notNull(),
|
||||
canSell: boolean("can_sell"),
|
||||
sellabilityStatus: text("sellability_status"),
|
||||
sellabilityReason: text("sellability_reason"),
|
||||
productTitle: text("product_title"),
|
||||
brand: text("brand"),
|
||||
categoryTree: text("category_tree"),
|
||||
currentPrice: real("current_price"),
|
||||
avgPrice90d: real("avg_price_90d"),
|
||||
salesRank: integer("sales_rank"),
|
||||
monthlySold: integer("monthly_sold"),
|
||||
sellerCount: integer("seller_count"),
|
||||
amazonIsSeller: boolean("amazon_is_seller"),
|
||||
rawProductJson: text("raw_product_json"),
|
||||
productAsin: text("product_asin")
|
||||
.notNull()
|
||||
.references(() => products.asin),
|
||||
observationId: integer("observation_id")
|
||||
.notNull()
|
||||
.references(() => productObservations.id, { onDelete: "cascade" }),
|
||||
lastSeenAt: timestamp("last_seen_at", { withTimezone: true }).notNull(),
|
||||
rawInventoryJson: text("raw_inventory_json"),
|
||||
},
|
||||
(t) => [
|
||||
unique("uq_stalker_inventory_run_seller_asin").on(
|
||||
unique("uq_stalker_inventory_items_run_seller_asin").on(
|
||||
t.runId,
|
||||
t.sellerId,
|
||||
t.asin,
|
||||
t.productAsin,
|
||||
),
|
||||
index("idx_stalker_inventory_seller_id").on(t.sellerId),
|
||||
index("idx_stalker_inventory_asin").on(t.asin),
|
||||
index("idx_stalker_inventory_product_title").on(t.productTitle),
|
||||
index("idx_stalker_inventory_product_asin").on(t.productAsin),
|
||||
],
|
||||
);
|
||||
|
||||
@@ -42,7 +42,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
|
||||
JSON.stringify({
|
||||
products: [
|
||||
{
|
||||
asin: "B000FOUND01",
|
||||
asin: "B000FND001",
|
||||
upcList: ["012345678901"],
|
||||
stats: {
|
||||
current: [null, null, null, 1234],
|
||||
@@ -51,7 +51,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
|
||||
csv: [[5000000, 2999, 5000100]],
|
||||
},
|
||||
{
|
||||
asin: "B000MULTI01",
|
||||
asin: "B000MUL001",
|
||||
upcList: ["098765432109"],
|
||||
stats: {
|
||||
current: [null, null, null, 2000],
|
||||
@@ -60,7 +60,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
|
||||
csv: [[1, 1999]],
|
||||
},
|
||||
{
|
||||
asin: "B000MULTI02",
|
||||
asin: "B000MUL002",
|
||||
upcList: ["098765432109"],
|
||||
stats: {
|
||||
current: [null, null, null, 2100],
|
||||
@@ -83,14 +83,14 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
|
||||
]);
|
||||
|
||||
expect(details.get("012345678901")?.status).toBe("found");
|
||||
expect(details.get("012345678901")?.asin).toBe("B000FOUND01");
|
||||
expect(details.get("012345678901")?.asin).toBe("B000FND001");
|
||||
expect(details.get("012345678901")?.keepaData?.currentPrice).toBe(29.99);
|
||||
expect(details.get("012345678901")?.keepaData?.currentPrice).toBe(29.99);
|
||||
|
||||
expect(details.get("098765432109")?.status).toBe("multiple_asins");
|
||||
expect(details.get("098765432109")?.candidateAsins).toEqual([
|
||||
"B000MULTI01",
|
||||
"B000MULTI02",
|
||||
"B000MUL001",
|
||||
"B000MUL002",
|
||||
]);
|
||||
|
||||
expect(details.get("111111111111")?.status).toBe("not_found");
|
||||
@@ -100,7 +100,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
|
||||
"098765432109",
|
||||
"111111111111",
|
||||
]);
|
||||
expect(simpleMap.get("012345678901")).toBe("B000FOUND01");
|
||||
expect(simpleMap.get("012345678901")).toBe("B000FND001");
|
||||
expect(simpleMap.has("098765432109")).toBe(false);
|
||||
expect(simpleMap.has("111111111111")).toBe(false);
|
||||
});
|
||||
@@ -128,7 +128,7 @@ test("lookupKeepaUpcs keeps partial success when one chunk fails", async () => {
|
||||
JSON.stringify({
|
||||
products: [
|
||||
{
|
||||
asin: "B000LAST001",
|
||||
asin: "B000LST001",
|
||||
upcList: [secondChunkUpc],
|
||||
stats: {
|
||||
current: [null, null, null, 1000],
|
||||
@@ -148,11 +148,11 @@ test("lookupKeepaUpcs keeps partial success when one chunk fails", async () => {
|
||||
|
||||
expect(details.get(firstChunkFirstUpc)?.status).toBe("request_failed");
|
||||
expect(details.get(secondChunkUpc)?.status).toBe("found");
|
||||
expect(details.get(secondChunkUpc)?.asin).toBe("B000LAST001");
|
||||
expect(details.get(secondChunkUpc)?.asin).toBe("B000LST001");
|
||||
|
||||
const simpleMap = await mapUpcsToAsins(upcs);
|
||||
expect(simpleMap.has(firstChunkFirstUpc)).toBe(false);
|
||||
expect(simpleMap.get(secondChunkUpc)).toBe("B000LAST001");
|
||||
expect(simpleMap.get(secondChunkUpc)).toBe("B000LST001");
|
||||
});
|
||||
|
||||
test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () => {
|
||||
@@ -175,7 +175,7 @@ test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () =
|
||||
JSON.stringify({
|
||||
products: [
|
||||
{
|
||||
asin: "B000RETRY01",
|
||||
asin: "B000RTY001",
|
||||
upcList: [targetUpc],
|
||||
stats: {
|
||||
current: [null, null, null, 1111],
|
||||
@@ -197,7 +197,7 @@ test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () =
|
||||
|
||||
expect(fetchMock.mock.calls.length).toBe(2);
|
||||
expect(details.get(targetUpc)?.status).toBe("found");
|
||||
expect(details.get(targetUpc)?.asin).toBe("B000RETRY01");
|
||||
expect(details.get(targetUpc)?.asin).toBe("B000RTY001");
|
||||
});
|
||||
|
||||
test("lookupKeepaUpcs uses lightweight query params for code mapping", async () => {
|
||||
@@ -220,7 +220,7 @@ test("lookupKeepaUpcs uses lightweight query params for code mapping", async ()
|
||||
JSON.stringify({
|
||||
products: [
|
||||
{
|
||||
asin: "B000LIGHT01",
|
||||
asin: "B000LGT001",
|
||||
upcList: [targetUpc],
|
||||
categoryTree: [{ name: "Test Category" }],
|
||||
},
|
||||
@@ -238,5 +238,5 @@ test("lookupKeepaUpcs uses lightweight query params for code mapping", async ()
|
||||
|
||||
expect(fetchMock.mock.calls.length).toBe(1);
|
||||
expect(details.get(targetUpc)?.status).toBe("found");
|
||||
expect(details.get(targetUpc)?.asin).toBe("B000LIGHT01");
|
||||
expect(details.get(targetUpc)?.asin).toBe("B000LGT001");
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { config } from "../config.ts";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import type { KeepaData, KeepaUpcLookupDetail } from "../types.ts";
|
||||
|
||||
const KEEPA_BASE = "https://api.keepa.com";
|
||||
@@ -228,10 +229,17 @@ export async function fetchKeepaDataBatch(
|
||||
asins: string[],
|
||||
): Promise<Map<string, KeepaData>> {
|
||||
const results = new Map<string, KeepaData>();
|
||||
const canonicalAsins = Array.from(
|
||||
new Set(
|
||||
asins
|
||||
.map((asin) => normalizeAsin(asin))
|
||||
.filter((asin): asin is string => asin !== null),
|
||||
),
|
||||
);
|
||||
|
||||
// Split into chunks of MAX_ASINS_PER_REQUEST
|
||||
for (let i = 0; i < asins.length; i += MAX_ASINS_PER_REQUEST) {
|
||||
const chunk = asins.slice(i, i + MAX_ASINS_PER_REQUEST);
|
||||
for (let i = 0; i < canonicalAsins.length; i += MAX_ASINS_PER_REQUEST) {
|
||||
const chunk = canonicalAsins.slice(i, i + MAX_ASINS_PER_REQUEST);
|
||||
const url = buildProductUrl("asin", chunk, {
|
||||
includeStats: true,
|
||||
includeBuybox: true,
|
||||
@@ -250,7 +258,7 @@ export async function fetchKeepaDataBatch(
|
||||
|
||||
if (data.products) {
|
||||
for (const product of data.products) {
|
||||
const asin = product.asin;
|
||||
const asin = normalizeAsin(product.asin);
|
||||
if (!asin) continue;
|
||||
results.set(asin, parseKeepaProduct(product));
|
||||
}
|
||||
@@ -309,7 +317,7 @@ export async function lookupKeepaUpcs(
|
||||
|
||||
const byUpc = new Map<string, Map<string, KeepaData>>();
|
||||
for (const product of data.products ?? []) {
|
||||
const asin = String(product.asin ?? "").trim();
|
||||
const asin = normalizeAsin(product.asin);
|
||||
if (!asin) continue;
|
||||
|
||||
const keepaData = parseKeepaProduct(product);
|
||||
|
||||
@@ -13,6 +13,7 @@ afterAll(() => {
|
||||
|
||||
test("normalizeAsin uppercases and validates ASINs", () => {
|
||||
expect(normalizeAsin(" b07sn9bhvv ")).toBe("B07SN9BHVV");
|
||||
expect(normalizeAsin("0306406152")).toBe("0306406152");
|
||||
expect(() => normalizeAsin("not-an-asin")).toThrow("Invalid ASIN");
|
||||
});
|
||||
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
import { normalizeAsin as normalizeCanonicalAsin } from "../asin.ts";
|
||||
|
||||
const DEFAULT_SEARXNG_URL = "https://searxng.nvictor.me/";
|
||||
const DEFAULT_GOOGLE_CUSTOM_SEARCH_URL =
|
||||
"https://www.googleapis.com/customsearch/v1";
|
||||
const DEFAULT_SERPAPI_URL = "https://serpapi.com/search.json";
|
||||
const DEFAULT_TIMEOUT_MS = 10_000;
|
||||
const DEFAULT_MAX_RESULTS = 10;
|
||||
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
|
||||
const ASIN_MATCH_REGEX = /\bB[0-9A-Z]{9}\b/gi;
|
||||
const PRICE_LABELS = [
|
||||
"selling price",
|
||||
@@ -127,16 +128,15 @@ export async function searchProductOffers(
|
||||
}
|
||||
|
||||
export function normalizeAsin(value: string): string {
|
||||
const asin = value.trim().toUpperCase();
|
||||
if (!ASIN_REGEX.test(asin)) {
|
||||
const asin = normalizeCanonicalAsin(value);
|
||||
if (!asin) {
|
||||
throw new Error(`Invalid ASIN: ${value}`);
|
||||
}
|
||||
return asin;
|
||||
}
|
||||
|
||||
function getAsinQuery(value: string): string | undefined {
|
||||
const normalized = value.trim().toUpperCase();
|
||||
return ASIN_REGEX.test(normalized) ? normalized : undefined;
|
||||
return normalizeCanonicalAsin(value) ?? undefined;
|
||||
}
|
||||
|
||||
async function fetchSearxngResults(
|
||||
|
||||
@@ -20,6 +20,15 @@ test("parseCatalogUpcLookupResponse marks no match", () => {
|
||||
expect(detail.asin).toBeNull();
|
||||
});
|
||||
|
||||
test("parseCatalogUpcLookupResponse ignores invalid ASIN identifiers", () => {
|
||||
const detail = parseCatalogUpcLookupResponse("012345678901", {
|
||||
items: [{ asin: "012345678901" }],
|
||||
});
|
||||
|
||||
expect(detail.status).toBe("not_found");
|
||||
expect(detail.asin).toBeNull();
|
||||
});
|
||||
|
||||
test("parseCatalogUpcLookupResponse marks multiple ASINs", () => {
|
||||
const detail = parseCatalogUpcLookupResponse("012345678901", {
|
||||
payload: {
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { SellingPartner } from "amazon-sp-api";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import { config } from "../config.ts";
|
||||
import type {
|
||||
KeepaUpcLookupStatus,
|
||||
@@ -222,8 +223,7 @@ function extractCatalogAsin(item: any): string | null {
|
||||
item?.identifiers?.marketplaceASIN?.asin ??
|
||||
item?.Identifiers?.MarketplaceASIN?.ASIN;
|
||||
if (typeof raw !== "string") return null;
|
||||
const asin = raw.trim().toUpperCase();
|
||||
return asin ? asin : null;
|
||||
return normalizeAsin(raw);
|
||||
}
|
||||
|
||||
export function parseCatalogUpcLookupResponse(
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
import * as XLSX from "xlsx";
|
||||
import { normalizeAsin } from "./asin.ts";
|
||||
import type { ProductRecord } from "./types.ts";
|
||||
|
||||
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
|
||||
|
||||
const COLUMN_CANDIDATES = {
|
||||
asin: ["asin"],
|
||||
name: ["name", "product name", "title", "product title"],
|
||||
@@ -133,11 +132,9 @@ function getKnownColumns(columns: ColumnMap): Set<string> {
|
||||
}
|
||||
|
||||
function parseAsin(value: unknown): string | undefined {
|
||||
const asin = String(value ?? "")
|
||||
.trim()
|
||||
.toUpperCase();
|
||||
if (!asin || !ASIN_REGEX.test(asin)) {
|
||||
console.warn(`Skipping invalid ASIN: "${asin}"`);
|
||||
const asin = normalizeAsin(value);
|
||||
if (!asin) {
|
||||
console.warn(`Skipping invalid ASIN: "${String(value ?? "").trim()}"`);
|
||||
return undefined;
|
||||
}
|
||||
return asin;
|
||||
|
||||
2621
src/server.ts
2621
src/server.ts
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,7 @@
|
||||
import { db } from "../db/index.ts";
|
||||
import { categoryProductResults, runs } from "../db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { persistLlmResults, refreshRunStats } from "../db/persistence.ts";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import { analyzeProducts } from "../integrations/llm.ts";
|
||||
import { fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
|
||||
import type {
|
||||
@@ -22,6 +23,7 @@ type Args = {
|
||||
};
|
||||
|
||||
type InventoryRow = {
|
||||
inventoryItemId: number;
|
||||
asin: string;
|
||||
productTitle: string | null;
|
||||
brand: string | null;
|
||||
@@ -49,8 +51,8 @@ function parseArgs(argv = process.argv.slice(2)): Args {
|
||||
const useClaude = argv.includes("--claude");
|
||||
const asins = (readFlagValue(argv, "--asins") ?? "")
|
||||
.split(",")
|
||||
.map((asin) => asin.trim().toUpperCase())
|
||||
.filter(Boolean);
|
||||
.map((asin) => normalizeAsin(asin))
|
||||
.filter((asin): asin is string => asin !== null);
|
||||
|
||||
if (!Number.isInteger(stalkerRunId) || stalkerRunId <= 0) {
|
||||
throw new Error("--stalker-run-id must be a positive integer");
|
||||
@@ -68,15 +70,7 @@ function wait(ms: number): Promise<void> {
|
||||
}
|
||||
|
||||
function parseCategoryTree(value: string | null): string[] {
|
||||
if (!value) return [];
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
return Array.isArray(parsed)
|
||||
? parsed.filter((item): item is string => typeof item === "string")
|
||||
: [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
return value ? value.split(" > ").filter(Boolean) : [];
|
||||
}
|
||||
|
||||
function toProductRecord(row: InventoryRow): ProductRecord {
|
||||
@@ -128,25 +122,29 @@ async function loadInventoryRows(
|
||||
): Promise<InventoryRow[]> {
|
||||
if (asins.length === 0) return [];
|
||||
return db.execute(
|
||||
sql<InventoryRow>`SELECT DISTINCT ON (asin)
|
||||
asin,
|
||||
product_title AS "productTitle",
|
||||
brand,
|
||||
category_tree AS "categoryTree",
|
||||
current_price AS "currentPrice",
|
||||
avg_price_90d AS "avgPrice90d",
|
||||
sales_rank AS "salesRank",
|
||||
monthly_sold AS "monthlySold",
|
||||
seller_count AS "sellerCount",
|
||||
amazon_is_seller AS "amazonIsSeller",
|
||||
can_sell AS "canSell",
|
||||
sellability_status AS "sellabilityStatus",
|
||||
sellability_reason AS "sellabilityReason"
|
||||
FROM stalker_seller_inventory
|
||||
WHERE run_id = ${stalkerRunId}
|
||||
AND can_sell = true
|
||||
AND sellability_status = 'available'
|
||||
AND asin = ANY(${asins})`,
|
||||
sql<InventoryRow>`SELECT DISTINCT ON (inventory.product_asin)
|
||||
inventory.id AS "inventoryItemId",
|
||||
inventory.product_asin AS asin,
|
||||
product.name AS "productTitle",
|
||||
product.brand,
|
||||
product.category AS "categoryTree",
|
||||
observation.current_price AS "currentPrice",
|
||||
observation.avg_price_90d AS "avgPrice90d",
|
||||
observation.sales_rank AS "salesRank",
|
||||
observation.monthly_sold AS "monthlySold",
|
||||
observation.seller_count AS "sellerCount",
|
||||
observation.amazon_is_seller AS "amazonIsSeller",
|
||||
observation.can_sell AS "canSell",
|
||||
observation.sellability_status AS "sellabilityStatus",
|
||||
observation.sellability_reason AS "sellabilityReason"
|
||||
FROM stalker_inventory_items inventory
|
||||
JOIN products product ON product.asin = inventory.product_asin
|
||||
JOIN product_observations observation ON observation.id = inventory.observation_id
|
||||
WHERE inventory.run_id = ${stalkerRunId}
|
||||
AND observation.can_sell = true
|
||||
AND observation.sellability_status = 'available'
|
||||
AND inventory.product_asin = ANY(${asins})
|
||||
ORDER BY inventory.product_asin, observation.fetched_at DESC`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -177,111 +175,18 @@ async function buildEnrichedProducts(
|
||||
async function insertProductAnalysisResults(
|
||||
runId: number,
|
||||
results: AnalysisResult[],
|
||||
sourceInventoryIds: Map<string, number>,
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((result) => {
|
||||
const keepa = result.product.keepa;
|
||||
const record = result.product.record;
|
||||
const spApi = result.product.spApi;
|
||||
const canSell =
|
||||
spApi.canSell == null ? "unknown" : spApi.canSell ? "yes" : "no";
|
||||
|
||||
return {
|
||||
asin: record.asin,
|
||||
runId,
|
||||
name: record.name,
|
||||
brand: record.brand ?? null,
|
||||
category: record.category ?? keepa?.categoryTree.join(" > ") ?? null,
|
||||
unitCost: record.unitCost ?? null,
|
||||
currentPrice: keepa?.currentPrice ?? spApi.estimatedSalePrice ?? null,
|
||||
avgPrice90d: keepa?.avgPrice90 ?? null,
|
||||
avgPrice90dSheet: record.avgPrice90FromSheet ?? null,
|
||||
sellingPriceSheet: record.sellingPriceFromSheet ?? null,
|
||||
salesRank: keepa?.salesRank ?? record.amazonRank ?? null,
|
||||
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
|
||||
sellerCount: keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
monthlySold: keepa?.monthlySold ?? null,
|
||||
rankDrops30d: keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: keepa?.salesRankDrops90 ?? null,
|
||||
fbaFee: spApi.fbaFee ?? null,
|
||||
fbmFee: spApi.fbmFee ?? null,
|
||||
referralPercent: spApi.referralFeePercent ?? null,
|
||||
canSell,
|
||||
sellabilityStatus: spApi.sellabilityStatus ?? null,
|
||||
sellabilityReason: spApi.sellabilityReason ?? null,
|
||||
verdict: result.verdict.verdict,
|
||||
confidence: result.verdict.confidence ?? 0,
|
||||
reasoning: result.verdict.reasoning ?? null,
|
||||
fetchedAt: new Date(result.product.fetchedAt),
|
||||
};
|
||||
await persistLlmResults(runId, results, {
|
||||
source: "stalker_analysis",
|
||||
metadataSource: "catalog",
|
||||
sourceInventoryIds,
|
||||
});
|
||||
|
||||
await db
|
||||
.insert(categoryProductResults)
|
||||
.values(rows)
|
||||
.onConflictDoUpdate({
|
||||
target: categoryProductResults.asin,
|
||||
set: {
|
||||
runId: sql`EXCLUDED.run_id`,
|
||||
name: sql`EXCLUDED.name`,
|
||||
brand: sql`EXCLUDED.brand`,
|
||||
category: sql`EXCLUDED.category`,
|
||||
unitCost: sql`EXCLUDED.unit_cost`,
|
||||
currentPrice: sql`EXCLUDED.current_price`,
|
||||
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
|
||||
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
|
||||
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
|
||||
salesRank: sql`EXCLUDED.sales_rank`,
|
||||
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
|
||||
sellerCount: sql`EXCLUDED.seller_count`,
|
||||
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
|
||||
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
|
||||
monthlySold: sql`EXCLUDED.monthly_sold`,
|
||||
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
|
||||
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
|
||||
fbaFee: sql`EXCLUDED.fba_fee`,
|
||||
fbmFee: sql`EXCLUDED.fbm_fee`,
|
||||
referralPercent: sql`EXCLUDED.referral_percent`,
|
||||
canSell: sql`EXCLUDED.can_sell`,
|
||||
sellabilityStatus: sql`EXCLUDED.sellability_status`,
|
||||
sellabilityReason: sql`EXCLUDED.sellability_reason`,
|
||||
verdict: sql`EXCLUDED.verdict`,
|
||||
confidence: sql`EXCLUDED.confidence`,
|
||||
reasoning: sql`EXCLUDED.reasoning`,
|
||||
fetchedAt: sql`EXCLUDED.fetched_at`,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function refreshAnalysisRun(runId: number): Promise<void> {
|
||||
const [stats] = await db.execute(
|
||||
sql<{
|
||||
total: string;
|
||||
fba: string | null;
|
||||
fbm: string | null;
|
||||
skip: string | null;
|
||||
}>`SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
|
||||
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
|
||||
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
|
||||
FROM category_product_results
|
||||
WHERE run_id = ${runId}`,
|
||||
);
|
||||
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
topAsinsChecked: Number(stats?.total ?? 0),
|
||||
availableAsins: Number(stats?.total ?? 0),
|
||||
fbaCount: Number(stats?.fba ?? 0),
|
||||
fbmCount: Number(stats?.fbm ?? 0),
|
||||
skipCount: Number(stats?.skip ?? 0),
|
||||
})
|
||||
.where(eq(runs.id, runId));
|
||||
await refreshRunStats(runId);
|
||||
}
|
||||
|
||||
async function analyzeInBatches(
|
||||
@@ -344,7 +249,14 @@ async function main(): Promise<void> {
|
||||
console.log(`Stalker analysis: analyzing ${rows.length} sellable ASIN(s).`);
|
||||
const enriched = await buildEnrichedProducts(rows);
|
||||
const results = await analyzeInBatches(enriched, args.useClaude);
|
||||
await insertProductAnalysisResults(args.analysisRunId, results);
|
||||
const sourceInventoryIds = new Map(
|
||||
rows.map((row) => [row.asin, row.inventoryItemId]),
|
||||
);
|
||||
await insertProductAnalysisResults(
|
||||
args.analysisRunId,
|
||||
results,
|
||||
sourceInventoryIds,
|
||||
);
|
||||
await refreshAnalysisRun(args.analysisRunId);
|
||||
}
|
||||
|
||||
|
||||
@@ -101,12 +101,17 @@ test("readAsinsFromXlsx extracts valid ASINs and deduplicates in order", () => {
|
||||
{ ASIN: "invalid" },
|
||||
{ ASIN: "B000000002" },
|
||||
{ ASIN: "B000000001" },
|
||||
{ ASIN: "0306406152" },
|
||||
{ ASIN: "" },
|
||||
]);
|
||||
XLSX.utils.book_append_sheet(workbook, sheet, "Input");
|
||||
XLSX.writeFile(workbook, filePath);
|
||||
|
||||
expect(readAsinsFromXlsx(filePath)).toEqual(["B000000001", "B000000002"]);
|
||||
expect(readAsinsFromXlsx(filePath)).toEqual([
|
||||
"B000000001",
|
||||
"B000000002",
|
||||
"0306406152",
|
||||
]);
|
||||
});
|
||||
|
||||
test("isQualifyingSeller accepts rating counts from 1 to 30 only", () => {
|
||||
|
||||
@@ -1,13 +1,17 @@
|
||||
import * as XLSX from "xlsx";
|
||||
import path from "node:path";
|
||||
import { normalizeAsin } from "../asin.ts";
|
||||
import { db } from "../db/index.ts";
|
||||
import { refreshRunStats, upsertProduct } from "../db/persistence.ts";
|
||||
import {
|
||||
analysisRunStats,
|
||||
productObservations,
|
||||
runs,
|
||||
stalkerRuns,
|
||||
stalkerAsinScans,
|
||||
stalkerRunDetails,
|
||||
stalkerScans,
|
||||
sellers,
|
||||
stalkerAsinSellers,
|
||||
stalkerSellerInventory,
|
||||
stalkerScanSellers,
|
||||
stalkerInventoryItems,
|
||||
} from "../db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { fetchSellabilityBatch } from "../integrations/sp-api.ts";
|
||||
@@ -16,7 +20,6 @@ import type { SellabilityInfo } from "../types.ts";
|
||||
const KEEPA_BASE = "https://api.keepa.com";
|
||||
const DOMAIN_US = "1";
|
||||
const AMAZON_US_SELLER_ID = "ATVPDKIKX0DER";
|
||||
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
|
||||
const DEFAULT_STOREFRONT_UPDATE_HOURS = 168;
|
||||
const DEFAULT_OFFER_LIMIT = 100;
|
||||
const DEFAULT_SELLER_LIMIT = 30;
|
||||
@@ -333,7 +336,7 @@ export async function runStalker(args: StalkerArgs, deps: StalkerDeps = {}): Pro
|
||||
: await startStalkerRun(args.input, resumeFilteredAsins.length);
|
||||
const analysisRunId =
|
||||
!args.dryRun && args.analyzeSellable
|
||||
? await startStalkerAnalysisRun(args.input)
|
||||
? await startStalkerAnalysisRun(args.input, runId!)
|
||||
: null;
|
||||
const stats: StalkerRunStats = {
|
||||
scannedAsins: 0,
|
||||
@@ -841,11 +844,12 @@ async function persistAsinResult(
|
||||
|
||||
await db.transaction(async (tx) => {
|
||||
const scanId = await upsertAsinScan(tx, runId, result, fetchedAt);
|
||||
const observationIds = new Map<string, number>();
|
||||
|
||||
for (const { seller, offer } of result.matchedSellers) {
|
||||
await upsertSeller(tx, seller, fetchedAt);
|
||||
await upsertAsinSeller(tx, scanId, seller, offer);
|
||||
await upsertSellerInventory(tx, runId, seller, fetchedAt);
|
||||
await upsertSellerInventory(tx, runId, seller, fetchedAt, observationIds);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -856,37 +860,58 @@ async function upsertAsinScan(
|
||||
result: StalkerAsinResult,
|
||||
fetchedAt: Date,
|
||||
): Promise<number> {
|
||||
await tx
|
||||
.insert(stalkerAsinScans)
|
||||
const sourceProductAsin = await upsertProduct(
|
||||
{
|
||||
asin: result.asin,
|
||||
name: result.title,
|
||||
metadataSource: "catalog",
|
||||
fetchedAt,
|
||||
},
|
||||
tx,
|
||||
);
|
||||
const [observation] = await tx
|
||||
.insert(productObservations)
|
||||
.values({
|
||||
productAsin: sourceProductAsin,
|
||||
runId,
|
||||
sourceAsin: result.asin,
|
||||
title: result.title,
|
||||
offerCount: result.offerCount,
|
||||
candidateSellerCount: result.candidateSellerCount,
|
||||
matchedSellerCount: result.matchedSellers.length,
|
||||
source: "stalker_scan",
|
||||
fetchedAt,
|
||||
rawProductJson: JSON.stringify(
|
||||
result.product ?? { error: result.error ?? null },
|
||||
),
|
||||
})
|
||||
.returning({ id: productObservations.id });
|
||||
if (!observation) {
|
||||
throw new Error(`Failed to insert stalker observation for ${result.asin}`);
|
||||
}
|
||||
|
||||
await tx
|
||||
.insert(stalkerScans)
|
||||
.values({
|
||||
runId,
|
||||
sourceProductAsin,
|
||||
observationId: observation.id,
|
||||
offerCount: result.offerCount,
|
||||
candidateSellerCount: result.candidateSellerCount,
|
||||
matchedSellerCount: result.matchedSellers.length,
|
||||
fetchedAt,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: [stalkerAsinScans.runId, stalkerAsinScans.sourceAsin],
|
||||
target: [stalkerScans.runId, stalkerScans.sourceProductAsin],
|
||||
set: {
|
||||
title: sql`EXCLUDED.title`,
|
||||
observationId: sql`EXCLUDED.observation_id`,
|
||||
offerCount: sql`EXCLUDED.offer_count`,
|
||||
candidateSellerCount: sql`EXCLUDED.candidate_seller_count`,
|
||||
matchedSellerCount: sql`EXCLUDED.matched_seller_count`,
|
||||
fetchedAt: sql`EXCLUDED.fetched_at`,
|
||||
rawProductJson: sql`EXCLUDED.raw_product_json`,
|
||||
},
|
||||
});
|
||||
|
||||
const [row] = await tx
|
||||
.select({ id: stalkerAsinScans.id })
|
||||
.from(stalkerAsinScans)
|
||||
.select({ id: stalkerScans.id })
|
||||
.from(stalkerScans)
|
||||
.where(
|
||||
sql`${stalkerAsinScans.runId} = ${runId} AND ${stalkerAsinScans.sourceAsin} = ${result.asin}`,
|
||||
sql`${stalkerScans.runId} = ${runId} AND ${stalkerScans.sourceProductAsin} = ${sourceProductAsin}`,
|
||||
);
|
||||
if (!row)
|
||||
throw new Error(`Failed to load stalker scan row for ${result.asin}`);
|
||||
@@ -931,7 +956,7 @@ async function upsertAsinSeller(
|
||||
offer: StalkerOffer,
|
||||
): Promise<void> {
|
||||
await tx
|
||||
.insert(stalkerAsinSellers)
|
||||
.insert(stalkerScanSellers)
|
||||
.values({
|
||||
scanId,
|
||||
sellerId: seller.sellerId,
|
||||
@@ -944,7 +969,7 @@ async function upsertAsinSeller(
|
||||
rawOfferJson: JSON.stringify(offer.rawOffer),
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: [stalkerAsinSellers.scanId, stalkerAsinSellers.sellerId],
|
||||
target: [stalkerScanSellers.scanId, stalkerScanSellers.sellerId],
|
||||
set: {
|
||||
offerPrice: sql`EXCLUDED.offer_price`,
|
||||
condition: sql`EXCLUDED.condition`,
|
||||
@@ -962,6 +987,7 @@ async function upsertSellerInventory(
|
||||
runId: number,
|
||||
seller: StalkerSeller,
|
||||
fetchedAt: Date,
|
||||
observationIds: Map<string, number>,
|
||||
): Promise<void> {
|
||||
const items = seller.storefrontItems.filter(
|
||||
(item) =>
|
||||
@@ -971,58 +997,71 @@ async function upsertSellerInventory(
|
||||
|
||||
if (items.length === 0) return;
|
||||
|
||||
await tx
|
||||
.insert(stalkerSellerInventory)
|
||||
.values(
|
||||
items.map((item) => ({
|
||||
for (const item of items) {
|
||||
let observationId = observationIds.get(item.asin);
|
||||
if (observationId == null) {
|
||||
const productAsin = await upsertProduct(
|
||||
{
|
||||
asin: item.asin,
|
||||
name: item.productDetails?.title,
|
||||
brand: item.productDetails?.brand,
|
||||
category: item.productDetails?.categoryTree.join(" > "),
|
||||
metadataSource: "catalog",
|
||||
fetchedAt,
|
||||
},
|
||||
tx,
|
||||
);
|
||||
const [observation] = await tx
|
||||
.insert(productObservations)
|
||||
.values({
|
||||
productAsin,
|
||||
runId,
|
||||
source: "stalker_inventory",
|
||||
canSell: item.sellability?.canSell ?? null,
|
||||
sellabilityStatus: item.sellability?.sellabilityStatus ?? null,
|
||||
sellabilityReason: item.sellability?.sellabilityReason ?? null,
|
||||
currentPrice: item.productDetails?.currentPrice ?? null,
|
||||
avgPrice90d: item.productDetails?.avgPrice90 ?? null,
|
||||
salesRank: item.productDetails?.salesRank ?? null,
|
||||
monthlySold: item.productDetails?.monthlySold ?? null,
|
||||
sellerCount: item.productDetails?.sellerCount ?? null,
|
||||
amazonIsSeller: item.productDetails?.amazonIsSeller ?? null,
|
||||
rawProductJson: item.productDetails
|
||||
? JSON.stringify(item.productDetails.rawProduct)
|
||||
: null,
|
||||
fetchedAt,
|
||||
})
|
||||
.returning({ id: productObservations.id });
|
||||
if (!observation) {
|
||||
throw new Error(`Failed to insert inventory observation for ${item.asin}`);
|
||||
}
|
||||
observationId = observation.id;
|
||||
observationIds.set(item.asin, observationId);
|
||||
}
|
||||
|
||||
await tx
|
||||
.insert(stalkerInventoryItems)
|
||||
.values({
|
||||
runId,
|
||||
sellerId: seller.sellerId,
|
||||
asin: item.asin,
|
||||
canSell: item.sellability?.canSell ?? null,
|
||||
sellabilityStatus: item.sellability?.sellabilityStatus ?? null,
|
||||
sellabilityReason: item.sellability?.sellabilityReason ?? null,
|
||||
productTitle: item.productDetails?.title ?? null,
|
||||
brand: item.productDetails?.brand ?? null,
|
||||
categoryTree: item.productDetails
|
||||
? JSON.stringify(item.productDetails.categoryTree)
|
||||
: null,
|
||||
currentPrice: item.productDetails?.currentPrice ?? null,
|
||||
avgPrice90d: item.productDetails?.avgPrice90 ?? null,
|
||||
salesRank: item.productDetails?.salesRank ?? null,
|
||||
monthlySold: item.productDetails?.monthlySold ?? null,
|
||||
sellerCount: item.productDetails?.sellerCount ?? null,
|
||||
amazonIsSeller: item.productDetails?.amazonIsSeller ?? null,
|
||||
rawProductJson: item.productDetails
|
||||
? JSON.stringify(item.productDetails.rawProduct)
|
||||
: null,
|
||||
productAsin: item.asin,
|
||||
observationId,
|
||||
lastSeenAt: fetchedAt,
|
||||
rawInventoryJson: JSON.stringify(item.rawInventory),
|
||||
})),
|
||||
)
|
||||
.onConflictDoUpdate({
|
||||
target: [
|
||||
stalkerSellerInventory.runId,
|
||||
stalkerSellerInventory.sellerId,
|
||||
stalkerSellerInventory.asin,
|
||||
],
|
||||
set: {
|
||||
canSell: sql`EXCLUDED.can_sell`,
|
||||
sellabilityStatus: sql`EXCLUDED.sellability_status`,
|
||||
sellabilityReason: sql`EXCLUDED.sellability_reason`,
|
||||
productTitle: sql`EXCLUDED.product_title`,
|
||||
brand: sql`EXCLUDED.brand`,
|
||||
categoryTree: sql`EXCLUDED.category_tree`,
|
||||
currentPrice: sql`EXCLUDED.current_price`,
|
||||
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
|
||||
salesRank: sql`EXCLUDED.sales_rank`,
|
||||
monthlySold: sql`EXCLUDED.monthly_sold`,
|
||||
sellerCount: sql`EXCLUDED.seller_count`,
|
||||
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
|
||||
rawProductJson: sql`EXCLUDED.raw_product_json`,
|
||||
lastSeenAt: sql`EXCLUDED.last_seen_at`,
|
||||
rawInventoryJson: sql`EXCLUDED.raw_inventory_json`,
|
||||
},
|
||||
});
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: [
|
||||
stalkerInventoryItems.runId,
|
||||
stalkerInventoryItems.sellerId,
|
||||
stalkerInventoryItems.productAsin,
|
||||
],
|
||||
set: {
|
||||
observationId: sql`EXCLUDED.observation_id`,
|
||||
lastSeenAt: sql`EXCLUDED.last_seen_at`,
|
||||
rawInventoryJson: sql`EXCLUDED.raw_inventory_json`,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function startStalkerRun(
|
||||
@@ -1030,42 +1069,45 @@ async function startStalkerRun(
|
||||
totalAsins: number,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(stalkerRuns)
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "stalker",
|
||||
inputFile,
|
||||
startedAt: new Date(),
|
||||
requestedAsins: totalAsins,
|
||||
status: "running",
|
||||
})
|
||||
.returning({ id: stalkerRuns.id });
|
||||
.returning({ id: runs.id });
|
||||
if (!row) throw new Error("Failed to insert stalker run record.");
|
||||
await db.insert(stalkerRunDetails).values({
|
||||
runId: row.id,
|
||||
requestedAsins: totalAsins,
|
||||
});
|
||||
return row.id;
|
||||
}
|
||||
|
||||
async function startStalkerAnalysisRun(inputFile: string): Promise<number> {
|
||||
async function startStalkerAnalysisRun(
|
||||
inputFile: string,
|
||||
parentRunId: number,
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "category_analysis",
|
||||
categoryId: 0,
|
||||
categoryLabel: `Stalker: ${path.basename(inputFile)}`,
|
||||
topAsinsChecked: 0,
|
||||
availableAsins: 0,
|
||||
fbaCount: 0,
|
||||
fbmCount: 0,
|
||||
skipCount: 0,
|
||||
type: "stalker_analysis",
|
||||
parentRunId,
|
||||
inputFile: `Stalker: ${path.basename(inputFile)}`,
|
||||
status: "running",
|
||||
startedAt: new Date(),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
if (!row) throw new Error("Failed to insert stalker analysis run record.");
|
||||
await db.insert(analysisRunStats).values({ runId: row.id });
|
||||
return row.id;
|
||||
}
|
||||
|
||||
async function loadPreviouslyScannedAsins(): Promise<Set<string>> {
|
||||
const rows = await db
|
||||
.selectDistinct({ sourceAsin: stalkerAsinScans.sourceAsin })
|
||||
.from(stalkerAsinScans);
|
||||
.selectDistinct({ sourceAsin: stalkerScans.sourceProductAsin })
|
||||
.from(stalkerScans);
|
||||
return new Set(rows.map((row) => row.sourceAsin));
|
||||
}
|
||||
|
||||
@@ -1133,8 +1175,9 @@ async function refreshStalkerRun(
|
||||
status: string,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(stalkerRuns)
|
||||
.update(stalkerRunDetails)
|
||||
.set({
|
||||
skippedAsins: stats.skippedAsins,
|
||||
scannedAsins: stats.scannedAsins,
|
||||
sourceAsinsWithMatches: stats.sourceAsinsWithMatches,
|
||||
candidateSellers: stats.candidateSellers,
|
||||
@@ -1147,10 +1190,15 @@ async function refreshStalkerRun(
|
||||
stats.inventorySellabilityAvailableAsins,
|
||||
inventorySellabilityExcludedAsins: stats.inventorySellabilityExcludedAsins,
|
||||
persistedInventoryAsins: stats.persistedInventoryAsins,
|
||||
status,
|
||||
})
|
||||
.where(eq(stalkerRunDetails.runId, runId));
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
status: status === "running" ? "running" : "completed",
|
||||
...(status !== "running" ? { completedAt: new Date() } : {}),
|
||||
})
|
||||
.where(eq(stalkerRuns.id, runId));
|
||||
.where(eq(runs.id, runId));
|
||||
}
|
||||
|
||||
async function finishStalkerRunWithError(
|
||||
@@ -1159,8 +1207,9 @@ async function finishStalkerRunWithError(
|
||||
errorMessage: string,
|
||||
): Promise<void> {
|
||||
await db
|
||||
.update(stalkerRuns)
|
||||
.update(stalkerRunDetails)
|
||||
.set({
|
||||
skippedAsins: stats.skippedAsins,
|
||||
scannedAsins: stats.scannedAsins,
|
||||
sourceAsinsWithMatches: stats.sourceAsinsWithMatches,
|
||||
candidateSellers: stats.candidateSellers,
|
||||
@@ -1173,11 +1222,16 @@ async function finishStalkerRunWithError(
|
||||
stats.inventorySellabilityAvailableAsins,
|
||||
inventorySellabilityExcludedAsins: stats.inventorySellabilityExcludedAsins,
|
||||
persistedInventoryAsins: stats.persistedInventoryAsins,
|
||||
})
|
||||
.where(eq(stalkerRunDetails.runId, runId));
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
status: "failed",
|
||||
errorMessage,
|
||||
completedAt: new Date(),
|
||||
})
|
||||
.where(eq(stalkerRuns.id, runId));
|
||||
.where(eq(runs.id, runId));
|
||||
}
|
||||
|
||||
async function finishStalkerAnalysisRun(
|
||||
@@ -1185,29 +1239,10 @@ async function finishStalkerAnalysisRun(
|
||||
status: "completed" | "failed",
|
||||
errorMessage: string | null = null,
|
||||
): Promise<void> {
|
||||
const [stats] = await db.execute(
|
||||
sql<{
|
||||
total: string;
|
||||
fba: string | null;
|
||||
fbm: string | null;
|
||||
skip: string | null;
|
||||
}>`SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
|
||||
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
|
||||
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
|
||||
FROM category_product_results
|
||||
WHERE run_id = ${runId}`,
|
||||
);
|
||||
|
||||
await refreshRunStats(runId);
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
topAsinsChecked: Number(stats?.total ?? 0),
|
||||
availableAsins: Number(stats?.total ?? 0),
|
||||
fbaCount: Number(stats?.fba ?? 0),
|
||||
fbmCount: Number(stats?.fbm ?? 0),
|
||||
skipCount: Number(stats?.skip ?? 0),
|
||||
status,
|
||||
errorMessage,
|
||||
completedAt: new Date(),
|
||||
@@ -1480,13 +1515,6 @@ async function runSellableAnalysisChild(
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeAsin(value: unknown): string | null {
|
||||
const asin = String(value ?? "")
|
||||
.trim()
|
||||
.toUpperCase();
|
||||
return ASIN_REGEX.test(asin) ? asin : null;
|
||||
}
|
||||
|
||||
function normalizeSellerId(value: unknown): string | null {
|
||||
const sellerId = String(value ?? "")
|
||||
.trim()
|
||||
|
||||
@@ -16,12 +16,12 @@ function result(overrides: Partial<SupplierAnalysisResult> = {}): SupplierAnalys
|
||||
upc: "012345678901",
|
||||
rowNumber: 2,
|
||||
record: {
|
||||
asin: "B000000001",
|
||||
name: "Test Product",
|
||||
unitCost: 10,
|
||||
brand: "Brand",
|
||||
category: "Grocery",
|
||||
},
|
||||
product: { asin: "B000000001", name: "Test Product", unitCost: 10 },
|
||||
lookup: {
|
||||
requestedUpc: "012345678901",
|
||||
normalizedUpc: "012345678901",
|
||||
@@ -81,7 +81,8 @@ test("writeSupplierWorkbook writes ranked, skipped, and summary sheets", async (
|
||||
result(),
|
||||
result({
|
||||
upc: "111111111111",
|
||||
record: { asin: "111111111111", name: "Missing", unitCost: 0 },
|
||||
record: { name: "Missing", unitCost: 0 },
|
||||
product: null,
|
||||
lookup: {
|
||||
requestedUpc: "111111111111",
|
||||
normalizedUpc: "111111111111",
|
||||
|
||||
@@ -63,7 +63,8 @@ function addRowsSheet(
|
||||
const sheet = workbook.addWorksheet(name);
|
||||
const headers = rows[0] ? Object.keys(rows[0]) : Object.keys(rowForResult({
|
||||
upc: "",
|
||||
record: { asin: "", name: "", unitCost: 0 },
|
||||
record: { name: "", unitCost: 0 },
|
||||
product: null,
|
||||
lookup: {
|
||||
requestedUpc: "",
|
||||
normalizedUpc: "",
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import path from "node:path";
|
||||
import { requireAsin } from "../asin.ts";
|
||||
import { fetchKeepaDataBatch, lookupKeepaUpcs } from "../integrations/keepa.ts";
|
||||
import {
|
||||
fetchSellabilityBatch,
|
||||
@@ -11,6 +12,8 @@ import {
|
||||
} from "./upc-file-reader.ts";
|
||||
import {
|
||||
appendSupplierResultsToRun,
|
||||
completeRunInDb,
|
||||
failRunInDb,
|
||||
refreshRunCountsInDb,
|
||||
startRunInDb,
|
||||
type RunCounts,
|
||||
@@ -239,8 +242,8 @@ async function lookupUpcsWithChunking(
|
||||
chunkDetails.set(
|
||||
upc,
|
||||
fallbackDetail && fallbackDetail.status !== "request_failed"
|
||||
? fallbackDetail
|
||||
: spDetail!,
|
||||
? { ...fallbackDetail, provider: "keepa" }
|
||||
: { ...spDetail!, provider: "sp_api" },
|
||||
);
|
||||
}
|
||||
|
||||
@@ -266,7 +269,7 @@ function toProductRecord(
|
||||
const keepaCategory = detail.keepaData?.categoryTree?.[0];
|
||||
|
||||
return {
|
||||
asin: detail.asin ?? row.upc,
|
||||
asin: requireAsin(detail.asin),
|
||||
name: row.name ?? detail.asin ?? row.upc,
|
||||
unitCost: row.unitCost ?? 0,
|
||||
brand: row.brand,
|
||||
@@ -274,6 +277,15 @@ function toProductRecord(
|
||||
};
|
||||
}
|
||||
|
||||
function toSupplierInputRecord(row: UpcInputRow) {
|
||||
return {
|
||||
name: row.name ?? row.upc,
|
||||
unitCost: row.unitCost ?? 0,
|
||||
brand: row.brand,
|
||||
category: row.category,
|
||||
};
|
||||
}
|
||||
|
||||
async function fetchFeesForProducts(
|
||||
products: ProductRecord[],
|
||||
keepaResults: Map<string, NonNullable<SupplierAnalysisResult["keepa"]>>,
|
||||
@@ -359,7 +371,7 @@ export async function runUpcFileAnalysis(
|
||||
let processedRows = 0;
|
||||
let matchedRows = 0;
|
||||
|
||||
const runId = await startRunInDb(options.inputFile, outputFile);
|
||||
const runId = await startRunInDb(options.inputFile, outputFile, undefined, "supplier_upc");
|
||||
|
||||
try {
|
||||
const readerSummary = await processUpcFileInBatches(
|
||||
@@ -382,12 +394,19 @@ export async function runUpcFileAnalysis(
|
||||
product: ProductRecord;
|
||||
}> = [];
|
||||
for (const row of rows) {
|
||||
const detail = detailMap.get(row.upc);
|
||||
if (!detail) {
|
||||
unresolvedByStatus.request_failed += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const detail =
|
||||
detailMap.get(row.upc) ??
|
||||
({
|
||||
requestedUpc: row.upc,
|
||||
normalizedUpc: row.upc,
|
||||
status: "request_failed",
|
||||
asin: null,
|
||||
candidateAsins: [],
|
||||
keepaData: null,
|
||||
provider: "sp_api",
|
||||
reason: "UPC lookup returned no result",
|
||||
} satisfies UpcLookupDetail);
|
||||
if (!detailMap.has(row.upc)) detailMap.set(row.upc, detail);
|
||||
unresolvedByStatus[detail.status] += 1;
|
||||
|
||||
if (detail.status === "found" && detail.asin) {
|
||||
@@ -407,30 +426,15 @@ export async function runUpcFileAnalysis(
|
||||
|
||||
const batchResults: SupplierAnalysisResult[] = [];
|
||||
for (const row of rows) {
|
||||
const detail = detailMap.get(row.upc);
|
||||
if (!detail || detail.status === "found") continue;
|
||||
const detail = detailMap.get(row.upc)!;
|
||||
if (detail.status === "found") continue;
|
||||
|
||||
batchResults.push({
|
||||
upc: row.upc,
|
||||
rowNumber: row.rowNumber,
|
||||
record: {
|
||||
asin: detail?.asin ?? row.upc,
|
||||
name: row.name ?? row.upc,
|
||||
unitCost: row.unitCost ?? 0,
|
||||
brand: row.brand,
|
||||
category: row.category,
|
||||
},
|
||||
lookup:
|
||||
detail ??
|
||||
({
|
||||
requestedUpc: row.upc,
|
||||
normalizedUpc: row.upc,
|
||||
status: "request_failed",
|
||||
asin: null,
|
||||
candidateAsins: [],
|
||||
keepaData: null,
|
||||
reason: "UPC lookup returned no result",
|
||||
} satisfies UpcLookupDetail),
|
||||
record: toSupplierInputRecord(row),
|
||||
product: null,
|
||||
lookup: detail,
|
||||
keepa: null,
|
||||
spApi: null,
|
||||
score: skippedScore(detail?.reason ?? "UPC unresolved"),
|
||||
@@ -465,7 +469,8 @@ export async function runUpcFileAnalysis(
|
||||
batchResults.push({
|
||||
upc: entry.detail.normalizedUpc,
|
||||
rowNumber: entry.row.rowNumber,
|
||||
record: entry.product,
|
||||
record: toSupplierInputRecord(entry.row),
|
||||
product: entry.product,
|
||||
lookup: entry.detail,
|
||||
keepa,
|
||||
spApi,
|
||||
@@ -488,6 +493,7 @@ export async function runUpcFileAnalysis(
|
||||
|
||||
const exportSummary = summarizeSupplierResults(allResults, unresolvedByStatus);
|
||||
await writeSupplierWorkbook(outputFile, allResults, exportSummary);
|
||||
await completeRunInDb(runId);
|
||||
|
||||
if (allResults.length > 0) {
|
||||
const ranked = allResults
|
||||
@@ -530,6 +536,9 @@ export async function runUpcFileAnalysis(
|
||||
skippedInvalidUpc: readerSummary.skippedInvalidUpc,
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
await failRunInDb(runId, error);
|
||||
throw error;
|
||||
} finally {
|
||||
if (manageResources) {
|
||||
await disconnectCache();
|
||||
|
||||
56
src/types.ts
56
src/types.ts
@@ -59,6 +59,7 @@ export interface KeepaUpcLookupDetail {
|
||||
asin: string | null;
|
||||
candidateAsins: string[];
|
||||
keepaData: KeepaData | null;
|
||||
provider?: "sp_api" | "keepa";
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
@@ -114,7 +115,8 @@ export interface SupplierScore {
|
||||
export interface SupplierAnalysisResult {
|
||||
upc: string;
|
||||
rowNumber?: number;
|
||||
record: ProductRecord;
|
||||
record: SupplierInputRecord;
|
||||
product: ProductRecord | null;
|
||||
lookup: UpcLookupDetail;
|
||||
keepa: KeepaData | null;
|
||||
spApi: SpApiData | null;
|
||||
@@ -122,6 +124,58 @@ export interface SupplierAnalysisResult {
|
||||
fetchedAt: string;
|
||||
}
|
||||
|
||||
export interface SupplierInputRecord {
|
||||
name: string;
|
||||
unitCost: number;
|
||||
brand?: string;
|
||||
category?: string;
|
||||
}
|
||||
|
||||
export interface Product {
|
||||
asin: string;
|
||||
name: string | null;
|
||||
brand: string | null;
|
||||
category: string | null;
|
||||
firstSeenAt: string;
|
||||
lastSeenAt: string;
|
||||
}
|
||||
|
||||
export interface ProductObservation {
|
||||
id: number;
|
||||
productAsin: string;
|
||||
runId: number;
|
||||
source: string;
|
||||
fetchedAt: string;
|
||||
}
|
||||
|
||||
export interface Run {
|
||||
id: number;
|
||||
type:
|
||||
| "lead_analysis"
|
||||
| "category_analysis"
|
||||
| "supplier_upc"
|
||||
| "stalker"
|
||||
| "stalker_analysis";
|
||||
parentRunId?: number | null;
|
||||
status: string;
|
||||
}
|
||||
|
||||
export interface RunItem {
|
||||
id: number;
|
||||
runId: number;
|
||||
productAsin: string | null;
|
||||
sourceRow?: number | null;
|
||||
}
|
||||
|
||||
export interface AnalysisRevision {
|
||||
id: number;
|
||||
runItemId: number;
|
||||
decision: "FBA" | "FBM" | "BUY" | "WATCH" | "SKIP";
|
||||
confidence: number | null;
|
||||
reasoning: string | null;
|
||||
analyzedAt: string;
|
||||
}
|
||||
|
||||
export interface CategoryRunSummaryDb {
|
||||
categoryId: number;
|
||||
categoryLabel: string;
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
|
||||
type ProcessType = "lead_analysis" | "category_analysis";
|
||||
type ProcessType = "lead_analysis" | "category_analysis" | "supplier_upc" | "stalker" | "stalker_analysis";
|
||||
type AnalysisDecision = "FBA" | "FBM" | "BUY" | "WATCH" | "SKIP";
|
||||
type SortDirection = "ASC" | "DESC";
|
||||
|
||||
type Run = {
|
||||
@@ -15,6 +16,8 @@ type Run = {
|
||||
totalProducts: number;
|
||||
fbaCount: number;
|
||||
fbmCount: number;
|
||||
buyCount: number;
|
||||
watchCount: number;
|
||||
skipCount: number;
|
||||
};
|
||||
|
||||
@@ -37,11 +40,15 @@ type RunDetail = {
|
||||
totalProducts: number;
|
||||
fbaCount: number;
|
||||
fbmCount: number;
|
||||
buyCount: number;
|
||||
watchCount: number;
|
||||
skipCount: number;
|
||||
summary: {
|
||||
totalProducts: number;
|
||||
fbaCount: number;
|
||||
fbmCount: number;
|
||||
buyCount: number;
|
||||
watchCount: number;
|
||||
skipCount: number;
|
||||
};
|
||||
errorMessage?: string;
|
||||
@@ -49,6 +56,8 @@ type RunDetail = {
|
||||
};
|
||||
|
||||
type ResultItem = {
|
||||
item_id: number;
|
||||
product_asin: string | null;
|
||||
id?: number;
|
||||
run_id: number;
|
||||
asin: string;
|
||||
@@ -60,7 +69,7 @@ type ResultItem = {
|
||||
sales_rank: number | null;
|
||||
seller_count: number | null;
|
||||
monthly_sold: number | null;
|
||||
verdict: "FBA" | "FBM" | "SKIP";
|
||||
verdict: AnalysisDecision | null;
|
||||
amazon_is_seller: number | null;
|
||||
amazon_buybox_share_pct_90d: number | null;
|
||||
confidence: number | null;
|
||||
@@ -77,17 +86,18 @@ type ResultsResponse = {
|
||||
totalPages: number;
|
||||
};
|
||||
|
||||
type VerdictFilter = "" | "FBA" | "FBM" | "SKIP";
|
||||
type VerdictFilter = "" | AnalysisDecision;
|
||||
type AmazonSellerFilter = "" | "yes" | "no";
|
||||
|
||||
type ProductListItem = {
|
||||
processType: ProcessType;
|
||||
runId: number;
|
||||
item_id: number | null;
|
||||
processType: ProcessType | null;
|
||||
runId: number | null;
|
||||
asin: string;
|
||||
product_name: string | null;
|
||||
brand: string | null;
|
||||
category: string | null;
|
||||
verdict: "FBA" | "FBM" | "SKIP";
|
||||
verdict: AnalysisDecision | null;
|
||||
confidence: number | null;
|
||||
sellability_status: string | null;
|
||||
monthly_sold: number | null;
|
||||
@@ -98,7 +108,7 @@ type ProductListItem = {
|
||||
current_price: number | null;
|
||||
avg_price_90d: number | null;
|
||||
reasoning: string | null;
|
||||
fetched_at: string;
|
||||
fetched_at: string | null;
|
||||
};
|
||||
|
||||
type ProductListResponse = {
|
||||
@@ -179,6 +189,35 @@ type StalkerProductsResponse = {
|
||||
totalPages: number;
|
||||
};
|
||||
|
||||
type ProductHistoryResponse = {
|
||||
product: {
|
||||
asin: string;
|
||||
name: string | null;
|
||||
brand: string | null;
|
||||
category: string | null;
|
||||
first_seen_at: string;
|
||||
last_seen_at: string;
|
||||
};
|
||||
observations: Array<{
|
||||
id: number;
|
||||
run_id: number;
|
||||
source: string;
|
||||
current_price: number | null;
|
||||
monthly_sold: number | null;
|
||||
sales_rank: number | null;
|
||||
sellability_status: string | null;
|
||||
fetched_at: string;
|
||||
}>;
|
||||
analyses: Array<{
|
||||
id: number;
|
||||
run_id: number;
|
||||
decision: string;
|
||||
confidence: number | null;
|
||||
reasoning: string | null;
|
||||
analyzed_at: string;
|
||||
}>;
|
||||
};
|
||||
|
||||
type SortState = {
|
||||
field: string;
|
||||
direction: SortDirection;
|
||||
@@ -362,7 +401,7 @@ function Dashboard({
|
||||
|
||||
setDeletingKey(key);
|
||||
try {
|
||||
const response = await fetch(`/api/runs/${run.processType}/${run.runId}`, { method: "DELETE" });
|
||||
const response = await fetch(`/api/runs/${run.runId}`, { method: "DELETE" });
|
||||
if (!response.ok) {
|
||||
const errorPayload = await response.json().catch(() => null) as { error?: string } | null;
|
||||
const message = errorPayload?.error ?? "Failed to delete run";
|
||||
@@ -545,7 +584,7 @@ function RunDetails({
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
async function loadRun() {
|
||||
const res = await fetch(`/api/runs/${processType}/${runId}`);
|
||||
const res = await fetch(`/api/runs/${runId}`);
|
||||
const payload = (await res.json()) as RunDetail;
|
||||
if (!cancelled) {
|
||||
setRun(payload);
|
||||
@@ -573,7 +612,7 @@ function RunDetails({
|
||||
if (minConfidence) params.set("minConfidence", minConfidence);
|
||||
if (maxConfidence) params.set("maxConfidence", maxConfidence);
|
||||
|
||||
const res = await fetch(`/api/runs/${processType}/${runId}/results?${params.toString()}`);
|
||||
const res = await fetch(`/api/runs/${runId}/items?${params.toString()}`);
|
||||
const payload = (await res.json()) as ResultsResponse;
|
||||
if (!cancelled) {
|
||||
setResults(payload);
|
||||
@@ -596,12 +635,13 @@ function RunDetails({
|
||||
};
|
||||
}, [processType, runId]);
|
||||
|
||||
async function reanalyzeAsin(asin: string) {
|
||||
if (reanalyzing[asin]) return;
|
||||
setReanalyzing((prev) => ({ ...prev, [asin]: true }));
|
||||
async function reanalyzeItem(item: ResultItem) {
|
||||
const key = String(item.item_id);
|
||||
if (reanalyzing[key]) return;
|
||||
setReanalyzing((prev) => ({ ...prev, [key]: true }));
|
||||
try {
|
||||
const response = await fetch(
|
||||
`/api/runs/${processType}/${runId}/asins/${encodeURIComponent(asin)}/reanalyze`,
|
||||
`/api/run-items/${item.item_id}/reanalyze`,
|
||||
{ method: "POST" },
|
||||
);
|
||||
if (!response.ok) {
|
||||
@@ -613,7 +653,7 @@ function RunDetails({
|
||||
} finally {
|
||||
setReanalyzing((prev) => {
|
||||
const next = { ...prev };
|
||||
delete next[asin];
|
||||
delete next[key];
|
||||
return next;
|
||||
});
|
||||
}
|
||||
@@ -626,14 +666,14 @@ function RunDetails({
|
||||
<div className="card">
|
||||
<h2>Run Detail</h2>
|
||||
<div className="meta-grid" style={{ marginTop: 12 }}>
|
||||
<div className="meta"><strong>Process:</strong> {processType}</div>
|
||||
<div className="meta"><strong>Process:</strong> {run?.processType ?? processType}</div>
|
||||
<div className="meta"><strong>Run ID:</strong> {runId}</div>
|
||||
<div className="meta"><strong>Status:</strong> {run ? <span className={statusBadgeClass(run.status)}>{run.status}</span> : "-"}</div>
|
||||
<div className="meta"><strong>Timestamp:</strong> {run ? formatDate(run.timestamp) : "-"}</div>
|
||||
<div className="meta"><strong>Job:</strong> {run?.jobType ?? "-"}</div>
|
||||
<div className="meta"><strong>Source:</strong> {run?.source ?? "-"}</div>
|
||||
<div className="meta"><strong>Total:</strong> {formatNumber(run?.summary.totalProducts)}</div>
|
||||
<div className="meta"><strong>FBA/FBM/SKIP:</strong> {formatNumber(run?.summary.fbaCount)}/{formatNumber(run?.summary.fbmCount)}/{formatNumber(run?.summary.skipCount)}</div>
|
||||
<div className="meta"><strong>FBA/FBM/BUY/WATCH/SKIP:</strong> {formatNumber(run?.summary.fbaCount)}/{formatNumber(run?.summary.fbmCount)}/{formatNumber(run?.summary.buyCount)}/{formatNumber(run?.summary.watchCount)}/{formatNumber(run?.summary.skipCount)}</div>
|
||||
</div>
|
||||
<div style={{ marginTop: 10 }}>
|
||||
<TinyBar fba={run?.summary.fbaCount ?? 0} fbm={run?.summary.fbmCount ?? 0} skip={run?.summary.skipCount ?? 0} />
|
||||
@@ -647,6 +687,8 @@ function RunDetails({
|
||||
<option value="">All verdicts</option>
|
||||
<option value="FBA">FBA</option>
|
||||
<option value="FBM">FBM</option>
|
||||
<option value="BUY">BUY</option>
|
||||
<option value="WATCH">WATCH</option>
|
||||
<option value="SKIP">SKIP</option>
|
||||
</select>
|
||||
<select value={sellabilityStatus} onChange={(e) => { setPage(1); setSellabilityStatus(e.target.value); }}>
|
||||
@@ -677,7 +719,7 @@ function RunDetails({
|
||||
</div>
|
||||
<div style={{ marginTop: 10 }}>
|
||||
<a
|
||||
href={`/api/runs/${processType}/${runId}/export.csv?q=${encodeURIComponent(search)}&verdict=${encodeURIComponent(verdict)}&sellabilityStatus=${encodeURIComponent(sellabilityStatus)}&amazonIsSeller=${encodeURIComponent(amazonSellerFilter)}&minConfidence=${encodeURIComponent(minConfidence)}&maxConfidence=${encodeURIComponent(maxConfidence)}&sort=${encodeURIComponent(buildSortValue(sort))}`}
|
||||
href={`/api/runs/${runId}/export.csv?q=${encodeURIComponent(search)}&verdict=${encodeURIComponent(verdict)}&sellabilityStatus=${encodeURIComponent(sellabilityStatus)}&amazonIsSeller=${encodeURIComponent(amazonSellerFilter)}&minConfidence=${encodeURIComponent(minConfidence)}&maxConfidence=${encodeURIComponent(maxConfidence)}&sort=${encodeURIComponent(buildSortValue(sort))}`}
|
||||
>
|
||||
<button>Export filtered CSV</button>
|
||||
</a>
|
||||
@@ -692,8 +734,8 @@ function RunDetails({
|
||||
<div className="anomaly-list" style={{ marginTop: 8 }}>
|
||||
{anomalies.slice(0, 8).map((item) => (
|
||||
<div key={`anom-${item.asin}-${item.fetched_at}`} className="anomaly-item">
|
||||
<a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a>
|
||||
<span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span>
|
||||
{item.product_asin ? <a href={`/products/${item.product_asin}`}>{item.asin}</a> : item.asin}
|
||||
{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}
|
||||
<span>{detectAnomaly(item)}</span>
|
||||
</div>
|
||||
))}
|
||||
@@ -729,8 +771,8 @@ function RunDetails({
|
||||
) : results?.items.length ? (
|
||||
results.items.map((item) => (
|
||||
<tr key={`${item.asin}-${item.fetched_at}`}>
|
||||
<td><a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a></td>
|
||||
<td><span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span></td>
|
||||
<td>{item.product_asin ? <a href={`/products/${item.product_asin}`}>{item.asin}</a> : item.asin}</td>
|
||||
<td>{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}</td>
|
||||
<td>{formatNumber(item.monthly_sold)}</td>
|
||||
<td>{formatNumber(item.seller_count)}</td>
|
||||
<td>{formatAmazonSeller(item.amazon_is_seller)}</td>
|
||||
@@ -744,12 +786,14 @@ function RunDetails({
|
||||
<td>{formatNumber(item.confidence)}</td>
|
||||
<td className="reason-col" title={item.reasoning || undefined}>{item.reasoning || "-"}</td>
|
||||
<td>
|
||||
<button
|
||||
onClick={() => reanalyzeAsin(item.asin)}
|
||||
disabled={Boolean(reanalyzing[item.asin])}
|
||||
>
|
||||
{reanalyzing[item.asin] ? "Re-analyzing..." : "Re-analyze"}
|
||||
</button>
|
||||
{item.product_asin && run?.processType !== "supplier_upc" ? (
|
||||
<button
|
||||
onClick={() => reanalyzeItem(item)}
|
||||
disabled={Boolean(reanalyzing[String(item.item_id)])}
|
||||
>
|
||||
{reanalyzing[String(item.item_id)] ? "Re-analyzing..." : "Re-analyze"}
|
||||
</button>
|
||||
) : "-"}
|
||||
</td>
|
||||
</tr>
|
||||
))
|
||||
@@ -813,12 +857,13 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
|
||||
}, [search, activeVerdict, amazonSellerFilter, page, pageSize, sort]);
|
||||
|
||||
async function reanalyzeAsin(item: ProductListItem) {
|
||||
const key = `${item.processType}:${item.runId}:${item.asin}`;
|
||||
if (item.item_id == null) return;
|
||||
const key = String(item.item_id);
|
||||
if (reanalyzing[key]) return;
|
||||
setReanalyzing((prev) => ({ ...prev, [key]: true }));
|
||||
try {
|
||||
const response = await fetch(
|
||||
`/api/runs/${item.processType}/${item.runId}/asins/${encodeURIComponent(item.asin)}/reanalyze`,
|
||||
`/api/run-items/${item.item_id}/reanalyze`,
|
||||
{ method: "POST" },
|
||||
);
|
||||
if (!response.ok) {
|
||||
@@ -854,6 +899,8 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
|
||||
<option value="">All verdicts</option>
|
||||
<option value="FBA">FBA</option>
|
||||
<option value="FBM">FBM</option>
|
||||
<option value="BUY">BUY</option>
|
||||
<option value="WATCH">WATCH</option>
|
||||
<option value="SKIP">SKIP</option>
|
||||
</select>
|
||||
<select
|
||||
@@ -902,8 +949,8 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
|
||||
) : items?.items.length ? (
|
||||
items.items.map((item) => (
|
||||
<tr key={`${item.processType}-${item.runId}-${item.asin}-${item.fetched_at}`}>
|
||||
<td><a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a></td>
|
||||
<td><span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span></td>
|
||||
<td><a href={`/products/${item.asin}`}>{item.asin}</a></td>
|
||||
<td>{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}</td>
|
||||
<td>{formatNumber(item.monthly_sold)}</td>
|
||||
<td>{formatNumber(item.seller_count)}</td>
|
||||
<td>{formatAmazonSeller(item.amazon_is_seller)}</td>
|
||||
@@ -917,12 +964,14 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
|
||||
<td>{formatNumber(item.confidence)}</td>
|
||||
<td className="reason-col" title={item.reasoning || undefined}>{item.reasoning || "-"}</td>
|
||||
<td>
|
||||
<button
|
||||
onClick={() => reanalyzeAsin(item)}
|
||||
disabled={Boolean(reanalyzing[`${item.processType}:${item.runId}:${item.asin}`])}
|
||||
>
|
||||
{reanalyzing[`${item.processType}:${item.runId}:${item.asin}`] ? "Re-analyzing..." : "Re-analyze"}
|
||||
</button>
|
||||
{item.item_id == null || item.processType === "supplier_upc" ? "-" : (
|
||||
<button
|
||||
onClick={() => reanalyzeAsin(item)}
|
||||
disabled={Boolean(reanalyzing[String(item.item_id)])}
|
||||
>
|
||||
{reanalyzing[String(item.item_id)] ? "Re-analyzing..." : "Re-analyze"}
|
||||
</button>
|
||||
)}
|
||||
</td>
|
||||
</tr>
|
||||
))
|
||||
@@ -995,7 +1044,7 @@ function StalkerExplorer({
|
||||
}, [search, sellerId, runId, minRatingCount, maxRatingCount, page, pageSize, sort, refreshTick]);
|
||||
|
||||
async function purgeStalkerData() {
|
||||
const confirmed = window.confirm("Permanently delete all Stalker runs, sellers, and sellable products from the database?");
|
||||
const confirmed = window.confirm("Permanently delete all Stalker runs and unreferenced seller records? Canonical products are retained.");
|
||||
if (!confirmed) return;
|
||||
|
||||
setPurging(true);
|
||||
@@ -1384,17 +1433,95 @@ function StalkerProductsExplorer({
|
||||
);
|
||||
}
|
||||
|
||||
function ProductDetails({
|
||||
asin,
|
||||
onBack,
|
||||
}: {
|
||||
asin: string;
|
||||
onBack: () => void;
|
||||
}) {
|
||||
const [data, setData] = useState<ProductHistoryResponse | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
fetch(`/api/products/${encodeURIComponent(asin)}`)
|
||||
.then((response) => response.json())
|
||||
.then((payload: ProductHistoryResponse) => {
|
||||
if (!cancelled) setData(payload);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [asin]);
|
||||
|
||||
return (
|
||||
<div className="page">
|
||||
<button className="back" onClick={onBack}>Back</button>
|
||||
<div className="card">
|
||||
<h2>{data?.product.name ?? asin}</h2>
|
||||
<div className="meta-grid" style={{ marginTop: 12 }}>
|
||||
<div className="meta"><strong>ASIN:</strong> <a href={`https://amazon.com/dp/${asin}`} target="_blank" rel="noreferrer">{asin}</a></div>
|
||||
<div className="meta"><strong>Brand:</strong> {data?.product.brand ?? "-"}</div>
|
||||
<div className="meta"><strong>Category:</strong> {data?.product.category ?? "-"}</div>
|
||||
<div className="meta"><strong>Last seen:</strong> {data ? formatDate(data.product.last_seen_at) : "-"}</div>
|
||||
</div>
|
||||
</div>
|
||||
<div className="card">
|
||||
<h3>Analysis History</h3>
|
||||
<div className="table-wrap">
|
||||
<table>
|
||||
<thead><tr><th>Run</th><th>Decision</th><th>Confidence</th><th>Reasoning</th><th>Analyzed</th></tr></thead>
|
||||
<tbody>
|
||||
{data?.analyses.length ? data.analyses.map((analysis) => (
|
||||
<tr key={analysis.id}>
|
||||
<td>{analysis.run_id}</td>
|
||||
<td>{analysis.decision}</td>
|
||||
<td>{formatNumber(analysis.confidence)}</td>
|
||||
<td>{analysis.reasoning ?? "-"}</td>
|
||||
<td>{formatDate(analysis.analyzed_at)}</td>
|
||||
</tr>
|
||||
)) : <tr><td colSpan={5}>No analysis history</td></tr>}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
<div className="card">
|
||||
<h3>Observations</h3>
|
||||
<div className="table-wrap">
|
||||
<table>
|
||||
<thead><tr><th>Run</th><th>Source</th><th>Price</th><th>Monthly Sold</th><th>Sales Rank</th><th>Sellability</th><th>Fetched</th></tr></thead>
|
||||
<tbody>
|
||||
{data?.observations.length ? data.observations.map((observation) => (
|
||||
<tr key={observation.id}>
|
||||
<td>{observation.run_id}</td>
|
||||
<td>{observation.source}</td>
|
||||
<td>{formatCurrency(observation.current_price)}</td>
|
||||
<td>{formatNumber(observation.monthly_sold)}</td>
|
||||
<td>{formatNumber(observation.sales_rank)}</td>
|
||||
<td>{observation.sellability_status ?? "-"}</td>
|
||||
<td>{formatDate(observation.fetched_at)}</td>
|
||||
</tr>
|
||||
)) : <tr><td colSpan={7}>No observations</td></tr>}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
type AppRoute =
|
||||
| { kind: "dashboard" }
|
||||
| { kind: "run"; processType: ProcessType; runId: number }
|
||||
| { kind: "products"; verdict: VerdictFilter }
|
||||
| { kind: "product"; asin: string }
|
||||
| { kind: "stalker" }
|
||||
| { kind: "stalker-products" };
|
||||
|
||||
function parseRoute(pathname: string, search: string): AppRoute {
|
||||
const runMatch = pathname.match(/^\/runs\/(lead_analysis|category_analysis)\/(\d+)$/);
|
||||
const runMatch = pathname.match(/^\/runs\/(\d+)$/);
|
||||
if (runMatch) {
|
||||
return { kind: "run", processType: runMatch[1] as ProcessType, runId: Number(runMatch[2]) };
|
||||
return { kind: "run", processType: "lead_analysis", runId: Number(runMatch[1]) };
|
||||
}
|
||||
|
||||
if (pathname === "/products") {
|
||||
@@ -1404,6 +1531,11 @@ function parseRoute(pathname: string, search: string): AppRoute {
|
||||
return { kind: "products", verdict };
|
||||
}
|
||||
|
||||
const productMatch = pathname.match(/^\/products\/([A-Z0-9]{10})$/i);
|
||||
if (productMatch) {
|
||||
return { kind: "product", asin: productMatch[1]!.toUpperCase() };
|
||||
}
|
||||
|
||||
if (pathname === "/stalker") {
|
||||
return { kind: "stalker" };
|
||||
}
|
||||
@@ -1425,7 +1557,7 @@ function App() {
|
||||
}, []);
|
||||
|
||||
function openRun(run: Run) {
|
||||
const path = `/runs/${run.processType}/${run.runId}`;
|
||||
const path = `/runs/${run.runId}`;
|
||||
history.pushState({}, "", path);
|
||||
setRoute({ kind: "run", processType: run.processType, runId: run.runId });
|
||||
}
|
||||
@@ -1459,6 +1591,10 @@ function App() {
|
||||
return <ProductList verdict={route.verdict} onBack={backToDashboard} />;
|
||||
}
|
||||
|
||||
if (route.kind === "product") {
|
||||
return <ProductDetails asin={route.asin} onBack={backToDashboard} />;
|
||||
}
|
||||
|
||||
if (route.kind === "stalker") {
|
||||
return <StalkerExplorer onBack={backToDashboard} onOpenProducts={openStalkerProducts} />;
|
||||
}
|
||||
|
||||
201
src/writer.ts
201
src/writer.ts
@@ -1,6 +1,11 @@
|
||||
import { eq } from "drizzle-orm";
|
||||
import { db } from "./db/index.ts";
|
||||
import { runs, analysisResults } from "./db/schema.ts";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { analysisRunStats, runs } from "./db/schema.ts";
|
||||
import {
|
||||
persistLlmResults,
|
||||
persistSupplierResults,
|
||||
refreshRunStats,
|
||||
} from "./db/persistence.ts";
|
||||
import type { AnalysisResult, SupplierAnalysisResult } from "./types.ts";
|
||||
import { mkdirSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
@@ -13,18 +18,6 @@ export type RunCounts = {
|
||||
skipCount: number;
|
||||
};
|
||||
|
||||
function computeRunCountsFromResults(results: AnalysisResult[]): RunCounts {
|
||||
const fbaCount = results.filter((r) => r.verdict.verdict === "FBA").length;
|
||||
const fbmCount = results.filter((r) => r.verdict.verdict === "FBM").length;
|
||||
const skipCount = results.filter((r) => r.verdict.verdict === "SKIP").length;
|
||||
return {
|
||||
totalProducts: results.length,
|
||||
fbaCount,
|
||||
fbmCount,
|
||||
skipCount,
|
||||
};
|
||||
}
|
||||
|
||||
function buildRow(r: AnalysisResult) {
|
||||
const price =
|
||||
r.product.keepa?.currentPrice ??
|
||||
@@ -91,9 +84,15 @@ export async function writeResultsToDb(
|
||||
inputFile: string,
|
||||
outputFile: string | undefined,
|
||||
): Promise<void> {
|
||||
const runCounts = computeRunCountsFromResults(results);
|
||||
const runId = await startRunInDb(inputFile, outputFile, runCounts);
|
||||
await appendResultsToRun(runId, results);
|
||||
const runId = await startRunInDb(inputFile, outputFile);
|
||||
try {
|
||||
await appendResultsToRun(runId, results);
|
||||
await refreshRunCountsInDb(runId);
|
||||
await completeRunInDb(runId);
|
||||
} catch (error) {
|
||||
await failRunInDb(runId, error);
|
||||
throw error;
|
||||
}
|
||||
console.log(`Results written to database for run_id: ${runId}`);
|
||||
}
|
||||
|
||||
@@ -122,24 +121,28 @@ export async function startRunInDb(
|
||||
fbmCount: 0,
|
||||
skipCount: 0,
|
||||
},
|
||||
type: "lead_analysis" | "supplier_upc" = "lead_analysis",
|
||||
): Promise<number> {
|
||||
const [row] = await db
|
||||
.insert(runs)
|
||||
.values({
|
||||
type: "lead_analysis",
|
||||
type,
|
||||
inputFile,
|
||||
outputFile: outputFile ?? null,
|
||||
status: "ok",
|
||||
totalProducts: counts.totalProducts,
|
||||
fbaCount: counts.fbaCount,
|
||||
fbmCount: counts.fbmCount,
|
||||
skipCount: counts.skipCount,
|
||||
status: "running",
|
||||
startedAt: new Date(),
|
||||
completedAt: new Date(),
|
||||
})
|
||||
.returning({ id: runs.id });
|
||||
|
||||
if (!row) throw new Error("Failed to insert run record.");
|
||||
await db.insert(analysisRunStats).values({
|
||||
runId: row.id,
|
||||
processedCount: counts.totalProducts,
|
||||
analyzedCount: counts.totalProducts,
|
||||
fbaCount: counts.fbaCount,
|
||||
fbmCount: counts.fbmCount,
|
||||
skipCount: counts.skipCount,
|
||||
});
|
||||
return row.id;
|
||||
}
|
||||
|
||||
@@ -148,60 +151,11 @@ export async function appendResultsToRun(
|
||||
results: AnalysisResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((r) => {
|
||||
const row = buildRow(r);
|
||||
return {
|
||||
runId,
|
||||
asin: row.ASIN,
|
||||
productName: row.Name || null,
|
||||
brand: row.Brand || null,
|
||||
category: row.Category || null,
|
||||
unitCost: (row["Unit Cost"] as number) ?? null,
|
||||
currentPrice: (row["Current Price"] as number) || null,
|
||||
avgPrice90d: (row["Avg Price 90d"] as number) || null,
|
||||
avgPrice90dSheet: (row["Avg Price 90d (sheet)"] as number) || null,
|
||||
sellingPriceSheet: (row["Selling Price (sheet)"] as number) || null,
|
||||
salesRank: (row["Sales Rank"] as number) || null,
|
||||
rankAvg90d: (row["Rank Avg 90d"] as number) || null,
|
||||
sellerCount: (row.Sellers as number) || null,
|
||||
amazonIsSeller:
|
||||
row["Amazon Is Seller"] == null
|
||||
? null
|
||||
: Boolean(row["Amazon Is Seller"]),
|
||||
amazonBuyboxSharePct90d:
|
||||
(row["Amazon Buy Box Share 90d %"] as number) || null,
|
||||
monthlySold: (row["Monthly Sold"] as number) || null,
|
||||
rankDrops30d: (row["Rank Drops 30d"] as number) || null,
|
||||
rankDrops90d: (row["Rank Drops 90d"] as number) || null,
|
||||
fbaNetSheet: (row["FBA Net (sheet)"] as number) || null,
|
||||
grossProfitDollar: (row["Gross Profit $"] as number) || null,
|
||||
grossProfitPct: (row["Gross Profit %"] as number) || null,
|
||||
netProfitSheet: (row["Net Profit (sheet)"] as number) || null,
|
||||
roiSheet: (row["ROI (sheet)"] as number) || null,
|
||||
moq: (row.MOQ as number) || null,
|
||||
moqCost: (row["MOQ Cost"] as number) || null,
|
||||
qtyAvailable: (row["Qty Available"] as number) || null,
|
||||
supplier: row.Supplier || null,
|
||||
sourceUrl: row["Source URL"] || null,
|
||||
asinLink: row["ASIN Link"] || null,
|
||||
promoCouponCode: row["Promo/Coupon Code"] || null,
|
||||
notes: row.Notes || null,
|
||||
leadDate: row["Lead Date"] || null,
|
||||
fbaFee: row["FBA Fee"] ?? null,
|
||||
fbmFee: row["FBM Fee"] ?? null,
|
||||
referralPercent: row["Referral %"] ?? null,
|
||||
canSell: row["Can Sell"],
|
||||
sellabilityStatus: row.Sellability,
|
||||
sellabilityReason: row["Sellability Reason"] || null,
|
||||
verdict: row.Verdict,
|
||||
confidence: row.Confidence ?? null,
|
||||
reasoning: row.Reasoning,
|
||||
fetchedAt: new Date(r.product.fetchedAt),
|
||||
};
|
||||
await persistLlmResults(runId, results, {
|
||||
source: "lead_analysis",
|
||||
metadataSource: "input",
|
||||
preserveSourcingInput: true,
|
||||
});
|
||||
|
||||
await db.insert(analysisResults).values(rows);
|
||||
}
|
||||
|
||||
export async function appendSupplierResultsToRun(
|
||||
@@ -209,92 +163,29 @@ export async function appendSupplierResultsToRun(
|
||||
results: SupplierAnalysisResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
|
||||
const rows = results.map((result) => {
|
||||
const keepa = result.keepa;
|
||||
const spApi = result.spApi;
|
||||
const asin = result.lookup.asin ?? result.record.asin ?? result.upc;
|
||||
const category =
|
||||
result.record.category ?? keepa?.categoryTree?.join(" > ") ?? null;
|
||||
const canSell =
|
||||
spApi?.canSell == null ? null : spApi.canSell ? "yes" : "no";
|
||||
|
||||
return {
|
||||
runId,
|
||||
asin,
|
||||
productName: result.record.name,
|
||||
brand: result.record.brand ?? null,
|
||||
category,
|
||||
unitCost: result.record.unitCost || null,
|
||||
currentPrice: result.score.salePrice,
|
||||
avgPrice90d: keepa?.avgPrice90 ?? null,
|
||||
salesRank: keepa?.salesRank ?? null,
|
||||
rankAvg90d: keepa?.salesRankAvg90 ?? null,
|
||||
sellerCount: keepa?.sellerCount ?? null,
|
||||
amazonIsSeller: keepa?.amazonIsSeller ?? null,
|
||||
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
|
||||
monthlySold: keepa?.monthlySold ?? null,
|
||||
rankDrops30d: keepa?.salesRankDrops30 ?? null,
|
||||
rankDrops90d: keepa?.salesRankDrops90 ?? null,
|
||||
upc: result.upc,
|
||||
fbaFee: result.score.fbaFee,
|
||||
fbmFee: spApi?.fbmFee ?? null,
|
||||
referralPercent: spApi?.referralFeePercent ?? null,
|
||||
supplierScore: result.score.score,
|
||||
supplierProfit: result.score.profit,
|
||||
supplierMargin: result.score.margin,
|
||||
supplierRoi: result.score.roi,
|
||||
supplierReason: result.score.reason,
|
||||
upcLookupStatus: result.lookup.status,
|
||||
upcLookupReason: result.lookup.reason ?? null,
|
||||
candidateAsins: result.lookup.candidateAsins.join(","),
|
||||
canSell,
|
||||
sellabilityStatus: spApi?.sellabilityStatus ?? null,
|
||||
sellabilityReason: spApi?.sellabilityReason ?? null,
|
||||
verdict: result.score.verdict,
|
||||
confidence: result.score.score,
|
||||
reasoning: result.score.reason,
|
||||
fetchedAt: new Date(result.fetchedAt),
|
||||
};
|
||||
});
|
||||
|
||||
await db.insert(analysisResults).values(rows);
|
||||
await persistSupplierResults(runId, results);
|
||||
}
|
||||
|
||||
export async function refreshRunCountsInDb(runId: number): Promise<RunCounts> {
|
||||
const [stats] = await db.execute(
|
||||
sql<{
|
||||
total: string;
|
||||
fba: string | null;
|
||||
fbm: string | null;
|
||||
skip: string | null;
|
||||
}>`SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
|
||||
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
|
||||
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
|
||||
FROM analysis_results
|
||||
WHERE run_id = ${runId}`,
|
||||
);
|
||||
|
||||
const counts: RunCounts = {
|
||||
totalProducts: Number(stats?.total ?? 0),
|
||||
fbaCount: Number(stats?.fba ?? 0),
|
||||
fbmCount: Number(stats?.fbm ?? 0),
|
||||
skipCount: Number(stats?.skip ?? 0),
|
||||
};
|
||||
return refreshRunStats(runId);
|
||||
}
|
||||
|
||||
export async function completeRunInDb(runId: number): Promise<void> {
|
||||
await db
|
||||
.update(runs)
|
||||
.set({
|
||||
totalProducts: counts.totalProducts,
|
||||
fbaCount: counts.fbaCount,
|
||||
fbmCount: counts.fbmCount,
|
||||
skipCount: counts.skipCount,
|
||||
})
|
||||
.set({ status: "completed", completedAt: new Date(), errorMessage: null })
|
||||
.where(eq(runs.id, runId));
|
||||
}
|
||||
|
||||
return counts;
|
||||
export async function failRunInDb(
|
||||
runId: number,
|
||||
error: unknown,
|
||||
): Promise<void> {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
await db
|
||||
.update(runs)
|
||||
.set({ status: "failed", completedAt: new Date(), errorMessage })
|
||||
.where(eq(runs.id, runId));
|
||||
}
|
||||
|
||||
export function printResults(results: AnalysisResult[]): void {
|
||||
|
||||
Reference in New Issue
Block a user