Refactor supplier analysis and product handling

- Updated `SupplierAnalysisResult` to include a `product` field and modified related tests.
- Refactored `addRowsSheet` to accommodate changes in the product structure.
- Enhanced UPC file analysis to utilize a new `toSupplierInputRecord` function for cleaner record creation.
- Introduced new types for supplier input records and product observations.
- Updated frontend components to handle new product details and analysis history.
- Improved database writing functions to streamline run completion and error handling.
- Added new API endpoints for product details and adjusted routing in the frontend.
This commit is contained in:
Victor Noguera
2026-05-25 12:27:41 -04:00
parent c006d87c54
commit 923ebbaec5
33 changed files with 2536 additions and 4872 deletions

View File

@@ -19,4 +19,5 @@ GOOGLE_API_KEY=your_google_api_key
GOOGLE_CSE_ID=your_google_programmable_search_engine_id
SERPAPI_API_KEY=your_serpapi_api_key_for_google_shopping
DB_CONNECTION_STRING=your_database_connection_string
# Matches the default PostgreSQL service in docker-compose.yaml.
DB_CONNECTION_STRING=postgres://asin_check:asin_check@localhost:5432/asin_check

View File

@@ -90,6 +90,7 @@ Tracks competitor sellers across ASINs. Fetches storefronts, checks sellability
| `src/config.ts` | Env var loading via `Bun.env` |
| `src/db/index.ts` | Drizzle Postgres connection (shared pool) |
| `src/db/schema.ts` | Drizzle schema for all tables |
| `src/db/persistence.ts` | Product, observation, unified run-item, UPC resolution, and revision persistence |
| `src/integrations/keepa.ts` | Keepa API: batch ASIN fetch, UPC lookup, auto rate-limiting |
| `src/integrations/sp-api.ts` | SP-API: sellability, pricing+fees, UPC catalog lookup |
| `src/integrations/cache.ts` | Redis caching (24h TTL for lead-list; 12h for mid-range) |
@@ -112,4 +113,6 @@ Tracks competitor sellers across ASINs. Fetches storefronts, checks sellability
- The supplier UPC pipeline must not call LM Studio.
- Supplier UPC files resolve UPC/EAN through SP-API catalog lookup first; Keepa UPC lookup is fallback only (no-match or request-failure cases).
- Supplier workbook output must keep `Ranked Leads`, `Skipped`, and `Summary` sheets.
- Treat `products.asin` as the canonical normalized product identity; UPC values belong only in identifier and resolution records.
- Store time-varying data in observations or revisions and retain run history rather than overwriting prior analysis.
- When changing UPC supplier behavior, cover SP-API UPC parsing, deterministic scoring, and workbook export with `bun test`.

View File

@@ -155,7 +155,7 @@ ranked sourcing workbook:
2. Resolves UPCs to ASINs with SP-API catalog lookup first, then falls back to Keepa for no-match/request-failure cases.
3. Enriches resolved ASINs with Keepa demand/competition data and SP-API sellability + FBA fees.
4. Scores products with deterministic BUY/WATCH/SKIP logic; this path does not call LM Studio.
5. Writes a ranked Excel workbook and persists rows into the existing `runs` + `results` tables.
5. Writes a ranked Excel workbook and persists rows through unified runs, UPC resolution, product observation, and scoring-history tables.
CLI usage:
@@ -244,20 +244,28 @@ Numeric parsing accepts plain numbers as well as formatted values like `$12.50`,
4. **Keepa fetch** — batch the sellable (uncached) ASINs in a single API call (up to 100 per request)
5. **Enrich** — fetch SP-API pricing + FBA/FBM fees for sellable ASINs; combine with Keepa data and spreadsheet data
6. **LLM analysis** — send batches of 5 sellable products to LM Studio for FBA/FBM/SKIP verdict; skipped ASINs get auto-SKIP verdict (confidence 100) and bypass LLM entirely
7. **Output** — print results table to console (includes all ASINs), optionally write CSV/XLSX, and **persist results to a SQLite database**.
7. **Output** — print results table to console (includes all ASINs), optionally write CSV/XLSX, and persist products, observations, run items, and analysis revisions to PostgreSQL.
## Persistent Storage with SQLite
## Persistent Storage
Results from each run are now stored in a SQLite database named `db/results.db` by default. The SQLite implementation details are handled in `src/database.ts`. This allows you to:
PostgreSQL persistence is managed with Drizzle in `src/db/schema.ts` and `src/db/persistence.ts`. ASINs are canonical product identities: all inputs normalize to uppercase 10-character alphanumeric keys before any product reference is stored.
- Revisit past analysis results.
- Query and analyze historical data.
- Track product performance over time.
Core tables:
The database will automatically be created if it doesn't exist. Two tables are created:
- `products`: one canonical row per ASIN with latest descriptive metadata.
- `product_observations`: append-only marketplace, pricing, fee, and sellability snapshots.
- `runs` and `run_items`: unified lifecycle/history for lead, category, supplier UPC, and stalker workflows.
- `analysis_revisions` and `supplier_scores`: append-only analysis results; reanalysis does not overwrite prior decisions.
- `sourcing_inputs`, `upc_resolutions`, and `product_identifiers`: source-row and confirmed identifier data kept separate from catalog products.
- `stalker_run_details`, `stalker_scans`, and `stalker_inventory_items`: seller workflow provenance linked back to products and observations.
- `runs`: Stores metadata about each analysis run (timestamp, input file, output file, and summary counts).
- `results`: Stores detailed analysis results for each product from each run, linked to the `runs` table.
Unresolved or ambiguous supplier UPCs stay on their run item and resolution records; a UPC is never stored as an ASIN.
Web endpoints use unified identifiers:
- `GET /api/runs`, `GET /api/runs/:runId`, `GET /api/runs/:runId/items`
- `GET /api/products`, `GET /api/products/:asin`
- `POST /api/run-items/:itemId/reanalyze`
## Output columns

35
docker-compose.yaml Normal file
View File

@@ -0,0 +1,35 @@
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_DB: asin_check
POSTGRES_USER: asin_check
POSTGRES_PASSWORD: asin_check
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 10
redis:
image: redis:7-alpine
restart: unless-stopped
command: ["redis-server", "--appendonly", "yes"]
ports:
- "6379:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 10
volumes:
postgres_data:
redis_data:

View File

@@ -1,217 +0,0 @@
CREATE TYPE "public"."run_status" AS ENUM('running', 'ok', 'empty', 'failed', 'completed');--> statement-breakpoint
CREATE TYPE "public"."run_type" AS ENUM('lead_analysis', 'category_analysis', 'supplier_upc', 'stalker');--> statement-breakpoint
CREATE TABLE "analysis_results" (
"id" serial PRIMARY KEY NOT NULL,
"run_id" integer NOT NULL,
"asin" text NOT NULL,
"product_name" text,
"brand" text,
"category" text,
"upc" text,
"unit_cost" real,
"avg_price_90d_sheet" real,
"selling_price_sheet" real,
"fba_net_sheet" real,
"gross_profit_dollar" real,
"gross_profit_pct" real,
"net_profit_sheet" real,
"roi_sheet" real,
"moq" integer,
"moq_cost" real,
"qty_available" integer,
"supplier" text,
"source_url" text,
"asin_link" text,
"promo_coupon_code" text,
"notes" text,
"lead_date" text,
"current_price" real,
"avg_price_90d" real,
"sales_rank" integer,
"rank_avg_90d" integer,
"monthly_sold" integer,
"rank_drops_30d" integer,
"rank_drops_90d" integer,
"seller_count" integer,
"amazon_is_seller" boolean,
"amazon_buybox_share_pct_90d" real,
"fba_fee" real,
"fbm_fee" real,
"referral_percent" real,
"can_sell" text,
"sellability_status" text,
"sellability_reason" text,
"supplier_score" real,
"supplier_profit" real,
"supplier_margin" real,
"supplier_roi" real,
"supplier_reason" text,
"upc_lookup_status" text,
"upc_lookup_reason" text,
"candidate_asins" text,
"verdict" text NOT NULL,
"confidence" real,
"reasoning" text,
"fetched_at" timestamp with time zone NOT NULL
);
--> statement-breakpoint
CREATE TABLE "category_product_results" (
"id" serial PRIMARY KEY NOT NULL,
"asin" text NOT NULL,
"run_id" integer NOT NULL,
"name" text NOT NULL,
"brand" text,
"category" text,
"unit_cost" real,
"current_price" real,
"avg_price_90d" real,
"avg_price_90d_sheet" real,
"selling_price_sheet" real,
"sales_rank" integer,
"sales_rank_avg_90d" integer,
"seller_count" integer,
"amazon_is_seller" boolean,
"amazon_buybox_share_pct_90d" real,
"monthly_sold" integer,
"rank_drops_30d" integer,
"rank_drops_90d" integer,
"fba_fee" real,
"fbm_fee" real,
"referral_percent" real,
"can_sell" text,
"sellability_status" text,
"sellability_reason" text,
"verdict" text NOT NULL,
"confidence" real NOT NULL,
"reasoning" text,
"fetched_at" timestamp with time zone NOT NULL,
CONSTRAINT "category_product_results_asin_unique" UNIQUE("asin")
);
--> statement-breakpoint
CREATE TABLE "runs" (
"id" serial PRIMARY KEY NOT NULL,
"type" "run_type" NOT NULL,
"input_file" text,
"output_file" text,
"status" "run_status" DEFAULT 'running' NOT NULL,
"error_message" text,
"total_products" integer,
"fba_count" integer,
"fbm_count" integer,
"skip_count" integer,
"category_id" integer,
"category_label" text,
"top_asins_checked" integer,
"available_asins" integer,
"started_at" timestamp with time zone DEFAULT now() NOT NULL,
"completed_at" timestamp with time zone
);
--> statement-breakpoint
CREATE TABLE "sellers" (
"seller_id" text PRIMARY KEY NOT NULL,
"seller_name" text,
"rating" real,
"rating_count" integer,
"storefront_asin_total" integer,
"persisted_inventory_sample_count" integer,
"last_updated_at" timestamp with time zone NOT NULL,
"raw_seller_json" text
);
--> statement-breakpoint
CREATE TABLE "stalker_asin_scans" (
"id" serial PRIMARY KEY NOT NULL,
"run_id" integer NOT NULL,
"source_asin" text NOT NULL,
"title" text,
"offer_count" integer DEFAULT 0 NOT NULL,
"candidate_seller_count" integer DEFAULT 0 NOT NULL,
"matched_seller_count" integer DEFAULT 0 NOT NULL,
"fetched_at" timestamp with time zone NOT NULL,
"raw_product_json" text,
CONSTRAINT "uq_stalker_scans_run_asin" UNIQUE("run_id","source_asin")
);
--> statement-breakpoint
CREATE TABLE "stalker_asin_sellers" (
"id" serial PRIMARY KEY NOT NULL,
"scan_id" integer NOT NULL,
"seller_id" text NOT NULL,
"offer_price" real,
"condition" text,
"is_fba" boolean,
"stock" integer,
"seller_rating" real,
"seller_rating_count" integer,
"raw_offer_json" text,
CONSTRAINT "uq_stalker_asin_sellers_scan_seller" UNIQUE("scan_id","seller_id")
);
--> statement-breakpoint
CREATE TABLE "stalker_runs" (
"id" serial PRIMARY KEY NOT NULL,
"input_file" text NOT NULL,
"started_at" timestamp with time zone NOT NULL,
"completed_at" timestamp with time zone,
"requested_asins" integer DEFAULT 0 NOT NULL,
"skipped_asins" integer DEFAULT 0 NOT NULL,
"scanned_asins" integer DEFAULT 0 NOT NULL,
"source_asins_with_matches" integer DEFAULT 0 NOT NULL,
"candidate_sellers" integer DEFAULT 0 NOT NULL,
"qualifying_sellers" integer DEFAULT 0 NOT NULL,
"matched_sellers" integer DEFAULT 0 NOT NULL,
"seller_metadata_requests" integer DEFAULT 0 NOT NULL,
"seller_storefront_requests" integer DEFAULT 0 NOT NULL,
"inventory_sellability_checked_asins" integer DEFAULT 0 NOT NULL,
"inventory_sellability_available_asins" integer DEFAULT 0 NOT NULL,
"inventory_sellability_excluded_asins" integer DEFAULT 0 NOT NULL,
"persisted_inventory_asins" integer DEFAULT 0 NOT NULL,
"status" text NOT NULL,
"error_message" text
);
--> statement-breakpoint
CREATE TABLE "stalker_seller_inventory" (
"id" serial PRIMARY KEY NOT NULL,
"run_id" integer NOT NULL,
"seller_id" text NOT NULL,
"asin" text NOT NULL,
"can_sell" boolean,
"sellability_status" text,
"sellability_reason" text,
"product_title" text,
"brand" text,
"category_tree" text,
"current_price" real,
"avg_price_90d" real,
"sales_rank" integer,
"monthly_sold" integer,
"seller_count" integer,
"amazon_is_seller" boolean,
"raw_product_json" text,
"last_seen_at" timestamp with time zone NOT NULL,
"raw_inventory_json" text,
CONSTRAINT "uq_stalker_inventory_run_seller_asin" UNIQUE("run_id","seller_id","asin")
);
--> statement-breakpoint
ALTER TABLE "analysis_results" ADD CONSTRAINT "analysis_results_run_id_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."runs"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "category_product_results" ADD CONSTRAINT "category_product_results_run_id_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."runs"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "stalker_asin_scans" ADD CONSTRAINT "stalker_asin_scans_run_id_stalker_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."stalker_runs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "stalker_asin_sellers" ADD CONSTRAINT "stalker_asin_sellers_scan_id_stalker_asin_scans_id_fk" FOREIGN KEY ("scan_id") REFERENCES "public"."stalker_asin_scans"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "stalker_asin_sellers" ADD CONSTRAINT "stalker_asin_sellers_seller_id_sellers_seller_id_fk" FOREIGN KEY ("seller_id") REFERENCES "public"."sellers"("seller_id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "stalker_seller_inventory" ADD CONSTRAINT "stalker_seller_inventory_run_id_stalker_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."stalker_runs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "stalker_seller_inventory" ADD CONSTRAINT "stalker_seller_inventory_seller_id_sellers_seller_id_fk" FOREIGN KEY ("seller_id") REFERENCES "public"."sellers"("seller_id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
CREATE INDEX "idx_analysis_results_run_id" ON "analysis_results" USING btree ("run_id");--> statement-breakpoint
CREATE INDEX "idx_analysis_results_asin" ON "analysis_results" USING btree ("asin");--> statement-breakpoint
CREATE INDEX "idx_analysis_results_verdict" ON "analysis_results" USING btree ("verdict");--> statement-breakpoint
CREATE INDEX "idx_analysis_results_sellability_status" ON "analysis_results" USING btree ("sellability_status");--> statement-breakpoint
CREATE INDEX "idx_analysis_results_fetched_at" ON "analysis_results" USING btree ("fetched_at");--> statement-breakpoint
CREATE INDEX "idx_category_results_run_id" ON "category_product_results" USING btree ("run_id");--> statement-breakpoint
CREATE INDEX "idx_category_results_verdict" ON "category_product_results" USING btree ("verdict");--> statement-breakpoint
CREATE INDEX "idx_category_results_sellability_status" ON "category_product_results" USING btree ("sellability_status");--> statement-breakpoint
CREATE INDEX "idx_category_results_fetched_at" ON "category_product_results" USING btree ("fetched_at");--> statement-breakpoint
CREATE INDEX "idx_runs_started_at" ON "runs" USING btree ("started_at");--> statement-breakpoint
CREATE INDEX "idx_runs_type" ON "runs" USING btree ("type");--> statement-breakpoint
CREATE INDEX "idx_runs_status" ON "runs" USING btree ("status");--> statement-breakpoint
CREATE INDEX "idx_stalker_scans_run_id" ON "stalker_asin_scans" USING btree ("run_id");--> statement-breakpoint
CREATE INDEX "idx_stalker_scans_source_asin" ON "stalker_asin_scans" USING btree ("source_asin");--> statement-breakpoint
CREATE INDEX "idx_stalker_runs_started_at" ON "stalker_runs" USING btree ("started_at");--> statement-breakpoint
CREATE INDEX "idx_stalker_inventory_seller_id" ON "stalker_seller_inventory" USING btree ("seller_id");--> statement-breakpoint
CREATE INDEX "idx_stalker_inventory_asin" ON "stalker_seller_inventory" USING btree ("asin");--> statement-breakpoint
CREATE INDEX "idx_stalker_inventory_product_title" ON "stalker_seller_inventory" USING btree ("product_title");

File diff suppressed because it is too large Load Diff

View File

@@ -1,13 +0,0 @@
{
"version": "7",
"dialect": "postgresql",
"entries": [
{
"idx": 0,
"version": "7",
"when": 1779683900467,
"tag": "0000_gorgeous_william_stryker",
"breakpoints": true
}
]
}

View File

@@ -0,0 +1,114 @@
import { beforeEach, expect, mock, test } from "bun:test";
import { processProductChunk } from "./analysis-pipeline.ts";
import type { ProductRecord } from "./types.ts";
const fetchKeepaDataBatchMock = mock(async (asins: string[]) => {
return new Map(
asins.map((asin) => [
asin,
{
currentPrice: 20,
avgPrice90: 18,
minPrice90: null,
maxPrice90: null,
salesRank: 100,
salesRankAvg90: null,
salesRankDrops30: null,
salesRankDrops90: null,
sellerCount: 3,
amazonIsSeller: false,
amazonBuyboxSharePct90d: null,
buyBoxSeller: null,
buyBoxPrice: null,
monthlySold: 50,
categoryTree: [],
},
]),
);
});
const fetchSellabilityBatchMock = mock(async (asins: string[]) => {
return new Map(
asins.map((asin) => [
asin,
asin === "B000000002"
? {
canSell: false,
sellabilityStatus: "restricted" as const,
sellabilityReason: "Approval required",
}
: {
canSell: true,
sellabilityStatus: "available" as const,
sellabilityReason: "Available",
},
]),
);
});
const fetchSpApiPricingAndFeesMock = mock(async () => ({
fbaFee: 4,
fbmFee: 2,
referralFeePercent: 15,
estimatedSalePrice: 20,
canSell: true,
sellabilityStatus: "available" as const,
sellabilityReason: "Available",
}));
const analyzeProductsMock = mock(async (products: any[]) =>
products.map((product) => ({
asin: product.record.asin,
verdict: "FBA" as const,
confidence: 95,
reasoning: "Analyzed",
})),
);
const getCacheMock = mock(async () => null);
const setCacheMock = mock(async () => undefined);
beforeEach(() => {
fetchKeepaDataBatchMock.mockClear();
fetchSellabilityBatchMock.mockClear();
fetchSpApiPricingAndFeesMock.mockClear();
analyzeProductsMock.mockClear();
getCacheMock.mockClear();
setCacheMock.mockClear();
});
test("lead analysis retains restricted input rows as SKIP without LLM analysis", async () => {
const products: ProductRecord[] = [
{ asin: "B000000001", name: "Available", unitCost: 5 },
{ asin: "B000000002", name: "Restricted", unitCost: 6 },
];
const results = await processProductChunk(products, {
llmBatchDelayMs: 0,
llmRetryDelayMs: 0,
dependencies: {
fetchKeepaDataBatch: fetchKeepaDataBatchMock,
fetchSellabilityBatch: fetchSellabilityBatchMock,
fetchSpApiPricingAndFees: fetchSpApiPricingAndFeesMock,
analyzeProducts: analyzeProductsMock,
getCache: getCacheMock,
setCache: setCacheMock,
},
});
expect(results).toHaveLength(2);
expect(results.map((result) => result.product.record.asin)).toEqual([
"B000000001",
"B000000002",
]);
expect(results.find((result) => result.product.record.asin === "B000000002")?.verdict)
.toEqual({
asin: "B000000002",
verdict: "SKIP",
confidence: 100,
reasoning: "Approval required",
});
expect(fetchKeepaDataBatchMock.mock.calls[0]?.[0]).toEqual(["B000000001"]);
expect(fetchSpApiPricingAndFeesMock.mock.calls).toHaveLength(1);
expect(analyzeProductsMock.mock.calls[0]?.[0]).toHaveLength(1);
});

View File

@@ -16,6 +16,15 @@ export const DEFAULT_PRICING_CONCURRENCY = 5;
export type SellabilityFilter = "available" | "all";
type AnalysisPipelineDependencies = {
fetchKeepaDataBatch: typeof fetchKeepaDataBatch;
fetchSellabilityBatch: typeof fetchSellabilityBatch;
fetchSpApiPricingAndFees: typeof fetchSpApiPricingAndFees;
getCache: typeof getCache;
setCache: typeof setCache;
analyzeProducts: typeof analyzeProducts;
};
export type AnalysisPipelineOptions = {
llmBatchSize?: number;
pricingConcurrency?: number;
@@ -23,6 +32,7 @@ export type AnalysisPipelineOptions = {
llmRetryDelayMs?: number;
sellability?: SellabilityFilter;
useClaude?: boolean;
dependencies?: Partial<AnalysisPipelineDependencies>;
};
export function chunkArray<T>(items: T[], chunkSize: number): T[][] {
@@ -62,23 +72,33 @@ export async function processProductChunk(
const llmRetryDelayMs = Math.max(0, options.llmRetryDelayMs ?? 10_000);
const sellabilityFilter = options.sellability ?? "available";
const useClaude = options.useClaude === true;
const dependencies: AnalysisPipelineDependencies = {
fetchKeepaDataBatch,
fetchSellabilityBatch,
fetchSpApiPricingAndFees,
getCache,
setCache,
analyzeProducts,
...options.dependencies,
};
console.log(`\nChecking cache for ${products.length} products...`);
const cached = new Map<string, EnrichedProduct>();
const excludedCachedAsins = new Set<string>();
const excludedCached = new Map<string, EnrichedProduct>();
const uncachedProducts: ProductRecord[] = [];
for (const p of products) {
const hit = await getCache(p.asin);
const hit = await dependencies.getCache(p.asin);
if (hit) {
const currentSourceProduct = { ...hit, record: p };
if (
sellabilityFilter === "all" ||
hit.spApi.sellabilityStatus === "available"
) {
console.log(` [cache hit] ${p.asin}`);
cached.set(p.asin, hit);
cached.set(p.asin, currentSourceProduct);
} else {
excludedCachedAsins.add(p.asin);
excludedCached.set(p.asin, currentSourceProduct);
console.log(
` [exclude cached] ${p.asin} - status=${hit.spApi.sellabilityStatus}`,
);
@@ -89,7 +109,7 @@ export async function processProductChunk(
}
console.log(
`${cached.size} cached available, ${excludedCachedAsins.size} cached excluded, ${uncachedProducts.length} to fetch`,
`${cached.size} cached available, ${excludedCached.size} cached excluded, ${uncachedProducts.length} to fetch`,
);
const sellabilityMap = new Map<string, SellabilityInfo>();
@@ -100,7 +120,7 @@ export async function processProductChunk(
console.log(
`\nChecking sellability for ${uncachedProducts.length} ASINs...`,
);
const sellResults = await fetchSellabilityBatch(
const sellResults = await dependencies.fetchSellabilityBatch(
uncachedProducts.map((p) => p.asin),
);
@@ -143,7 +163,7 @@ export async function processProductChunk(
if (availableProducts.length > 0) {
console.log(`\nFetching ${availableProducts.length} ASINs from Keepa...`);
try {
keepaResults = await fetchKeepaDataBatch(
keepaResults = await dependencies.fetchKeepaDataBatch(
availableProducts.map((p) => p.asin),
);
} catch (err) {
@@ -168,7 +188,10 @@ export async function processProductChunk(
sellabilityStatus: "unknown" as const,
sellabilityReason: "Sellability check returned no result",
};
const spApi = await fetchSpApiPricingAndFees(p.asin, sellability);
const spApi = await dependencies.fetchSpApiPricingAndFees(
p.asin,
sellability,
);
const keepa = keepaResults.get(p.asin);
if (keepa?.currentPrice && spApi.estimatedSalePrice === 0) {
@@ -196,17 +219,33 @@ export async function processProductChunk(
const availableAsins = new Set(availableProducts.map((ap) => ap.asin));
for (const p of products) {
if (excludedCachedAsins.has(p.asin)) {
const excludedCachedProduct = excludedCached.get(p.asin);
if (excludedCachedProduct) {
enriched.push({ ...excludedCachedProduct, record: p });
continue;
}
const cachedProduct = cached.get(p.asin);
if (cachedProduct) {
enriched.push(cachedProduct);
enriched.push({ ...cachedProduct, record: p });
continue;
}
if (!availableAsins.has(p.asin)) {
const sellability = sellabilityMap.get(p.asin);
if (sellability) {
enriched.push({
record: p,
keepa: null,
spApi: {
...unknownSpApiData(
sellability.sellabilityReason ?? "Product is not available",
),
...sellability,
},
fetchedAt: new Date().toISOString(),
});
}
continue;
}
@@ -221,19 +260,41 @@ export async function processProductChunk(
fetchedAt: new Date().toISOString(),
};
await setCache(p.asin, product);
await dependencies.setCache(p.asin, product);
enriched.push(product);
}
const resultsByProduct = new Map<EnrichedProduct, AnalysisResult>();
const llmProducts: EnrichedProduct[] = [];
for (const product of enriched) {
if (
sellabilityFilter !== "all" &&
product.spApi.sellabilityStatus !== "available"
) {
resultsByProduct.set(product, {
product,
verdict: {
asin: product.record.asin,
verdict: "SKIP",
confidence: 100,
reasoning:
product.spApi.sellabilityReason ??
`Sellability status: ${product.spApi.sellabilityStatus}`,
},
});
} else {
llmProducts.push(product);
}
}
console.log(
`\nAnalyzing ${enriched.length} products via LLM (batch size: ${llmBatchSize})...\n`,
`\nAnalyzing ${llmProducts.length} products via LLM (batch size: ${llmBatchSize})...\n`,
);
const results: AnalysisResult[] = [];
for (let i = 0; i < enriched.length; i += llmBatchSize) {
const batch = enriched.slice(i, i + llmBatchSize);
for (let i = 0; i < llmProducts.length; i += llmBatchSize) {
const batch = llmProducts.slice(i, i + llmBatchSize);
const batchNum = Math.floor(i / llmBatchSize) + 1;
const totalBatches = Math.ceil(enriched.length / llmBatchSize);
const totalBatches = Math.ceil(llmProducts.length / llmBatchSize);
console.log(` LLM batch ${batchNum}/${totalBatches}...`);
if (i > 0 && llmBatchDelayMs > 0) {
@@ -242,7 +303,7 @@ export async function processProductChunk(
let verdicts;
try {
verdicts = await analyzeProducts(batch, {
verdicts = await dependencies.analyzeProducts(batch, {
ignoreSellability: sellabilityFilter === "all",
useClaude,
});
@@ -251,7 +312,7 @@ export async function processProductChunk(
await wait(llmRetryDelayMs);
}
try {
verdicts = await analyzeProducts(batch, {
verdicts = await dependencies.analyzeProducts(batch, {
ignoreSellability: sellabilityFilter === "all",
useClaude,
});
@@ -264,7 +325,7 @@ export async function processProductChunk(
const enrichedProduct = batch[j];
if (!enrichedProduct) continue;
results.push({
resultsByProduct.set(enrichedProduct, {
product: enrichedProduct,
verdict: verdicts?.[j] ?? {
asin: enrichedProduct.record.asin,
@@ -276,5 +337,7 @@ export async function processProductChunk(
}
}
return results;
return enriched
.map((product) => resultsByProduct.get(product))
.filter((result): result is AnalysisResult => result !== undefined);
}

13
src/asin.test.ts Normal file
View File

@@ -0,0 +1,13 @@
import { expect, test } from "bun:test";
import { normalizeAsin, requireAsin } from "./asin.ts";
test("normalizes any valid ten-character ASIN including ISBN-style values", () => {
expect(normalizeAsin(" b07sn9bhvv ")).toBe("B07SN9BHVV");
expect(normalizeAsin("0306406152")).toBe("0306406152");
});
test("rejects values that cannot be canonical product ASIN keys", () => {
expect(normalizeAsin("short")).toBeNull();
expect(normalizeAsin("B07SN9BHV!")).toBeNull();
expect(() => requireAsin("012345678901")).toThrow("Invalid ASIN");
});

14
src/asin.ts Normal file
View File

@@ -0,0 +1,14 @@
export const ASIN_PATTERN = /^[A-Z0-9]{10}$/;
export function normalizeAsin(value: unknown): string | null {
const asin = String(value ?? "").trim().toUpperCase();
return ASIN_PATTERN.test(asin) ? asin : null;
}
export function requireAsin(value: unknown): string {
const asin = normalizeAsin(value);
if (!asin) {
throw new Error(`Invalid ASIN: "${String(value ?? "").trim()}"`);
}
return asin;
}

View File

@@ -1,8 +1,11 @@
import { existsSync, mkdirSync, readFileSync } from "node:fs";
import path from "node:path";
import { db } from "../db/index.ts";
import { runs, categoryProductResults } from "../db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { normalizeAsin } from "../asin.ts";
import {
createCategoryRun,
persistLlmResults,
updateCategoryRun,
} from "../db/persistence.ts";
import { config } from "../config.ts";
import { analyzeProducts } from "../integrations/llm.ts";
import { fetchSellabilityBatch, fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
@@ -144,26 +147,7 @@ export async function insertCategoryRunSummary(
summary: CategoryRunSummary,
runTimestamp: string,
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "category_analysis",
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
categoryId: summary.categoryId,
categoryLabel: summary.categoryLabel,
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
errorMessage: summary.error || null,
startedAt: new Date(runTimestamp),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert category run.");
return row.id;
return createCategoryRun(summary, runTimestamp);
}
export async function updateCategoryRunSummary(
@@ -179,20 +163,7 @@ export async function updateCategoryRunSummary(
| "error"
>,
): Promise<void> {
await db
.update(runs)
.set({
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
status: summary.status as typeof runs.$inferInsert.status,
errorMessage: summary.error || null,
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
})
.where(eq(runs.id, runId));
await updateCategoryRun(runId, summary);
}
export async function insertProductAnalysisResults(
@@ -200,89 +171,10 @@ export async function insertProductAnalysisResults(
results: AnalysisResult[],
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((r) => {
const price =
r.product.keepa?.currentPrice ??
r.product.record.sellingPriceFromSheet ??
r.product.spApi.estimatedSalePrice;
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
return {
asin: r.product.record.asin,
runId,
name: r.product.record.name,
brand: r.product.record.brand ?? null,
category:
r.product.record.category ??
r.product.keepa?.categoryTree?.join(" > ") ??
null,
unitCost: r.product.record.unitCost ?? null,
currentPrice: price ?? null,
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
salesRank: rank ?? null,
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
sellerCount: r.product.keepa?.sellerCount ?? null,
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
monthlySold: r.product.keepa?.monthlySold ?? null,
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
fbaFee: r.product.spApi.fbaFee ?? null,
fbmFee: r.product.spApi.fbmFee ?? null,
referralPercent: r.product.spApi.referralFeePercent ?? null,
canSell:
r.product.spApi.canSell == null
? "unknown"
: r.product.spApi.canSell
? "yes"
: "no",
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
verdict: r.verdict.verdict,
confidence: r.verdict.confidence,
reasoning: r.verdict.reasoning ?? null,
fetchedAt: new Date(r.product.fetchedAt),
};
await persistLlmResults(runId, results, {
source: "category_analysis",
metadataSource: "catalog",
});
await db
.insert(categoryProductResults)
.values(rows)
.onConflictDoUpdate({
target: categoryProductResults.asin,
set: {
runId: sql`EXCLUDED.run_id`,
name: sql`EXCLUDED.name`,
brand: sql`EXCLUDED.brand`,
category: sql`EXCLUDED.category`,
unitCost: sql`EXCLUDED.unit_cost`,
currentPrice: sql`EXCLUDED.current_price`,
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
salesRank: sql`EXCLUDED.sales_rank`,
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
sellerCount: sql`EXCLUDED.seller_count`,
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
monthlySold: sql`EXCLUDED.monthly_sold`,
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
fbaFee: sql`EXCLUDED.fba_fee`,
fbmFee: sql`EXCLUDED.fbm_fee`,
referralPercent: sql`EXCLUDED.referral_percent`,
canSell: sql`EXCLUDED.can_sell`,
sellabilityStatus: sql`EXCLUDED.sellability_status`,
sellabilityReason: sql`EXCLUDED.sellability_reason`,
verdict: sql`EXCLUDED.verdict`,
confidence: sql`EXCLUDED.confidence`,
reasoning: sql`EXCLUDED.reasoning`,
fetchedAt: sql`EXCLUDED.fetched_at`,
},
});
}
function loadCategoryBlacklist(filePath: string): Set<number> {
@@ -664,7 +556,11 @@ async function fetchCategoryBestSellerAsins(
for (const value of candidates) {
if (Array.isArray(value)) {
return [
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
...new Set(
value
.map((v) => normalizeAsin(v))
.filter((asin): asin is string => asin !== null),
),
].slice(0, limit);
}
}
@@ -919,7 +815,7 @@ async function fetchKeepaEnrichmentMap(
const products = Array.isArray(data?.products) ? data.products : [];
for (const product of products) {
const asin = String(product?.asin ?? "").trim();
const asin = normalizeAsin(product?.asin);
if (!asin) continue;
out.set(asin, {
keepa: parseKeepaProduct(product),

View File

@@ -2,9 +2,12 @@ import { existsSync, mkdirSync, readFileSync } from "node:fs";
import path from "node:path";
import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";
import { db } from "../db/index.ts";
import { runs, categoryProductResults } from "../db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { normalizeAsin } from "../asin.ts";
import {
createCategoryRun,
persistLlmResults,
updateCategoryRun,
} from "../db/persistence.ts";
import { config } from "../config.ts";
import {
connectCache,
@@ -479,26 +482,7 @@ export async function insertCategoryRunSummary(
summary: CategoryRunSummary,
runTimestamp: string,
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "category_analysis",
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
categoryId: summary.categoryId,
categoryLabel: summary.categoryLabel,
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
errorMessage: summary.error || null,
startedAt: new Date(runTimestamp),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert category run.");
return row.id;
return createCategoryRun(summary, runTimestamp);
}
export async function updateCategoryRunSummary(
@@ -514,20 +498,7 @@ export async function updateCategoryRunSummary(
| "error"
>,
): Promise<void> {
await db
.update(runs)
.set({
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
status: summary.status as typeof runs.$inferInsert.status,
errorMessage: summary.error || null,
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
})
.where(eq(runs.id, runId));
await updateCategoryRun(runId, summary);
}
export async function insertProductAnalysisResults(
@@ -535,89 +506,10 @@ export async function insertProductAnalysisResults(
results: AnalysisResult[],
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((r) => {
const price =
r.product.keepa?.currentPrice ??
r.product.record.sellingPriceFromSheet ??
r.product.spApi.estimatedSalePrice;
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
return {
asin: r.product.record.asin,
runId,
name: r.product.record.name,
brand: r.product.record.brand ?? null,
category:
r.product.record.category ??
r.product.keepa?.categoryTree?.join(" > ") ??
null,
unitCost: r.product.record.unitCost ?? null,
currentPrice: price ?? null,
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
salesRank: rank ?? null,
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
sellerCount: r.product.keepa?.sellerCount ?? null,
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
monthlySold: r.product.keepa?.monthlySold ?? null,
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
fbaFee: r.product.spApi.fbaFee ?? null,
fbmFee: r.product.spApi.fbmFee ?? null,
referralPercent: r.product.spApi.referralFeePercent ?? null,
canSell:
r.product.spApi.canSell == null
? "unknown"
: r.product.spApi.canSell
? "yes"
: "no",
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
verdict: r.verdict.verdict,
confidence: r.verdict.confidence,
reasoning: r.verdict.reasoning ?? null,
fetchedAt: new Date(r.product.fetchedAt),
};
await persistLlmResults(runId, results, {
source: "category_analysis",
metadataSource: "catalog",
});
await db
.insert(categoryProductResults)
.values(rows)
.onConflictDoUpdate({
target: categoryProductResults.asin,
set: {
runId: sql`EXCLUDED.run_id`,
name: sql`EXCLUDED.name`,
brand: sql`EXCLUDED.brand`,
category: sql`EXCLUDED.category`,
unitCost: sql`EXCLUDED.unit_cost`,
currentPrice: sql`EXCLUDED.current_price`,
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
salesRank: sql`EXCLUDED.sales_rank`,
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
sellerCount: sql`EXCLUDED.seller_count`,
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
monthlySold: sql`EXCLUDED.monthly_sold`,
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
fbaFee: sql`EXCLUDED.fba_fee`,
fbmFee: sql`EXCLUDED.fbm_fee`,
referralPercent: sql`EXCLUDED.referral_percent`,
canSell: sql`EXCLUDED.can_sell`,
sellabilityStatus: sql`EXCLUDED.sellability_status`,
sellabilityReason: sql`EXCLUDED.sellability_reason`,
verdict: sql`EXCLUDED.verdict`,
confidence: sql`EXCLUDED.confidence`,
reasoning: sql`EXCLUDED.reasoning`,
fetchedAt: sql`EXCLUDED.fetched_at`,
},
});
}
function loadCategoryBlacklist(filePath: string): Set<number> {
@@ -999,7 +891,11 @@ async function fetchCategoryBestSellerAsins(
for (const value of candidates) {
if (Array.isArray(value)) {
return [
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
...new Set(
value
.map((v) => normalizeAsin(v))
.filter((asin): asin is string => asin !== null),
),
].slice(0, limit);
}
}
@@ -1258,7 +1154,7 @@ async function fetchKeepaEnrichmentMap(
const products = Array.isArray(data?.products) ? data.products : [];
for (const product of products) {
const asin = String(product?.asin ?? "").trim();
const asin = normalizeAsin(product?.asin);
if (!asin) continue;
const parsed = {
keepa: parseKeepaProduct(product),

View File

@@ -1,8 +1,11 @@
import { existsSync, mkdirSync, readFileSync } from "node:fs";
import path from "node:path";
import { db } from "../db/index.ts";
import { runs, categoryProductResults } from "../db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { normalizeAsin } from "../asin.ts";
import {
createCategoryRun,
persistLlmResults,
updateCategoryRun,
} from "../db/persistence.ts";
import { config } from "../config.ts";
import { analyzeProducts } from "../integrations/llm.ts";
import { fetchSellabilityBatch, fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
@@ -176,26 +179,7 @@ export async function insertCategoryRunSummary(
summary: CategoryRunSummary,
runTimestamp: string,
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "category_analysis",
status: (summary.status as typeof runs.$inferInsert.status) ?? "running",
categoryId: summary.categoryId,
categoryLabel: summary.categoryLabel,
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
errorMessage: summary.error || null,
startedAt: new Date(runTimestamp),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert category run.");
return row.id;
return createCategoryRun(summary, runTimestamp);
}
export async function updateCategoryRunSummary(
@@ -211,20 +195,7 @@ export async function updateCategoryRunSummary(
| "error"
>,
): Promise<void> {
await db
.update(runs)
.set({
topAsinsChecked: summary.topAsinsChecked,
availableAsins: summary.availableAsins,
totalProducts: summary.topAsinsChecked,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
status: summary.status as typeof runs.$inferInsert.status,
errorMessage: summary.error || null,
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
})
.where(eq(runs.id, runId));
await updateCategoryRun(runId, summary);
}
export async function insertProductAnalysisResults(
@@ -232,89 +203,10 @@ export async function insertProductAnalysisResults(
results: AnalysisResult[],
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((r) => {
const price =
r.product.keepa?.currentPrice ??
r.product.record.sellingPriceFromSheet ??
r.product.spApi.estimatedSalePrice;
const rank = r.product.keepa?.salesRank ?? r.product.record.amazonRank;
return {
asin: r.product.record.asin,
runId,
name: r.product.record.name,
brand: r.product.record.brand ?? null,
category:
r.product.record.category ??
r.product.keepa?.categoryTree?.join(" > ") ??
null,
unitCost: r.product.record.unitCost ?? null,
currentPrice: price ?? null,
avgPrice90d: r.product.keepa?.avgPrice90 ?? null,
avgPrice90dSheet: r.product.record.avgPrice90FromSheet ?? null,
sellingPriceSheet: r.product.record.sellingPriceFromSheet ?? null,
salesRank: rank ?? null,
salesRankAvg90d: r.product.keepa?.salesRankAvg90 ?? null,
sellerCount: r.product.keepa?.sellerCount ?? null,
amazonIsSeller: r.product.keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: r.product.keepa?.amazonBuyboxSharePct90d ?? null,
monthlySold: r.product.keepa?.monthlySold ?? null,
rankDrops30d: r.product.keepa?.salesRankDrops30 ?? null,
rankDrops90d: r.product.keepa?.salesRankDrops90 ?? null,
fbaFee: r.product.spApi.fbaFee ?? null,
fbmFee: r.product.spApi.fbmFee ?? null,
referralPercent: r.product.spApi.referralFeePercent ?? null,
canSell:
r.product.spApi.canSell == null
? "unknown"
: r.product.spApi.canSell
? "yes"
: "no",
sellabilityStatus: r.product.spApi.sellabilityStatus ?? null,
sellabilityReason: r.product.spApi.sellabilityReason ?? null,
verdict: r.verdict.verdict,
confidence: r.verdict.confidence,
reasoning: r.verdict.reasoning ?? null,
fetchedAt: new Date(r.product.fetchedAt),
};
await persistLlmResults(runId, results, {
source: "category_analysis",
metadataSource: "catalog",
});
await db
.insert(categoryProductResults)
.values(rows)
.onConflictDoUpdate({
target: categoryProductResults.asin,
set: {
runId: sql`EXCLUDED.run_id`,
name: sql`EXCLUDED.name`,
brand: sql`EXCLUDED.brand`,
category: sql`EXCLUDED.category`,
unitCost: sql`EXCLUDED.unit_cost`,
currentPrice: sql`EXCLUDED.current_price`,
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
salesRank: sql`EXCLUDED.sales_rank`,
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
sellerCount: sql`EXCLUDED.seller_count`,
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
monthlySold: sql`EXCLUDED.monthly_sold`,
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
fbaFee: sql`EXCLUDED.fba_fee`,
fbmFee: sql`EXCLUDED.fbm_fee`,
referralPercent: sql`EXCLUDED.referral_percent`,
canSell: sql`EXCLUDED.can_sell`,
sellabilityStatus: sql`EXCLUDED.sellability_status`,
sellabilityReason: sql`EXCLUDED.sellability_reason`,
verdict: sql`EXCLUDED.verdict`,
confidence: sql`EXCLUDED.confidence`,
reasoning: sql`EXCLUDED.reasoning`,
fetchedAt: sql`EXCLUDED.fetched_at`,
},
});
}
function loadCategoryBlacklist(filePath: string): Set<number> {
@@ -696,7 +588,11 @@ async function fetchCategoryBestSellerAsins(
for (const value of candidates) {
if (Array.isArray(value)) {
return [
...new Set(value.map((v) => String(v).trim()).filter(Boolean)),
...new Set(
value
.map((v) => normalizeAsin(v))
.filter((asin): asin is string => asin !== null),
),
].slice(0, limit);
}
}
@@ -951,7 +847,7 @@ async function fetchKeepaEnrichmentMap(
const products = Array.isArray(data?.products) ? data.products : [];
for (const product of products) {
const asin = String(product?.asin ?? "").trim();
const asin = normalizeAsin(product?.asin);
if (!asin) continue;
out.set(asin, {
keepa: parseKeepaProduct(product),

541
src/db/persistence.ts Normal file
View File

@@ -0,0 +1,541 @@
import { sql } from "drizzle-orm";
import { requireAsin, normalizeAsin } from "../asin.ts";
import type {
AnalysisResult,
ProductRecord,
SupplierAnalysisResult,
} from "../types.ts";
import { db } from "./index.ts";
import {
analysisRevisions,
analysisRunStats,
categoryRunDetails,
productIdentifiers,
productObservations,
products,
runItems,
runs,
sourcingInputs,
supplierScores,
upcResolutionCandidates,
upcResolutions,
} from "./schema.ts";
type Executor = any;
type MetadataSource = "input" | "catalog";
type ProductSeed = {
asin: string;
name?: string | null;
brand?: string | null;
category?: string | null;
metadataSource?: MetadataSource;
fetchedAt?: Date;
};
export type CategoryRunSummaryInput = {
categoryId: number;
categoryLabel: string;
topAsinsChecked: number;
availableAsins: number;
fba: number;
fbm: number;
skip: number;
status: "running" | "ok" | "empty" | "failed";
error: string;
};
export type RunCounts = {
totalProducts: number;
fbaCount: number;
fbmCount: number;
skipCount: number;
};
function emptyToNull(value: string | undefined | null): string | null {
const trimmed = value?.trim();
return trimmed ? trimmed : null;
}
function productCategory(record: ProductRecord, result: AnalysisResult): string | null {
return emptyToNull(
record.category ?? result.product.keepa?.categoryTree?.join(" > "),
);
}
export async function upsertProduct(
seed: ProductSeed,
executor: Executor = db,
): Promise<string> {
const asin = requireAsin(seed.asin);
const now = seed.fetchedAt ?? new Date();
const isCatalog = seed.metadataSource === "catalog";
await executor
.insert(products)
.values({
asin,
name: emptyToNull(seed.name),
brand: emptyToNull(seed.brand),
category: emptyToNull(seed.category),
metadataFetchedAt: isCatalog ? now : null,
firstSeenAt: now,
lastSeenAt: now,
})
.onConflictDoUpdate({
target: products.asin,
set: {
lastSeenAt: sql`GREATEST(${products.lastSeenAt}, EXCLUDED.last_seen_at)`,
name: isCatalog
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.name, '') IS NOT NULL THEN EXCLUDED.name ELSE ${products.name} END`
: sql`COALESCE(${products.name}, NULLIF(EXCLUDED.name, ''))`,
brand: isCatalog
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.brand, '') IS NOT NULL THEN EXCLUDED.brand ELSE ${products.brand} END`
: sql`COALESCE(${products.brand}, NULLIF(EXCLUDED.brand, ''))`,
category: isCatalog
? sql`CASE WHEN EXCLUDED.metadata_fetched_at >= COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz) AND NULLIF(EXCLUDED.category, '') IS NOT NULL THEN EXCLUDED.category ELSE ${products.category} END`
: sql`COALESCE(${products.category}, NULLIF(EXCLUDED.category, ''))`,
metadataFetchedAt: isCatalog
? sql`GREATEST(COALESCE(${products.metadataFetchedAt}, '-infinity'::timestamptz), EXCLUDED.metadata_fetched_at)`
: products.metadataFetchedAt,
},
});
return asin;
}
export async function insertObservation(
runId: number,
result: AnalysisResult,
source: string,
executor: Executor = db,
): Promise<number> {
const fetchedAt = new Date(result.product.fetchedAt);
const record = result.product.record;
const keepa = result.product.keepa;
const spApi = result.product.spApi;
const asin = requireAsin(record.asin);
const [observation] = await executor
.insert(productObservations)
.values({
productAsin: asin,
runId,
source,
currentPrice:
keepa?.currentPrice ??
record.sellingPriceFromSheet ??
spApi.estimatedSalePrice ??
null,
avgPrice90d: keepa?.avgPrice90 ?? null,
salesRank: keepa?.salesRank ?? record.amazonRank ?? null,
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
monthlySold: keepa?.monthlySold ?? null,
rankDrops30d: keepa?.salesRankDrops30 ?? null,
rankDrops90d: keepa?.salesRankDrops90 ?? null,
sellerCount: keepa?.sellerCount ?? null,
amazonIsSeller: keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
fbaFee: spApi.fbaFee ?? null,
fbmFee: spApi.fbmFee ?? null,
referralPercent: spApi.referralFeePercent ?? null,
canSell: spApi.canSell,
sellabilityStatus: spApi.sellabilityStatus,
sellabilityReason: spApi.sellabilityReason ?? null,
fetchedAt,
})
.returning({ id: productObservations.id });
if (!observation) throw new Error(`Failed to insert observation for ${asin}`);
return observation.id;
}
function sourcingInputValues(runItemId: number, record: ProductRecord) {
return {
runItemId,
suppliedName: emptyToNull(record.name),
suppliedBrand: emptyToNull(record.brand),
suppliedCategory: emptyToNull(record.category),
unitCost: record.unitCost ?? null,
avgPrice90dSheet: record.avgPrice90FromSheet ?? null,
sellingPriceSheet: record.sellingPriceFromSheet ?? null,
fbaNetSheet: record.fbaNet ?? null,
grossProfitDollar: record.grossProfit ?? null,
grossProfitPct: record.grossProfitPct ?? null,
netProfitSheet: record.netProfitFromSheet ?? null,
roiSheet: record.roiFromSheet ?? null,
moq: record.moq ?? null,
moqCost: record.moqCost ?? null,
qtyAvailable: record.totalQtyAvail ?? null,
supplier: emptyToNull(record.supplier),
sourceUrl: emptyToNull(record.sourceUrl),
asinLink: emptyToNull(record.asinLink),
promoCouponCode: emptyToNull(record.promoCouponCode),
notes: emptyToNull(record.notes),
leadDate: emptyToNull(record.leadDate),
};
}
export async function persistLlmResults(
runId: number,
results: AnalysisResult[],
options: {
source: string;
metadataSource?: MetadataSource;
preserveSourcingInput?: boolean;
sourceInventoryIds?: Map<string, number>;
},
): Promise<void> {
for (const result of results) {
const record = result.product.record;
const fetchedAt = new Date(result.product.fetchedAt);
const asin = await upsertProduct({
asin: record.asin,
name: record.name,
brand: record.brand,
category: productCategory(record, result),
metadataSource: options.metadataSource ?? "input",
fetchedAt,
});
const [item] = await db
.insert(runItems)
.values({
runId,
productAsin: asin,
sourceInventoryItemId: options.sourceInventoryIds?.get(asin) ?? null,
})
.returning({ id: runItems.id });
if (!item) throw new Error(`Failed to insert run item for ${asin}`);
if (options.preserveSourcingInput) {
await db.insert(sourcingInputs).values(sourcingInputValues(item.id, record));
}
const observationId = await insertObservation(runId, result, options.source);
await db.insert(analysisRevisions).values({
runItemId: item.id,
observationId,
method: "llm",
decision: result.verdict.verdict,
confidence: result.verdict.confidence,
reasoning: result.verdict.reasoning ?? null,
analyzedAt: fetchedAt,
});
}
}
function supplierSourcingValues(runItemId: number, result: SupplierAnalysisResult) {
return {
runItemId,
suppliedName: emptyToNull(result.record.name),
suppliedBrand: emptyToNull(result.record.brand),
suppliedCategory: emptyToNull(result.record.category),
unitCost: result.record.unitCost ?? null,
};
}
async function insertSupplierObservation(
runId: number,
productAsin: string,
result: SupplierAnalysisResult,
): Promise<number | null> {
const keepa = result.keepa;
const spApi = result.spApi;
if (!spApi && !keepa) return null;
const [row] = await db
.insert(productObservations)
.values({
productAsin,
runId,
source: "supplier_upc",
currentPrice: result.score.salePrice,
avgPrice90d: keepa?.avgPrice90 ?? null,
salesRank: keepa?.salesRank ?? null,
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
monthlySold: keepa?.monthlySold ?? null,
rankDrops30d: keepa?.salesRankDrops30 ?? null,
rankDrops90d: keepa?.salesRankDrops90 ?? null,
sellerCount: keepa?.sellerCount ?? null,
amazonIsSeller: keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
fbaFee: spApi?.fbaFee ?? null,
fbmFee: spApi?.fbmFee ?? null,
referralPercent: spApi?.referralFeePercent ?? null,
canSell: spApi?.canSell ?? null,
sellabilityStatus: spApi?.sellabilityStatus ?? null,
sellabilityReason: spApi?.sellabilityReason ?? null,
fetchedAt: new Date(result.fetchedAt),
})
.returning({ id: productObservations.id });
return row?.id ?? null;
}
export async function persistSupplierResults(
runId: number,
results: SupplierAnalysisResult[],
): Promise<void> {
for (const result of results) {
const resolvedAsin = normalizeAsin(result.lookup.asin);
if (resolvedAsin) {
await upsertProduct({
asin: resolvedAsin,
name: result.record.name,
brand: result.record.brand,
category: result.record.category,
metadataSource: "input",
fetchedAt: new Date(result.fetchedAt),
});
if (result.keepa?.categoryTree?.length) {
await upsertProduct({
asin: resolvedAsin,
category: result.keepa.categoryTree.join(" > "),
metadataSource: "catalog",
fetchedAt: new Date(result.fetchedAt),
});
}
}
const [item] = await db
.insert(runItems)
.values({
runId,
productAsin: resolvedAsin,
sourceRow: result.rowNumber ?? null,
})
.returning({ id: runItems.id });
if (!item) throw new Error("Failed to insert supplier run item");
await db.insert(sourcingInputs).values(supplierSourcingValues(item.id, result));
await db.insert(upcResolutions).values({
runItemId: item.id,
requestedUpc: result.upc,
normalizedUpc: result.lookup.normalizedUpc,
provider: result.lookup.provider ?? "unknown",
status: result.lookup.status,
reason: result.lookup.reason ?? null,
resolvedProductAsin: resolvedAsin,
resolvedAt: new Date(result.fetchedAt),
});
for (const candidate of result.lookup.candidateAsins) {
const candidateAsin = normalizeAsin(candidate);
if (!candidateAsin) continue;
await upsertProduct({ asin: candidateAsin, fetchedAt: new Date(result.fetchedAt) });
await db
.insert(upcResolutionCandidates)
.values({ runItemId: item.id, productAsin: candidateAsin })
.onConflictDoUpdate({
target: [
upcResolutionCandidates.runItemId,
upcResolutionCandidates.productAsin,
],
set: { productAsin: sql`EXCLUDED.product_asin` },
});
}
if (resolvedAsin) {
await db
.insert(productIdentifiers)
.values({
productAsin: resolvedAsin,
identifierType:
result.lookup.normalizedUpc.length === 12
? "upc"
: result.lookup.normalizedUpc.length === 13
? "ean"
: "gtin",
identifierValue: result.lookup.normalizedUpc,
source: "supplier_upc",
confirmedAt: new Date(result.fetchedAt),
})
.onConflictDoUpdate({
target: [
productIdentifiers.identifierType,
productIdentifiers.identifierValue,
],
set: {
productAsin: resolvedAsin,
source: "supplier_upc",
confirmedAt: new Date(result.fetchedAt),
},
});
}
const observationId = resolvedAsin
? await insertSupplierObservation(runId, resolvedAsin, result)
: null;
const [revision] = await db
.insert(analysisRevisions)
.values({
runItemId: item.id,
observationId,
method: "supplier_scoring",
decision: result.score.verdict,
confidence: result.score.score,
reasoning: result.score.reason,
analyzedAt: new Date(result.fetchedAt),
})
.returning({ id: analysisRevisions.id });
if (!revision) throw new Error("Failed to insert supplier analysis revision");
await db.insert(supplierScores).values({
revisionId: revision.id,
score: result.score.score,
salePrice: result.score.salePrice,
fbaFee: result.score.fbaFee,
profit: result.score.profit,
margin: result.score.margin,
roi: result.score.roi,
reason: result.score.reason,
});
}
}
export async function createCategoryRun(
summary: CategoryRunSummaryInput,
runTimestamp: string,
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "category_analysis",
status: summary.status,
errorMessage: summary.error || null,
startedAt: new Date(runTimestamp),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert category run.");
await db.insert(categoryRunDetails).values({
runId: row.id,
categoryId: summary.categoryId,
categoryLabel: summary.categoryLabel,
checkedAsinCount: summary.topAsinsChecked,
});
await db.insert(analysisRunStats).values({
runId: row.id,
processedCount: summary.topAsinsChecked,
availableCount: summary.availableAsins,
analyzedCount: summary.fba + summary.fbm + summary.skip,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
});
return row.id;
}
export async function updateCategoryRun(
runId: number,
summary: Pick<
CategoryRunSummaryInput,
| "topAsinsChecked"
| "availableAsins"
| "fba"
| "fbm"
| "skip"
| "status"
| "error"
>,
): Promise<void> {
await db
.update(runs)
.set({
status: summary.status,
errorMessage: summary.error || null,
...(summary.status !== "running" ? { completedAt: new Date() } : {}),
})
.where(sql`${runs.id} = ${runId}`);
await db
.insert(categoryRunDetails)
.values({
runId,
categoryId: 0,
categoryLabel: "",
checkedAsinCount: summary.topAsinsChecked,
})
.onConflictDoUpdate({
target: categoryRunDetails.runId,
set: { checkedAsinCount: summary.topAsinsChecked },
});
await db
.insert(analysisRunStats)
.values({
runId,
processedCount: summary.topAsinsChecked,
availableCount: summary.availableAsins,
analyzedCount: summary.fba + summary.fbm + summary.skip,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
})
.onConflictDoUpdate({
target: analysisRunStats.runId,
set: {
processedCount: summary.topAsinsChecked,
availableCount: summary.availableAsins,
analyzedCount: summary.fba + summary.fbm + summary.skip,
fbaCount: summary.fba,
fbmCount: summary.fbm,
skipCount: summary.skip,
},
});
}
export async function refreshRunStats(runId: number): Promise<RunCounts> {
const [stats] = await db.execute(
sql<{
total: string;
fba: string | null;
fbm: string | null;
buy: string | null;
watch: string | null;
skip: string | null;
}>`WITH latest AS (
SELECT DISTINCT ON (ri.id) ar.decision
FROM run_items ri
JOIN analysis_revisions ar ON ar.run_item_id = ri.id
WHERE ri.run_id = ${runId}
ORDER BY ri.id, ar.analyzed_at DESC, ar.id DESC
)
SELECT
COUNT(*) AS total,
SUM(CASE WHEN decision = 'FBA' THEN 1 ELSE 0 END) AS fba,
SUM(CASE WHEN decision = 'FBM' THEN 1 ELSE 0 END) AS fbm,
SUM(CASE WHEN decision = 'BUY' THEN 1 ELSE 0 END) AS buy,
SUM(CASE WHEN decision = 'WATCH' THEN 1 ELSE 0 END) AS watch,
SUM(CASE WHEN decision = 'SKIP' THEN 1 ELSE 0 END) AS skip
FROM latest`,
);
const counts = {
totalProducts: Number(stats?.total ?? 0),
fbaCount: Number(stats?.fba ?? 0),
fbmCount: Number(stats?.fbm ?? 0),
skipCount: Number(stats?.skip ?? 0),
};
await db
.insert(analysisRunStats)
.values({
runId,
processedCount: counts.totalProducts,
analyzedCount: counts.totalProducts,
fbaCount: counts.fbaCount,
fbmCount: counts.fbmCount,
buyCount: Number(stats?.buy ?? 0),
watchCount: Number(stats?.watch ?? 0),
skipCount: counts.skipCount,
})
.onConflictDoUpdate({
target: analysisRunStats.runId,
set: {
processedCount: counts.totalProducts,
analyzedCount: counts.totalProducts,
fbaCount: counts.fbaCount,
fbmCount: counts.fbmCount,
buyCount: Number(stats?.buy ?? 0),
watchCount: Number(stats?.watch ?? 0),
skipCount: counts.skipCount,
},
});
return counts;
}

View File

@@ -1,9 +1,13 @@
import { sql } from "drizzle-orm";
import {
type AnyPgColumn,
boolean,
check,
index,
integer,
pgEnum,
pgTable,
primaryKey,
real,
serial,
text,
@@ -11,13 +15,12 @@ import {
unique,
} from "drizzle-orm/pg-core";
// ─── Enums ───────────────────────────────────────────────────────────────────
export const runTypeEnum = pgEnum("run_type", [
"lead_analysis",
"category_analysis",
"supplier_upc",
"stalker",
"stalker_analysis",
]);
export const runStatusEnum = pgEnum("run_status", [
@@ -28,29 +31,54 @@ export const runStatusEnum = pgEnum("run_status", [
"completed",
]);
// ─── Runs ─────────────────────────────────────────────────────────────────────
// Unified run log; replaces the old `runs` and `category_analysis_runs` tables.
// Category-specific columns (categoryId, categoryLabel, …) are null for
// lead_analysis / supplier_upc runs.
export const analysisMethodEnum = pgEnum("analysis_method", [
"llm",
"supplier_scoring",
]);
export const analysisDecisionEnum = pgEnum("analysis_decision", [
"FBA",
"FBM",
"BUY",
"WATCH",
"SKIP",
]);
export const products = pgTable(
"products",
{
asin: text("asin").primaryKey(),
name: text("name"),
brand: text("brand"),
category: text("category"),
metadataFetchedAt: timestamp("metadata_fetched_at", { withTimezone: true }),
firstSeenAt: timestamp("first_seen_at", { withTimezone: true })
.notNull()
.defaultNow(),
lastSeenAt: timestamp("last_seen_at", { withTimezone: true })
.notNull()
.defaultNow(),
},
(t) => [
check("ck_products_asin", sql`${t.asin} ~ '^[A-Z0-9]{10}$'`),
index("idx_products_name").on(t.name),
index("idx_products_last_seen_at").on(t.lastSeenAt),
],
);
export const runs = pgTable(
"runs",
{
id: serial("id").primaryKey(),
type: runTypeEnum("type").notNull(),
parentRunId: integer("parent_run_id").references(
(): AnyPgColumn => runs.id,
{ onDelete: "cascade" },
),
inputFile: text("input_file"),
outputFile: text("output_file"),
status: runStatusEnum("status").notNull().default("running"),
errorMessage: text("error_message"),
totalProducts: integer("total_products"),
fbaCount: integer("fba_count"),
fbmCount: integer("fbm_count"),
skipCount: integer("skip_count"),
// Category-pipeline only
categoryId: integer("category_id"),
categoryLabel: text("category_label"),
topAsinsChecked: integer("top_asins_checked"),
availableAsins: integer("available_asins"),
startedAt: timestamp("started_at", { withTimezone: true })
.notNull()
.defaultNow(),
@@ -60,210 +88,262 @@ export const runs = pgTable(
index("idx_runs_started_at").on(t.startedAt),
index("idx_runs_type").on(t.type),
index("idx_runs_status").on(t.status),
index("idx_runs_parent_run_id").on(t.parentRunId),
],
);
// ─── Analysis results ─────────────────────────────────────────────────────────
// Archival table: one row per product per run for the lead-list and supplier
// UPC pipelines. Multiple rows for the same ASIN across different runs is fine.
export const analysisRunStats = pgTable("analysis_run_stats", {
runId: integer("run_id")
.primaryKey()
.references(() => runs.id, { onDelete: "cascade" }),
processedCount: integer("processed_count").notNull().default(0),
analyzedCount: integer("analyzed_count").notNull().default(0),
availableCount: integer("available_count").notNull().default(0),
fbaCount: integer("fba_count").notNull().default(0),
fbmCount: integer("fbm_count").notNull().default(0),
buyCount: integer("buy_count").notNull().default(0),
watchCount: integer("watch_count").notNull().default(0),
skipCount: integer("skip_count").notNull().default(0),
});
export const analysisResults = pgTable(
"analysis_results",
export const categoryRunDetails = pgTable("category_run_details", {
runId: integer("run_id")
.primaryKey()
.references(() => runs.id, { onDelete: "cascade" }),
categoryId: integer("category_id").notNull(),
categoryLabel: text("category_label").notNull(),
checkedAsinCount: integer("checked_asin_count").notNull().default(0),
selectionParametersJson: text("selection_parameters_json"),
});
export const stalkerRunDetails = pgTable("stalker_run_details", {
runId: integer("run_id")
.primaryKey()
.references(() => runs.id, { onDelete: "cascade" }),
requestedAsins: integer("requested_asins").notNull().default(0),
skippedAsins: integer("skipped_asins").notNull().default(0),
scannedAsins: integer("scanned_asins").notNull().default(0),
sourceAsinsWithMatches: integer("source_asins_with_matches")
.notNull()
.default(0),
candidateSellers: integer("candidate_sellers").notNull().default(0),
qualifyingSellers: integer("qualifying_sellers").notNull().default(0),
matchedSellers: integer("matched_sellers").notNull().default(0),
sellerMetadataRequests: integer("seller_metadata_requests")
.notNull()
.default(0),
sellerStorefrontRequests: integer("seller_storefront_requests")
.notNull()
.default(0),
inventorySellabilityCheckedAsins: integer(
"inventory_sellability_checked_asins",
)
.notNull()
.default(0),
inventorySellabilityAvailableAsins: integer(
"inventory_sellability_available_asins",
)
.notNull()
.default(0),
inventorySellabilityExcludedAsins: integer(
"inventory_sellability_excluded_asins",
)
.notNull()
.default(0),
persistedInventoryAsins: integer("persisted_inventory_asins")
.notNull()
.default(0),
});
export const productIdentifiers = pgTable(
"product_identifiers",
{
id: serial("id").primaryKey(),
runId: integer("run_id")
productAsin: text("product_asin")
.notNull()
.references(() => runs.id),
asin: text("asin").notNull(),
// Product identity
productName: text("product_name"),
brand: text("brand"),
category: text("category"),
upc: text("upc"),
// Supplier sheet data (lead_analysis only)
unitCost: real("unit_cost"),
avgPrice90dSheet: real("avg_price_90d_sheet"),
sellingPriceSheet: real("selling_price_sheet"),
fbaNetSheet: real("fba_net_sheet"),
grossProfitDollar: real("gross_profit_dollar"),
grossProfitPct: real("gross_profit_pct"),
netProfitSheet: real("net_profit_sheet"),
roiSheet: real("roi_sheet"),
moq: integer("moq"),
moqCost: real("moq_cost"),
qtyAvailable: integer("qty_available"),
supplier: text("supplier"),
sourceUrl: text("source_url"),
asinLink: text("asin_link"),
promoCouponCode: text("promo_coupon_code"),
notes: text("notes"),
leadDate: text("lead_date"),
// Market data
currentPrice: real("current_price"),
avgPrice90d: real("avg_price_90d"),
salesRank: integer("sales_rank"),
rankAvg90d: integer("rank_avg_90d"),
monthlySold: integer("monthly_sold"),
rankDrops30d: integer("rank_drops_30d"),
rankDrops90d: integer("rank_drops_90d"),
sellerCount: integer("seller_count"),
amazonIsSeller: boolean("amazon_is_seller"),
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
// Fees
fbaFee: real("fba_fee"),
fbmFee: real("fbm_fee"),
referralPercent: real("referral_percent"),
// Sellability
canSell: text("can_sell"),
sellabilityStatus: text("sellability_status"),
sellabilityReason: text("sellability_reason"),
// Supplier-UPC scoring
supplierScore: real("supplier_score"),
supplierProfit: real("supplier_profit"),
supplierMargin: real("supplier_margin"),
supplierRoi: real("supplier_roi"),
supplierReason: text("supplier_reason"),
upcLookupStatus: text("upc_lookup_status"),
upcLookupReason: text("upc_lookup_reason"),
candidateAsins: text("candidate_asins"),
// Verdict
verdict: text("verdict").notNull(),
confidence: real("confidence"),
reasoning: text("reasoning"),
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
.references(() => products.asin),
identifierType: text("identifier_type").notNull(),
identifierValue: text("identifier_value").notNull(),
source: text("source").notNull(),
confirmedAt: timestamp("confirmed_at", { withTimezone: true })
.notNull()
.defaultNow(),
},
(t) => [
index("idx_analysis_results_run_id").on(t.runId),
index("idx_analysis_results_asin").on(t.asin),
index("idx_analysis_results_verdict").on(t.verdict),
index("idx_analysis_results_sellability_status").on(t.sellabilityStatus),
index("idx_analysis_results_fetched_at").on(t.fetchedAt),
unique("uq_product_identifier_type_value").on(
t.identifierType,
t.identifierValue,
),
index("idx_product_identifiers_asin").on(t.productAsin),
],
);
// ─── Category product results ──────────────────────────────────────────────────
// Latest-per-ASIN snapshot for the category pipelines (bestsellers, monthly-sold,
// mid-range, stalker analysis). Upserted on conflict so each ASIN has one row.
export const categoryProductResults = pgTable(
"category_product_results",
export const productObservations = pgTable(
"product_observations",
{
id: serial("id").primaryKey(),
asin: text("asin").notNull().unique(),
productAsin: text("product_asin")
.notNull()
.references(() => products.asin),
runId: integer("run_id")
.notNull()
.references(() => runs.id),
name: text("name").notNull(),
brand: text("brand"),
category: text("category"),
unitCost: real("unit_cost"),
.references(() => runs.id, { onDelete: "cascade" }),
source: text("source").notNull(),
marketplace: text("marketplace").notNull().default("US"),
currentPrice: real("current_price"),
avgPrice90d: real("avg_price_90d"),
avgPrice90dSheet: real("avg_price_90d_sheet"),
sellingPriceSheet: real("selling_price_sheet"),
salesRank: integer("sales_rank"),
salesRankAvg90d: integer("sales_rank_avg_90d"),
sellerCount: integer("seller_count"),
amazonIsSeller: boolean("amazon_is_seller"),
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
monthlySold: integer("monthly_sold"),
rankDrops30d: integer("rank_drops_30d"),
rankDrops90d: integer("rank_drops_90d"),
sellerCount: integer("seller_count"),
amazonIsSeller: boolean("amazon_is_seller"),
amazonBuyboxSharePct90d: real("amazon_buybox_share_pct_90d"),
fbaFee: real("fba_fee"),
fbmFee: real("fbm_fee"),
referralPercent: real("referral_percent"),
canSell: text("can_sell"),
canSell: boolean("can_sell"),
sellabilityStatus: text("sellability_status"),
sellabilityReason: text("sellability_reason"),
verdict: text("verdict").notNull(),
confidence: real("confidence").notNull(),
reasoning: text("reasoning"),
rawProductJson: text("raw_product_json"),
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
},
(t) => [
index("idx_category_results_run_id").on(t.runId),
index("idx_category_results_verdict").on(t.verdict),
index("idx_category_results_sellability_status").on(t.sellabilityStatus),
index("idx_category_results_fetched_at").on(t.fetchedAt),
index("idx_product_observations_product_time").on(
t.productAsin,
t.fetchedAt.desc(),
),
index("idx_product_observations_run_id").on(t.runId),
index("idx_product_observations_sellability").on(t.sellabilityStatus),
],
);
// ─── Stalker runs ─────────────────────────────────────────────────────────────
export const stalkerRuns = pgTable(
"stalker_runs",
{
id: serial("id").primaryKey(),
inputFile: text("input_file").notNull(),
startedAt: timestamp("started_at", { withTimezone: true }).notNull(),
completedAt: timestamp("completed_at", { withTimezone: true }),
requestedAsins: integer("requested_asins").notNull().default(0),
skippedAsins: integer("skipped_asins").notNull().default(0),
scannedAsins: integer("scanned_asins").notNull().default(0),
sourceAsinsWithMatches: integer("source_asins_with_matches")
.notNull()
.default(0),
candidateSellers: integer("candidate_sellers").notNull().default(0),
qualifyingSellers: integer("qualifying_sellers").notNull().default(0),
matchedSellers: integer("matched_sellers").notNull().default(0),
sellerMetadataRequests: integer("seller_metadata_requests")
.notNull()
.default(0),
sellerStorefrontRequests: integer("seller_storefront_requests")
.notNull()
.default(0),
inventorySellabilityCheckedAsins: integer(
"inventory_sellability_checked_asins",
)
.notNull()
.default(0),
inventorySellabilityAvailableAsins: integer(
"inventory_sellability_available_asins",
)
.notNull()
.default(0),
inventorySellabilityExcludedAsins: integer(
"inventory_sellability_excluded_asins",
)
.notNull()
.default(0),
persistedInventoryAsins: integer("persisted_inventory_asins")
.notNull()
.default(0),
status: text("status").notNull(),
errorMessage: text("error_message"),
},
(t) => [index("idx_stalker_runs_started_at").on(t.startedAt)],
);
// ─── Stalker ASIN scans ───────────────────────────────────────────────────────
export const stalkerAsinScans = pgTable(
"stalker_asin_scans",
export const runItems = pgTable(
"run_items",
{
id: serial("id").primaryKey(),
runId: integer("run_id")
.notNull()
.references(() => stalkerRuns.id, { onDelete: "cascade" }),
sourceAsin: text("source_asin").notNull(),
title: text("title"),
offerCount: integer("offer_count").notNull().default(0),
candidateSellerCount: integer("candidate_seller_count")
.references(() => runs.id, { onDelete: "cascade" }),
productAsin: text("product_asin").references(() => products.asin),
sourceInventoryItemId: integer("source_inventory_item_id").references(
(): AnyPgColumn => stalkerInventoryItems.id,
{ onDelete: "set null" },
),
ordinal: integer("ordinal"),
sourceRow: integer("source_row"),
status: text("status").notNull().default("completed"),
createdAt: timestamp("created_at", { withTimezone: true })
.notNull()
.default(0),
matchedSellerCount: integer("matched_seller_count").notNull().default(0),
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
rawProductJson: text("raw_product_json"),
.defaultNow(),
},
(t) => [
unique("uq_stalker_scans_run_asin").on(t.runId, t.sourceAsin),
index("idx_stalker_scans_run_id").on(t.runId),
index("idx_stalker_scans_source_asin").on(t.sourceAsin),
index("idx_run_items_run_id").on(t.runId),
index("idx_run_items_product_asin").on(t.productAsin),
],
);
// ─── Sellers ──────────────────────────────────────────────────────────────────
// General seller registry (was stalker_sellers).
export const sourcingInputs = pgTable("sourcing_inputs", {
runItemId: integer("run_item_id")
.primaryKey()
.references(() => runItems.id, { onDelete: "cascade" }),
suppliedName: text("supplied_name"),
suppliedBrand: text("supplied_brand"),
suppliedCategory: text("supplied_category"),
unitCost: real("unit_cost"),
avgPrice90dSheet: real("avg_price_90d_sheet"),
sellingPriceSheet: real("selling_price_sheet"),
fbaNetSheet: real("fba_net_sheet"),
grossProfitDollar: real("gross_profit_dollar"),
grossProfitPct: real("gross_profit_pct"),
netProfitSheet: real("net_profit_sheet"),
roiSheet: real("roi_sheet"),
moq: integer("moq"),
moqCost: real("moq_cost"),
qtyAvailable: integer("qty_available"),
supplier: text("supplier"),
sourceUrl: text("source_url"),
asinLink: text("asin_link"),
promoCouponCode: text("promo_coupon_code"),
notes: text("notes"),
leadDate: text("lead_date"),
});
export const upcResolutions = pgTable(
"upc_resolutions",
{
runItemId: integer("run_item_id")
.primaryKey()
.references(() => runItems.id, { onDelete: "cascade" }),
requestedUpc: text("requested_upc").notNull(),
normalizedUpc: text("normalized_upc").notNull(),
provider: text("provider").notNull(),
status: text("status").notNull(),
reason: text("reason"),
resolvedProductAsin: text("resolved_product_asin").references(
() => products.asin,
),
resolvedAt: timestamp("resolved_at", { withTimezone: true })
.notNull()
.defaultNow(),
},
(t) => [index("idx_upc_resolutions_normalized_upc").on(t.normalizedUpc)],
);
export const upcResolutionCandidates = pgTable(
"upc_resolution_candidates",
{
runItemId: integer("run_item_id")
.notNull()
.references(() => upcResolutions.runItemId, { onDelete: "cascade" }),
productAsin: text("product_asin")
.notNull()
.references(() => products.asin),
},
(t) => [
primaryKey({ columns: [t.runItemId, t.productAsin] }),
index("idx_upc_candidates_product_asin").on(t.productAsin),
],
);
export const analysisRevisions = pgTable(
"analysis_revisions",
{
id: serial("id").primaryKey(),
runItemId: integer("run_item_id")
.notNull()
.references(() => runItems.id, { onDelete: "cascade" }),
observationId: integer("observation_id").references(
() => productObservations.id,
{ onDelete: "set null" },
),
method: analysisMethodEnum("method").notNull(),
decision: analysisDecisionEnum("decision").notNull(),
confidence: real("confidence"),
reasoning: text("reasoning"),
analyzedAt: timestamp("analyzed_at", { withTimezone: true })
.notNull()
.defaultNow(),
},
(t) => [
index("idx_analysis_revisions_run_item_time").on(t.runItemId, t.analyzedAt),
index("idx_analysis_revisions_decision").on(t.decision),
],
);
export const supplierScores = pgTable("supplier_scores", {
revisionId: integer("revision_id")
.primaryKey()
.references(() => analysisRevisions.id, { onDelete: "cascade" }),
score: real("score"),
salePrice: real("sale_price"),
fbaFee: real("fba_fee"),
profit: real("profit"),
margin: real("margin"),
roi: real("roi"),
reason: text("reason"),
});
export const sellers = pgTable("sellers", {
sellerId: text("seller_id").primaryKey(),
@@ -276,15 +356,44 @@ export const sellers = pgTable("sellers", {
rawSellerJson: text("raw_seller_json"),
});
// ─── Stalker ASIN sellers ─────────────────────────────────────────────────────
export const stalkerScans = pgTable(
"stalker_scans",
{
id: serial("id").primaryKey(),
runId: integer("run_id")
.notNull()
.references(() => runs.id, { onDelete: "cascade" }),
sourceProductAsin: text("source_product_asin")
.notNull()
.references(() => products.asin),
observationId: integer("observation_id").references(
() => productObservations.id,
{ onDelete: "set null" },
),
offerCount: integer("offer_count").notNull().default(0),
candidateSellerCount: integer("candidate_seller_count")
.notNull()
.default(0),
matchedSellerCount: integer("matched_seller_count").notNull().default(0),
fetchedAt: timestamp("fetched_at", { withTimezone: true }).notNull(),
},
(t) => [
unique("uq_stalker_scans_run_source_product").on(
t.runId,
t.sourceProductAsin,
),
index("idx_stalker_scans_run_id").on(t.runId),
index("idx_stalker_scans_source_asin").on(t.sourceProductAsin),
],
);
export const stalkerAsinSellers = pgTable(
"stalker_asin_sellers",
export const stalkerScanSellers = pgTable(
"stalker_scan_sellers",
{
id: serial("id").primaryKey(),
scanId: integer("scan_id")
.notNull()
.references(() => stalkerAsinScans.id, { onDelete: "cascade" }),
.references(() => stalkerScans.id, { onDelete: "cascade" }),
sellerId: text("seller_id")
.notNull()
.references(() => sellers.sellerId),
@@ -297,47 +406,36 @@ export const stalkerAsinSellers = pgTable(
rawOfferJson: text("raw_offer_json"),
},
(t) => [
unique("uq_stalker_asin_sellers_scan_seller").on(t.scanId, t.sellerId),
unique("uq_stalker_scan_sellers_scan_seller").on(t.scanId, t.sellerId),
],
);
// ─── Stalker seller inventory ─────────────────────────────────────────────────
export const stalkerSellerInventory = pgTable(
"stalker_seller_inventory",
export const stalkerInventoryItems = pgTable(
"stalker_inventory_items",
{
id: serial("id").primaryKey(),
runId: integer("run_id")
.notNull()
.references(() => stalkerRuns.id, { onDelete: "cascade" }),
.references(() => runs.id, { onDelete: "cascade" }),
sellerId: text("seller_id")
.notNull()
.references(() => sellers.sellerId),
asin: text("asin").notNull(),
canSell: boolean("can_sell"),
sellabilityStatus: text("sellability_status"),
sellabilityReason: text("sellability_reason"),
productTitle: text("product_title"),
brand: text("brand"),
categoryTree: text("category_tree"),
currentPrice: real("current_price"),
avgPrice90d: real("avg_price_90d"),
salesRank: integer("sales_rank"),
monthlySold: integer("monthly_sold"),
sellerCount: integer("seller_count"),
amazonIsSeller: boolean("amazon_is_seller"),
rawProductJson: text("raw_product_json"),
productAsin: text("product_asin")
.notNull()
.references(() => products.asin),
observationId: integer("observation_id")
.notNull()
.references(() => productObservations.id, { onDelete: "cascade" }),
lastSeenAt: timestamp("last_seen_at", { withTimezone: true }).notNull(),
rawInventoryJson: text("raw_inventory_json"),
},
(t) => [
unique("uq_stalker_inventory_run_seller_asin").on(
unique("uq_stalker_inventory_items_run_seller_asin").on(
t.runId,
t.sellerId,
t.asin,
t.productAsin,
),
index("idx_stalker_inventory_seller_id").on(t.sellerId),
index("idx_stalker_inventory_asin").on(t.asin),
index("idx_stalker_inventory_product_title").on(t.productTitle),
index("idx_stalker_inventory_product_asin").on(t.productAsin),
],
);

View File

@@ -42,7 +42,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
JSON.stringify({
products: [
{
asin: "B000FOUND01",
asin: "B000FND001",
upcList: ["012345678901"],
stats: {
current: [null, null, null, 1234],
@@ -51,7 +51,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
csv: [[5000000, 2999, 5000100]],
},
{
asin: "B000MULTI01",
asin: "B000MUL001",
upcList: ["098765432109"],
stats: {
current: [null, null, null, 2000],
@@ -60,7 +60,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
csv: [[1, 1999]],
},
{
asin: "B000MULTI02",
asin: "B000MUL002",
upcList: ["098765432109"],
stats: {
current: [null, null, null, 2100],
@@ -83,14 +83,14 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
]);
expect(details.get("012345678901")?.status).toBe("found");
expect(details.get("012345678901")?.asin).toBe("B000FOUND01");
expect(details.get("012345678901")?.asin).toBe("B000FND001");
expect(details.get("012345678901")?.keepaData?.currentPrice).toBe(29.99);
expect(details.get("012345678901")?.keepaData?.currentPrice).toBe(29.99);
expect(details.get("098765432109")?.status).toBe("multiple_asins");
expect(details.get("098765432109")?.candidateAsins).toEqual([
"B000MULTI01",
"B000MULTI02",
"B000MUL001",
"B000MUL002",
]);
expect(details.get("111111111111")?.status).toBe("not_found");
@@ -100,7 +100,7 @@ test("lookupKeepaUpcs returns found, not_found, and multiple_asins outcomes", as
"098765432109",
"111111111111",
]);
expect(simpleMap.get("012345678901")).toBe("B000FOUND01");
expect(simpleMap.get("012345678901")).toBe("B000FND001");
expect(simpleMap.has("098765432109")).toBe(false);
expect(simpleMap.has("111111111111")).toBe(false);
});
@@ -128,7 +128,7 @@ test("lookupKeepaUpcs keeps partial success when one chunk fails", async () => {
JSON.stringify({
products: [
{
asin: "B000LAST001",
asin: "B000LST001",
upcList: [secondChunkUpc],
stats: {
current: [null, null, null, 1000],
@@ -148,11 +148,11 @@ test("lookupKeepaUpcs keeps partial success when one chunk fails", async () => {
expect(details.get(firstChunkFirstUpc)?.status).toBe("request_failed");
expect(details.get(secondChunkUpc)?.status).toBe("found");
expect(details.get(secondChunkUpc)?.asin).toBe("B000LAST001");
expect(details.get(secondChunkUpc)?.asin).toBe("B000LST001");
const simpleMap = await mapUpcsToAsins(upcs);
expect(simpleMap.has(firstChunkFirstUpc)).toBe(false);
expect(simpleMap.get(secondChunkUpc)).toBe("B000LAST001");
expect(simpleMap.get(secondChunkUpc)).toBe("B000LST001");
});
test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () => {
@@ -175,7 +175,7 @@ test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () =
JSON.stringify({
products: [
{
asin: "B000RETRY01",
asin: "B000RTY001",
upcList: [targetUpc],
stats: {
current: [null, null, null, 1111],
@@ -197,7 +197,7 @@ test("lookupKeepaUpcs retries on 429 and succeeds after refill wait", async () =
expect(fetchMock.mock.calls.length).toBe(2);
expect(details.get(targetUpc)?.status).toBe("found");
expect(details.get(targetUpc)?.asin).toBe("B000RETRY01");
expect(details.get(targetUpc)?.asin).toBe("B000RTY001");
});
test("lookupKeepaUpcs uses lightweight query params for code mapping", async () => {
@@ -220,7 +220,7 @@ test("lookupKeepaUpcs uses lightweight query params for code mapping", async ()
JSON.stringify({
products: [
{
asin: "B000LIGHT01",
asin: "B000LGT001",
upcList: [targetUpc],
categoryTree: [{ name: "Test Category" }],
},
@@ -238,5 +238,5 @@ test("lookupKeepaUpcs uses lightweight query params for code mapping", async ()
expect(fetchMock.mock.calls.length).toBe(1);
expect(details.get(targetUpc)?.status).toBe("found");
expect(details.get(targetUpc)?.asin).toBe("B000LIGHT01");
expect(details.get(targetUpc)?.asin).toBe("B000LGT001");
});

View File

@@ -1,4 +1,5 @@
import { config } from "../config.ts";
import { normalizeAsin } from "../asin.ts";
import type { KeepaData, KeepaUpcLookupDetail } from "../types.ts";
const KEEPA_BASE = "https://api.keepa.com";
@@ -228,10 +229,17 @@ export async function fetchKeepaDataBatch(
asins: string[],
): Promise<Map<string, KeepaData>> {
const results = new Map<string, KeepaData>();
const canonicalAsins = Array.from(
new Set(
asins
.map((asin) => normalizeAsin(asin))
.filter((asin): asin is string => asin !== null),
),
);
// Split into chunks of MAX_ASINS_PER_REQUEST
for (let i = 0; i < asins.length; i += MAX_ASINS_PER_REQUEST) {
const chunk = asins.slice(i, i + MAX_ASINS_PER_REQUEST);
for (let i = 0; i < canonicalAsins.length; i += MAX_ASINS_PER_REQUEST) {
const chunk = canonicalAsins.slice(i, i + MAX_ASINS_PER_REQUEST);
const url = buildProductUrl("asin", chunk, {
includeStats: true,
includeBuybox: true,
@@ -250,7 +258,7 @@ export async function fetchKeepaDataBatch(
if (data.products) {
for (const product of data.products) {
const asin = product.asin;
const asin = normalizeAsin(product.asin);
if (!asin) continue;
results.set(asin, parseKeepaProduct(product));
}
@@ -309,7 +317,7 @@ export async function lookupKeepaUpcs(
const byUpc = new Map<string, Map<string, KeepaData>>();
for (const product of data.products ?? []) {
const asin = String(product.asin ?? "").trim();
const asin = normalizeAsin(product.asin);
if (!asin) continue;
const keepaData = parseKeepaProduct(product);

View File

@@ -13,6 +13,7 @@ afterAll(() => {
test("normalizeAsin uppercases and validates ASINs", () => {
expect(normalizeAsin(" b07sn9bhvv ")).toBe("B07SN9BHVV");
expect(normalizeAsin("0306406152")).toBe("0306406152");
expect(() => normalizeAsin("not-an-asin")).toThrow("Invalid ASIN");
});

View File

@@ -1,10 +1,11 @@
import { normalizeAsin as normalizeCanonicalAsin } from "../asin.ts";
const DEFAULT_SEARXNG_URL = "https://searxng.nvictor.me/";
const DEFAULT_GOOGLE_CUSTOM_SEARCH_URL =
"https://www.googleapis.com/customsearch/v1";
const DEFAULT_SERPAPI_URL = "https://serpapi.com/search.json";
const DEFAULT_TIMEOUT_MS = 10_000;
const DEFAULT_MAX_RESULTS = 10;
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
const ASIN_MATCH_REGEX = /\bB[0-9A-Z]{9}\b/gi;
const PRICE_LABELS = [
"selling price",
@@ -127,16 +128,15 @@ export async function searchProductOffers(
}
export function normalizeAsin(value: string): string {
const asin = value.trim().toUpperCase();
if (!ASIN_REGEX.test(asin)) {
const asin = normalizeCanonicalAsin(value);
if (!asin) {
throw new Error(`Invalid ASIN: ${value}`);
}
return asin;
}
function getAsinQuery(value: string): string | undefined {
const normalized = value.trim().toUpperCase();
return ASIN_REGEX.test(normalized) ? normalized : undefined;
return normalizeCanonicalAsin(value) ?? undefined;
}
async function fetchSearxngResults(

View File

@@ -20,6 +20,15 @@ test("parseCatalogUpcLookupResponse marks no match", () => {
expect(detail.asin).toBeNull();
});
test("parseCatalogUpcLookupResponse ignores invalid ASIN identifiers", () => {
const detail = parseCatalogUpcLookupResponse("012345678901", {
items: [{ asin: "012345678901" }],
});
expect(detail.status).toBe("not_found");
expect(detail.asin).toBeNull();
});
test("parseCatalogUpcLookupResponse marks multiple ASINs", () => {
const detail = parseCatalogUpcLookupResponse("012345678901", {
payload: {

View File

@@ -1,4 +1,5 @@
import { SellingPartner } from "amazon-sp-api";
import { normalizeAsin } from "../asin.ts";
import { config } from "../config.ts";
import type {
KeepaUpcLookupStatus,
@@ -222,8 +223,7 @@ function extractCatalogAsin(item: any): string | null {
item?.identifiers?.marketplaceASIN?.asin ??
item?.Identifiers?.MarketplaceASIN?.ASIN;
if (typeof raw !== "string") return null;
const asin = raw.trim().toUpperCase();
return asin ? asin : null;
return normalizeAsin(raw);
}
export function parseCatalogUpcLookupResponse(

View File

@@ -1,8 +1,7 @@
import * as XLSX from "xlsx";
import { normalizeAsin } from "./asin.ts";
import type { ProductRecord } from "./types.ts";
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
const COLUMN_CANDIDATES = {
asin: ["asin"],
name: ["name", "product name", "title", "product title"],
@@ -133,11 +132,9 @@ function getKnownColumns(columns: ColumnMap): Set<string> {
}
function parseAsin(value: unknown): string | undefined {
const asin = String(value ?? "")
.trim()
.toUpperCase();
if (!asin || !ASIN_REGEX.test(asin)) {
console.warn(`Skipping invalid ASIN: "${asin}"`);
const asin = normalizeAsin(value);
if (!asin) {
console.warn(`Skipping invalid ASIN: "${String(value ?? "").trim()}"`);
return undefined;
}
return asin;

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,7 @@
import { db } from "../db/index.ts";
import { categoryProductResults, runs } from "../db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { persistLlmResults, refreshRunStats } from "../db/persistence.ts";
import { sql } from "drizzle-orm";
import { normalizeAsin } from "../asin.ts";
import { analyzeProducts } from "../integrations/llm.ts";
import { fetchSpApiPricingAndFees } from "../integrations/sp-api.ts";
import type {
@@ -22,6 +23,7 @@ type Args = {
};
type InventoryRow = {
inventoryItemId: number;
asin: string;
productTitle: string | null;
brand: string | null;
@@ -49,8 +51,8 @@ function parseArgs(argv = process.argv.slice(2)): Args {
const useClaude = argv.includes("--claude");
const asins = (readFlagValue(argv, "--asins") ?? "")
.split(",")
.map((asin) => asin.trim().toUpperCase())
.filter(Boolean);
.map((asin) => normalizeAsin(asin))
.filter((asin): asin is string => asin !== null);
if (!Number.isInteger(stalkerRunId) || stalkerRunId <= 0) {
throw new Error("--stalker-run-id must be a positive integer");
@@ -68,15 +70,7 @@ function wait(ms: number): Promise<void> {
}
function parseCategoryTree(value: string | null): string[] {
if (!value) return [];
try {
const parsed = JSON.parse(value);
return Array.isArray(parsed)
? parsed.filter((item): item is string => typeof item === "string")
: [];
} catch {
return [];
}
return value ? value.split(" > ").filter(Boolean) : [];
}
function toProductRecord(row: InventoryRow): ProductRecord {
@@ -128,25 +122,29 @@ async function loadInventoryRows(
): Promise<InventoryRow[]> {
if (asins.length === 0) return [];
return db.execute(
sql<InventoryRow>`SELECT DISTINCT ON (asin)
asin,
product_title AS "productTitle",
brand,
category_tree AS "categoryTree",
current_price AS "currentPrice",
avg_price_90d AS "avgPrice90d",
sales_rank AS "salesRank",
monthly_sold AS "monthlySold",
seller_count AS "sellerCount",
amazon_is_seller AS "amazonIsSeller",
can_sell AS "canSell",
sellability_status AS "sellabilityStatus",
sellability_reason AS "sellabilityReason"
FROM stalker_seller_inventory
WHERE run_id = ${stalkerRunId}
AND can_sell = true
AND sellability_status = 'available'
AND asin = ANY(${asins})`,
sql<InventoryRow>`SELECT DISTINCT ON (inventory.product_asin)
inventory.id AS "inventoryItemId",
inventory.product_asin AS asin,
product.name AS "productTitle",
product.brand,
product.category AS "categoryTree",
observation.current_price AS "currentPrice",
observation.avg_price_90d AS "avgPrice90d",
observation.sales_rank AS "salesRank",
observation.monthly_sold AS "monthlySold",
observation.seller_count AS "sellerCount",
observation.amazon_is_seller AS "amazonIsSeller",
observation.can_sell AS "canSell",
observation.sellability_status AS "sellabilityStatus",
observation.sellability_reason AS "sellabilityReason"
FROM stalker_inventory_items inventory
JOIN products product ON product.asin = inventory.product_asin
JOIN product_observations observation ON observation.id = inventory.observation_id
WHERE inventory.run_id = ${stalkerRunId}
AND observation.can_sell = true
AND observation.sellability_status = 'available'
AND inventory.product_asin = ANY(${asins})
ORDER BY inventory.product_asin, observation.fetched_at DESC`,
);
}
@@ -177,111 +175,18 @@ async function buildEnrichedProducts(
async function insertProductAnalysisResults(
runId: number,
results: AnalysisResult[],
sourceInventoryIds: Map<string, number>,
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((result) => {
const keepa = result.product.keepa;
const record = result.product.record;
const spApi = result.product.spApi;
const canSell =
spApi.canSell == null ? "unknown" : spApi.canSell ? "yes" : "no";
return {
asin: record.asin,
runId,
name: record.name,
brand: record.brand ?? null,
category: record.category ?? keepa?.categoryTree.join(" > ") ?? null,
unitCost: record.unitCost ?? null,
currentPrice: keepa?.currentPrice ?? spApi.estimatedSalePrice ?? null,
avgPrice90d: keepa?.avgPrice90 ?? null,
avgPrice90dSheet: record.avgPrice90FromSheet ?? null,
sellingPriceSheet: record.sellingPriceFromSheet ?? null,
salesRank: keepa?.salesRank ?? record.amazonRank ?? null,
salesRankAvg90d: keepa?.salesRankAvg90 ?? null,
sellerCount: keepa?.sellerCount ?? null,
amazonIsSeller: keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
monthlySold: keepa?.monthlySold ?? null,
rankDrops30d: keepa?.salesRankDrops30 ?? null,
rankDrops90d: keepa?.salesRankDrops90 ?? null,
fbaFee: spApi.fbaFee ?? null,
fbmFee: spApi.fbmFee ?? null,
referralPercent: spApi.referralFeePercent ?? null,
canSell,
sellabilityStatus: spApi.sellabilityStatus ?? null,
sellabilityReason: spApi.sellabilityReason ?? null,
verdict: result.verdict.verdict,
confidence: result.verdict.confidence ?? 0,
reasoning: result.verdict.reasoning ?? null,
fetchedAt: new Date(result.product.fetchedAt),
};
await persistLlmResults(runId, results, {
source: "stalker_analysis",
metadataSource: "catalog",
sourceInventoryIds,
});
await db
.insert(categoryProductResults)
.values(rows)
.onConflictDoUpdate({
target: categoryProductResults.asin,
set: {
runId: sql`EXCLUDED.run_id`,
name: sql`EXCLUDED.name`,
brand: sql`EXCLUDED.brand`,
category: sql`EXCLUDED.category`,
unitCost: sql`EXCLUDED.unit_cost`,
currentPrice: sql`EXCLUDED.current_price`,
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
avgPrice90dSheet: sql`EXCLUDED.avg_price_90d_sheet`,
sellingPriceSheet: sql`EXCLUDED.selling_price_sheet`,
salesRank: sql`EXCLUDED.sales_rank`,
salesRankAvg90d: sql`EXCLUDED.sales_rank_avg_90d`,
sellerCount: sql`EXCLUDED.seller_count`,
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
amazonBuyboxSharePct90d: sql`EXCLUDED.amazon_buybox_share_pct_90d`,
monthlySold: sql`EXCLUDED.monthly_sold`,
rankDrops30d: sql`EXCLUDED.rank_drops_30d`,
rankDrops90d: sql`EXCLUDED.rank_drops_90d`,
fbaFee: sql`EXCLUDED.fba_fee`,
fbmFee: sql`EXCLUDED.fbm_fee`,
referralPercent: sql`EXCLUDED.referral_percent`,
canSell: sql`EXCLUDED.can_sell`,
sellabilityStatus: sql`EXCLUDED.sellability_status`,
sellabilityReason: sql`EXCLUDED.sellability_reason`,
verdict: sql`EXCLUDED.verdict`,
confidence: sql`EXCLUDED.confidence`,
reasoning: sql`EXCLUDED.reasoning`,
fetchedAt: sql`EXCLUDED.fetched_at`,
},
});
}
async function refreshAnalysisRun(runId: number): Promise<void> {
const [stats] = await db.execute(
sql<{
total: string;
fba: string | null;
fbm: string | null;
skip: string | null;
}>`SELECT
COUNT(*) AS total,
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
FROM category_product_results
WHERE run_id = ${runId}`,
);
await db
.update(runs)
.set({
topAsinsChecked: Number(stats?.total ?? 0),
availableAsins: Number(stats?.total ?? 0),
fbaCount: Number(stats?.fba ?? 0),
fbmCount: Number(stats?.fbm ?? 0),
skipCount: Number(stats?.skip ?? 0),
})
.where(eq(runs.id, runId));
await refreshRunStats(runId);
}
async function analyzeInBatches(
@@ -344,7 +249,14 @@ async function main(): Promise<void> {
console.log(`Stalker analysis: analyzing ${rows.length} sellable ASIN(s).`);
const enriched = await buildEnrichedProducts(rows);
const results = await analyzeInBatches(enriched, args.useClaude);
await insertProductAnalysisResults(args.analysisRunId, results);
const sourceInventoryIds = new Map(
rows.map((row) => [row.asin, row.inventoryItemId]),
);
await insertProductAnalysisResults(
args.analysisRunId,
results,
sourceInventoryIds,
);
await refreshAnalysisRun(args.analysisRunId);
}

View File

@@ -101,12 +101,17 @@ test("readAsinsFromXlsx extracts valid ASINs and deduplicates in order", () => {
{ ASIN: "invalid" },
{ ASIN: "B000000002" },
{ ASIN: "B000000001" },
{ ASIN: "0306406152" },
{ ASIN: "" },
]);
XLSX.utils.book_append_sheet(workbook, sheet, "Input");
XLSX.writeFile(workbook, filePath);
expect(readAsinsFromXlsx(filePath)).toEqual(["B000000001", "B000000002"]);
expect(readAsinsFromXlsx(filePath)).toEqual([
"B000000001",
"B000000002",
"0306406152",
]);
});
test("isQualifyingSeller accepts rating counts from 1 to 30 only", () => {

View File

@@ -1,13 +1,17 @@
import * as XLSX from "xlsx";
import path from "node:path";
import { normalizeAsin } from "../asin.ts";
import { db } from "../db/index.ts";
import { refreshRunStats, upsertProduct } from "../db/persistence.ts";
import {
analysisRunStats,
productObservations,
runs,
stalkerRuns,
stalkerAsinScans,
stalkerRunDetails,
stalkerScans,
sellers,
stalkerAsinSellers,
stalkerSellerInventory,
stalkerScanSellers,
stalkerInventoryItems,
} from "../db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { fetchSellabilityBatch } from "../integrations/sp-api.ts";
@@ -16,7 +20,6 @@ import type { SellabilityInfo } from "../types.ts";
const KEEPA_BASE = "https://api.keepa.com";
const DOMAIN_US = "1";
const AMAZON_US_SELLER_ID = "ATVPDKIKX0DER";
const ASIN_REGEX = /^B[0-9A-Z]{9}$/;
const DEFAULT_STOREFRONT_UPDATE_HOURS = 168;
const DEFAULT_OFFER_LIMIT = 100;
const DEFAULT_SELLER_LIMIT = 30;
@@ -333,7 +336,7 @@ export async function runStalker(args: StalkerArgs, deps: StalkerDeps = {}): Pro
: await startStalkerRun(args.input, resumeFilteredAsins.length);
const analysisRunId =
!args.dryRun && args.analyzeSellable
? await startStalkerAnalysisRun(args.input)
? await startStalkerAnalysisRun(args.input, runId!)
: null;
const stats: StalkerRunStats = {
scannedAsins: 0,
@@ -841,11 +844,12 @@ async function persistAsinResult(
await db.transaction(async (tx) => {
const scanId = await upsertAsinScan(tx, runId, result, fetchedAt);
const observationIds = new Map<string, number>();
for (const { seller, offer } of result.matchedSellers) {
await upsertSeller(tx, seller, fetchedAt);
await upsertAsinSeller(tx, scanId, seller, offer);
await upsertSellerInventory(tx, runId, seller, fetchedAt);
await upsertSellerInventory(tx, runId, seller, fetchedAt, observationIds);
}
});
}
@@ -856,37 +860,58 @@ async function upsertAsinScan(
result: StalkerAsinResult,
fetchedAt: Date,
): Promise<number> {
await tx
.insert(stalkerAsinScans)
const sourceProductAsin = await upsertProduct(
{
asin: result.asin,
name: result.title,
metadataSource: "catalog",
fetchedAt,
},
tx,
);
const [observation] = await tx
.insert(productObservations)
.values({
productAsin: sourceProductAsin,
runId,
sourceAsin: result.asin,
title: result.title,
offerCount: result.offerCount,
candidateSellerCount: result.candidateSellerCount,
matchedSellerCount: result.matchedSellers.length,
source: "stalker_scan",
fetchedAt,
rawProductJson: JSON.stringify(
result.product ?? { error: result.error ?? null },
),
})
.returning({ id: productObservations.id });
if (!observation) {
throw new Error(`Failed to insert stalker observation for ${result.asin}`);
}
await tx
.insert(stalkerScans)
.values({
runId,
sourceProductAsin,
observationId: observation.id,
offerCount: result.offerCount,
candidateSellerCount: result.candidateSellerCount,
matchedSellerCount: result.matchedSellers.length,
fetchedAt,
})
.onConflictDoUpdate({
target: [stalkerAsinScans.runId, stalkerAsinScans.sourceAsin],
target: [stalkerScans.runId, stalkerScans.sourceProductAsin],
set: {
title: sql`EXCLUDED.title`,
observationId: sql`EXCLUDED.observation_id`,
offerCount: sql`EXCLUDED.offer_count`,
candidateSellerCount: sql`EXCLUDED.candidate_seller_count`,
matchedSellerCount: sql`EXCLUDED.matched_seller_count`,
fetchedAt: sql`EXCLUDED.fetched_at`,
rawProductJson: sql`EXCLUDED.raw_product_json`,
},
});
const [row] = await tx
.select({ id: stalkerAsinScans.id })
.from(stalkerAsinScans)
.select({ id: stalkerScans.id })
.from(stalkerScans)
.where(
sql`${stalkerAsinScans.runId} = ${runId} AND ${stalkerAsinScans.sourceAsin} = ${result.asin}`,
sql`${stalkerScans.runId} = ${runId} AND ${stalkerScans.sourceProductAsin} = ${sourceProductAsin}`,
);
if (!row)
throw new Error(`Failed to load stalker scan row for ${result.asin}`);
@@ -931,7 +956,7 @@ async function upsertAsinSeller(
offer: StalkerOffer,
): Promise<void> {
await tx
.insert(stalkerAsinSellers)
.insert(stalkerScanSellers)
.values({
scanId,
sellerId: seller.sellerId,
@@ -944,7 +969,7 @@ async function upsertAsinSeller(
rawOfferJson: JSON.stringify(offer.rawOffer),
})
.onConflictDoUpdate({
target: [stalkerAsinSellers.scanId, stalkerAsinSellers.sellerId],
target: [stalkerScanSellers.scanId, stalkerScanSellers.sellerId],
set: {
offerPrice: sql`EXCLUDED.offer_price`,
condition: sql`EXCLUDED.condition`,
@@ -962,6 +987,7 @@ async function upsertSellerInventory(
runId: number,
seller: StalkerSeller,
fetchedAt: Date,
observationIds: Map<string, number>,
): Promise<void> {
const items = seller.storefrontItems.filter(
(item) =>
@@ -971,58 +997,71 @@ async function upsertSellerInventory(
if (items.length === 0) return;
await tx
.insert(stalkerSellerInventory)
.values(
items.map((item) => ({
for (const item of items) {
let observationId = observationIds.get(item.asin);
if (observationId == null) {
const productAsin = await upsertProduct(
{
asin: item.asin,
name: item.productDetails?.title,
brand: item.productDetails?.brand,
category: item.productDetails?.categoryTree.join(" > "),
metadataSource: "catalog",
fetchedAt,
},
tx,
);
const [observation] = await tx
.insert(productObservations)
.values({
productAsin,
runId,
source: "stalker_inventory",
canSell: item.sellability?.canSell ?? null,
sellabilityStatus: item.sellability?.sellabilityStatus ?? null,
sellabilityReason: item.sellability?.sellabilityReason ?? null,
currentPrice: item.productDetails?.currentPrice ?? null,
avgPrice90d: item.productDetails?.avgPrice90 ?? null,
salesRank: item.productDetails?.salesRank ?? null,
monthlySold: item.productDetails?.monthlySold ?? null,
sellerCount: item.productDetails?.sellerCount ?? null,
amazonIsSeller: item.productDetails?.amazonIsSeller ?? null,
rawProductJson: item.productDetails
? JSON.stringify(item.productDetails.rawProduct)
: null,
fetchedAt,
})
.returning({ id: productObservations.id });
if (!observation) {
throw new Error(`Failed to insert inventory observation for ${item.asin}`);
}
observationId = observation.id;
observationIds.set(item.asin, observationId);
}
await tx
.insert(stalkerInventoryItems)
.values({
runId,
sellerId: seller.sellerId,
asin: item.asin,
canSell: item.sellability?.canSell ?? null,
sellabilityStatus: item.sellability?.sellabilityStatus ?? null,
sellabilityReason: item.sellability?.sellabilityReason ?? null,
productTitle: item.productDetails?.title ?? null,
brand: item.productDetails?.brand ?? null,
categoryTree: item.productDetails
? JSON.stringify(item.productDetails.categoryTree)
: null,
currentPrice: item.productDetails?.currentPrice ?? null,
avgPrice90d: item.productDetails?.avgPrice90 ?? null,
salesRank: item.productDetails?.salesRank ?? null,
monthlySold: item.productDetails?.monthlySold ?? null,
sellerCount: item.productDetails?.sellerCount ?? null,
amazonIsSeller: item.productDetails?.amazonIsSeller ?? null,
rawProductJson: item.productDetails
? JSON.stringify(item.productDetails.rawProduct)
: null,
productAsin: item.asin,
observationId,
lastSeenAt: fetchedAt,
rawInventoryJson: JSON.stringify(item.rawInventory),
})),
)
.onConflictDoUpdate({
target: [
stalkerSellerInventory.runId,
stalkerSellerInventory.sellerId,
stalkerSellerInventory.asin,
],
set: {
canSell: sql`EXCLUDED.can_sell`,
sellabilityStatus: sql`EXCLUDED.sellability_status`,
sellabilityReason: sql`EXCLUDED.sellability_reason`,
productTitle: sql`EXCLUDED.product_title`,
brand: sql`EXCLUDED.brand`,
categoryTree: sql`EXCLUDED.category_tree`,
currentPrice: sql`EXCLUDED.current_price`,
avgPrice90d: sql`EXCLUDED.avg_price_90d`,
salesRank: sql`EXCLUDED.sales_rank`,
monthlySold: sql`EXCLUDED.monthly_sold`,
sellerCount: sql`EXCLUDED.seller_count`,
amazonIsSeller: sql`EXCLUDED.amazon_is_seller`,
rawProductJson: sql`EXCLUDED.raw_product_json`,
lastSeenAt: sql`EXCLUDED.last_seen_at`,
rawInventoryJson: sql`EXCLUDED.raw_inventory_json`,
},
});
})
.onConflictDoUpdate({
target: [
stalkerInventoryItems.runId,
stalkerInventoryItems.sellerId,
stalkerInventoryItems.productAsin,
],
set: {
observationId: sql`EXCLUDED.observation_id`,
lastSeenAt: sql`EXCLUDED.last_seen_at`,
rawInventoryJson: sql`EXCLUDED.raw_inventory_json`,
},
});
}
}
async function startStalkerRun(
@@ -1030,42 +1069,45 @@ async function startStalkerRun(
totalAsins: number,
): Promise<number> {
const [row] = await db
.insert(stalkerRuns)
.insert(runs)
.values({
type: "stalker",
inputFile,
startedAt: new Date(),
requestedAsins: totalAsins,
status: "running",
})
.returning({ id: stalkerRuns.id });
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert stalker run record.");
await db.insert(stalkerRunDetails).values({
runId: row.id,
requestedAsins: totalAsins,
});
return row.id;
}
async function startStalkerAnalysisRun(inputFile: string): Promise<number> {
async function startStalkerAnalysisRun(
inputFile: string,
parentRunId: number,
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "category_analysis",
categoryId: 0,
categoryLabel: `Stalker: ${path.basename(inputFile)}`,
topAsinsChecked: 0,
availableAsins: 0,
fbaCount: 0,
fbmCount: 0,
skipCount: 0,
type: "stalker_analysis",
parentRunId,
inputFile: `Stalker: ${path.basename(inputFile)}`,
status: "running",
startedAt: new Date(),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert stalker analysis run record.");
await db.insert(analysisRunStats).values({ runId: row.id });
return row.id;
}
async function loadPreviouslyScannedAsins(): Promise<Set<string>> {
const rows = await db
.selectDistinct({ sourceAsin: stalkerAsinScans.sourceAsin })
.from(stalkerAsinScans);
.selectDistinct({ sourceAsin: stalkerScans.sourceProductAsin })
.from(stalkerScans);
return new Set(rows.map((row) => row.sourceAsin));
}
@@ -1133,8 +1175,9 @@ async function refreshStalkerRun(
status: string,
): Promise<void> {
await db
.update(stalkerRuns)
.update(stalkerRunDetails)
.set({
skippedAsins: stats.skippedAsins,
scannedAsins: stats.scannedAsins,
sourceAsinsWithMatches: stats.sourceAsinsWithMatches,
candidateSellers: stats.candidateSellers,
@@ -1147,10 +1190,15 @@ async function refreshStalkerRun(
stats.inventorySellabilityAvailableAsins,
inventorySellabilityExcludedAsins: stats.inventorySellabilityExcludedAsins,
persistedInventoryAsins: stats.persistedInventoryAsins,
status,
})
.where(eq(stalkerRunDetails.runId, runId));
await db
.update(runs)
.set({
status: status === "running" ? "running" : "completed",
...(status !== "running" ? { completedAt: new Date() } : {}),
})
.where(eq(stalkerRuns.id, runId));
.where(eq(runs.id, runId));
}
async function finishStalkerRunWithError(
@@ -1159,8 +1207,9 @@ async function finishStalkerRunWithError(
errorMessage: string,
): Promise<void> {
await db
.update(stalkerRuns)
.update(stalkerRunDetails)
.set({
skippedAsins: stats.skippedAsins,
scannedAsins: stats.scannedAsins,
sourceAsinsWithMatches: stats.sourceAsinsWithMatches,
candidateSellers: stats.candidateSellers,
@@ -1173,11 +1222,16 @@ async function finishStalkerRunWithError(
stats.inventorySellabilityAvailableAsins,
inventorySellabilityExcludedAsins: stats.inventorySellabilityExcludedAsins,
persistedInventoryAsins: stats.persistedInventoryAsins,
})
.where(eq(stalkerRunDetails.runId, runId));
await db
.update(runs)
.set({
status: "failed",
errorMessage,
completedAt: new Date(),
})
.where(eq(stalkerRuns.id, runId));
.where(eq(runs.id, runId));
}
async function finishStalkerAnalysisRun(
@@ -1185,29 +1239,10 @@ async function finishStalkerAnalysisRun(
status: "completed" | "failed",
errorMessage: string | null = null,
): Promise<void> {
const [stats] = await db.execute(
sql<{
total: string;
fba: string | null;
fbm: string | null;
skip: string | null;
}>`SELECT
COUNT(*) AS total,
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
FROM category_product_results
WHERE run_id = ${runId}`,
);
await refreshRunStats(runId);
await db
.update(runs)
.set({
topAsinsChecked: Number(stats?.total ?? 0),
availableAsins: Number(stats?.total ?? 0),
fbaCount: Number(stats?.fba ?? 0),
fbmCount: Number(stats?.fbm ?? 0),
skipCount: Number(stats?.skip ?? 0),
status,
errorMessage,
completedAt: new Date(),
@@ -1480,13 +1515,6 @@ async function runSellableAnalysisChild(
}
}
function normalizeAsin(value: unknown): string | null {
const asin = String(value ?? "")
.trim()
.toUpperCase();
return ASIN_REGEX.test(asin) ? asin : null;
}
function normalizeSellerId(value: unknown): string | null {
const sellerId = String(value ?? "")
.trim()

View File

@@ -16,12 +16,12 @@ function result(overrides: Partial<SupplierAnalysisResult> = {}): SupplierAnalys
upc: "012345678901",
rowNumber: 2,
record: {
asin: "B000000001",
name: "Test Product",
unitCost: 10,
brand: "Brand",
category: "Grocery",
},
product: { asin: "B000000001", name: "Test Product", unitCost: 10 },
lookup: {
requestedUpc: "012345678901",
normalizedUpc: "012345678901",
@@ -81,7 +81,8 @@ test("writeSupplierWorkbook writes ranked, skipped, and summary sheets", async (
result(),
result({
upc: "111111111111",
record: { asin: "111111111111", name: "Missing", unitCost: 0 },
record: { name: "Missing", unitCost: 0 },
product: null,
lookup: {
requestedUpc: "111111111111",
normalizedUpc: "111111111111",

View File

@@ -63,7 +63,8 @@ function addRowsSheet(
const sheet = workbook.addWorksheet(name);
const headers = rows[0] ? Object.keys(rows[0]) : Object.keys(rowForResult({
upc: "",
record: { asin: "", name: "", unitCost: 0 },
record: { name: "", unitCost: 0 },
product: null,
lookup: {
requestedUpc: "",
normalizedUpc: "",

View File

@@ -1,4 +1,5 @@
import path from "node:path";
import { requireAsin } from "../asin.ts";
import { fetchKeepaDataBatch, lookupKeepaUpcs } from "../integrations/keepa.ts";
import {
fetchSellabilityBatch,
@@ -11,6 +12,8 @@ import {
} from "./upc-file-reader.ts";
import {
appendSupplierResultsToRun,
completeRunInDb,
failRunInDb,
refreshRunCountsInDb,
startRunInDb,
type RunCounts,
@@ -239,8 +242,8 @@ async function lookupUpcsWithChunking(
chunkDetails.set(
upc,
fallbackDetail && fallbackDetail.status !== "request_failed"
? fallbackDetail
: spDetail!,
? { ...fallbackDetail, provider: "keepa" }
: { ...spDetail!, provider: "sp_api" },
);
}
@@ -266,7 +269,7 @@ function toProductRecord(
const keepaCategory = detail.keepaData?.categoryTree?.[0];
return {
asin: detail.asin ?? row.upc,
asin: requireAsin(detail.asin),
name: row.name ?? detail.asin ?? row.upc,
unitCost: row.unitCost ?? 0,
brand: row.brand,
@@ -274,6 +277,15 @@ function toProductRecord(
};
}
function toSupplierInputRecord(row: UpcInputRow) {
return {
name: row.name ?? row.upc,
unitCost: row.unitCost ?? 0,
brand: row.brand,
category: row.category,
};
}
async function fetchFeesForProducts(
products: ProductRecord[],
keepaResults: Map<string, NonNullable<SupplierAnalysisResult["keepa"]>>,
@@ -359,7 +371,7 @@ export async function runUpcFileAnalysis(
let processedRows = 0;
let matchedRows = 0;
const runId = await startRunInDb(options.inputFile, outputFile);
const runId = await startRunInDb(options.inputFile, outputFile, undefined, "supplier_upc");
try {
const readerSummary = await processUpcFileInBatches(
@@ -382,12 +394,19 @@ export async function runUpcFileAnalysis(
product: ProductRecord;
}> = [];
for (const row of rows) {
const detail = detailMap.get(row.upc);
if (!detail) {
unresolvedByStatus.request_failed += 1;
continue;
}
const detail =
detailMap.get(row.upc) ??
({
requestedUpc: row.upc,
normalizedUpc: row.upc,
status: "request_failed",
asin: null,
candidateAsins: [],
keepaData: null,
provider: "sp_api",
reason: "UPC lookup returned no result",
} satisfies UpcLookupDetail);
if (!detailMap.has(row.upc)) detailMap.set(row.upc, detail);
unresolvedByStatus[detail.status] += 1;
if (detail.status === "found" && detail.asin) {
@@ -407,30 +426,15 @@ export async function runUpcFileAnalysis(
const batchResults: SupplierAnalysisResult[] = [];
for (const row of rows) {
const detail = detailMap.get(row.upc);
if (!detail || detail.status === "found") continue;
const detail = detailMap.get(row.upc)!;
if (detail.status === "found") continue;
batchResults.push({
upc: row.upc,
rowNumber: row.rowNumber,
record: {
asin: detail?.asin ?? row.upc,
name: row.name ?? row.upc,
unitCost: row.unitCost ?? 0,
brand: row.brand,
category: row.category,
},
lookup:
detail ??
({
requestedUpc: row.upc,
normalizedUpc: row.upc,
status: "request_failed",
asin: null,
candidateAsins: [],
keepaData: null,
reason: "UPC lookup returned no result",
} satisfies UpcLookupDetail),
record: toSupplierInputRecord(row),
product: null,
lookup: detail,
keepa: null,
spApi: null,
score: skippedScore(detail?.reason ?? "UPC unresolved"),
@@ -465,7 +469,8 @@ export async function runUpcFileAnalysis(
batchResults.push({
upc: entry.detail.normalizedUpc,
rowNumber: entry.row.rowNumber,
record: entry.product,
record: toSupplierInputRecord(entry.row),
product: entry.product,
lookup: entry.detail,
keepa,
spApi,
@@ -488,6 +493,7 @@ export async function runUpcFileAnalysis(
const exportSummary = summarizeSupplierResults(allResults, unresolvedByStatus);
await writeSupplierWorkbook(outputFile, allResults, exportSummary);
await completeRunInDb(runId);
if (allResults.length > 0) {
const ranked = allResults
@@ -530,6 +536,9 @@ export async function runUpcFileAnalysis(
skippedInvalidUpc: readerSummary.skippedInvalidUpc,
},
};
} catch (error) {
await failRunInDb(runId, error);
throw error;
} finally {
if (manageResources) {
await disconnectCache();

View File

@@ -59,6 +59,7 @@ export interface KeepaUpcLookupDetail {
asin: string | null;
candidateAsins: string[];
keepaData: KeepaData | null;
provider?: "sp_api" | "keepa";
reason?: string;
}
@@ -114,7 +115,8 @@ export interface SupplierScore {
export interface SupplierAnalysisResult {
upc: string;
rowNumber?: number;
record: ProductRecord;
record: SupplierInputRecord;
product: ProductRecord | null;
lookup: UpcLookupDetail;
keepa: KeepaData | null;
spApi: SpApiData | null;
@@ -122,6 +124,58 @@ export interface SupplierAnalysisResult {
fetchedAt: string;
}
export interface SupplierInputRecord {
name: string;
unitCost: number;
brand?: string;
category?: string;
}
export interface Product {
asin: string;
name: string | null;
brand: string | null;
category: string | null;
firstSeenAt: string;
lastSeenAt: string;
}
export interface ProductObservation {
id: number;
productAsin: string;
runId: number;
source: string;
fetchedAt: string;
}
export interface Run {
id: number;
type:
| "lead_analysis"
| "category_analysis"
| "supplier_upc"
| "stalker"
| "stalker_analysis";
parentRunId?: number | null;
status: string;
}
export interface RunItem {
id: number;
runId: number;
productAsin: string | null;
sourceRow?: number | null;
}
export interface AnalysisRevision {
id: number;
runItemId: number;
decision: "FBA" | "FBM" | "BUY" | "WATCH" | "SKIP";
confidence: number | null;
reasoning: string | null;
analyzedAt: string;
}
export interface CategoryRunSummaryDb {
categoryId: number;
categoryLabel: string;

View File

@@ -1,7 +1,8 @@
import { createRoot } from "react-dom/client";
import { useEffect, useMemo, useState } from "react";
type ProcessType = "lead_analysis" | "category_analysis";
type ProcessType = "lead_analysis" | "category_analysis" | "supplier_upc" | "stalker" | "stalker_analysis";
type AnalysisDecision = "FBA" | "FBM" | "BUY" | "WATCH" | "SKIP";
type SortDirection = "ASC" | "DESC";
type Run = {
@@ -15,6 +16,8 @@ type Run = {
totalProducts: number;
fbaCount: number;
fbmCount: number;
buyCount: number;
watchCount: number;
skipCount: number;
};
@@ -37,11 +40,15 @@ type RunDetail = {
totalProducts: number;
fbaCount: number;
fbmCount: number;
buyCount: number;
watchCount: number;
skipCount: number;
summary: {
totalProducts: number;
fbaCount: number;
fbmCount: number;
buyCount: number;
watchCount: number;
skipCount: number;
};
errorMessage?: string;
@@ -49,6 +56,8 @@ type RunDetail = {
};
type ResultItem = {
item_id: number;
product_asin: string | null;
id?: number;
run_id: number;
asin: string;
@@ -60,7 +69,7 @@ type ResultItem = {
sales_rank: number | null;
seller_count: number | null;
monthly_sold: number | null;
verdict: "FBA" | "FBM" | "SKIP";
verdict: AnalysisDecision | null;
amazon_is_seller: number | null;
amazon_buybox_share_pct_90d: number | null;
confidence: number | null;
@@ -77,17 +86,18 @@ type ResultsResponse = {
totalPages: number;
};
type VerdictFilter = "" | "FBA" | "FBM" | "SKIP";
type VerdictFilter = "" | AnalysisDecision;
type AmazonSellerFilter = "" | "yes" | "no";
type ProductListItem = {
processType: ProcessType;
runId: number;
item_id: number | null;
processType: ProcessType | null;
runId: number | null;
asin: string;
product_name: string | null;
brand: string | null;
category: string | null;
verdict: "FBA" | "FBM" | "SKIP";
verdict: AnalysisDecision | null;
confidence: number | null;
sellability_status: string | null;
monthly_sold: number | null;
@@ -98,7 +108,7 @@ type ProductListItem = {
current_price: number | null;
avg_price_90d: number | null;
reasoning: string | null;
fetched_at: string;
fetched_at: string | null;
};
type ProductListResponse = {
@@ -179,6 +189,35 @@ type StalkerProductsResponse = {
totalPages: number;
};
type ProductHistoryResponse = {
product: {
asin: string;
name: string | null;
brand: string | null;
category: string | null;
first_seen_at: string;
last_seen_at: string;
};
observations: Array<{
id: number;
run_id: number;
source: string;
current_price: number | null;
monthly_sold: number | null;
sales_rank: number | null;
sellability_status: string | null;
fetched_at: string;
}>;
analyses: Array<{
id: number;
run_id: number;
decision: string;
confidence: number | null;
reasoning: string | null;
analyzed_at: string;
}>;
};
type SortState = {
field: string;
direction: SortDirection;
@@ -362,7 +401,7 @@ function Dashboard({
setDeletingKey(key);
try {
const response = await fetch(`/api/runs/${run.processType}/${run.runId}`, { method: "DELETE" });
const response = await fetch(`/api/runs/${run.runId}`, { method: "DELETE" });
if (!response.ok) {
const errorPayload = await response.json().catch(() => null) as { error?: string } | null;
const message = errorPayload?.error ?? "Failed to delete run";
@@ -545,7 +584,7 @@ function RunDetails({
useEffect(() => {
let cancelled = false;
async function loadRun() {
const res = await fetch(`/api/runs/${processType}/${runId}`);
const res = await fetch(`/api/runs/${runId}`);
const payload = (await res.json()) as RunDetail;
if (!cancelled) {
setRun(payload);
@@ -573,7 +612,7 @@ function RunDetails({
if (minConfidence) params.set("minConfidence", minConfidence);
if (maxConfidence) params.set("maxConfidence", maxConfidence);
const res = await fetch(`/api/runs/${processType}/${runId}/results?${params.toString()}`);
const res = await fetch(`/api/runs/${runId}/items?${params.toString()}`);
const payload = (await res.json()) as ResultsResponse;
if (!cancelled) {
setResults(payload);
@@ -596,12 +635,13 @@ function RunDetails({
};
}, [processType, runId]);
async function reanalyzeAsin(asin: string) {
if (reanalyzing[asin]) return;
setReanalyzing((prev) => ({ ...prev, [asin]: true }));
async function reanalyzeItem(item: ResultItem) {
const key = String(item.item_id);
if (reanalyzing[key]) return;
setReanalyzing((prev) => ({ ...prev, [key]: true }));
try {
const response = await fetch(
`/api/runs/${processType}/${runId}/asins/${encodeURIComponent(asin)}/reanalyze`,
`/api/run-items/${item.item_id}/reanalyze`,
{ method: "POST" },
);
if (!response.ok) {
@@ -613,7 +653,7 @@ function RunDetails({
} finally {
setReanalyzing((prev) => {
const next = { ...prev };
delete next[asin];
delete next[key];
return next;
});
}
@@ -626,14 +666,14 @@ function RunDetails({
<div className="card">
<h2>Run Detail</h2>
<div className="meta-grid" style={{ marginTop: 12 }}>
<div className="meta"><strong>Process:</strong> {processType}</div>
<div className="meta"><strong>Process:</strong> {run?.processType ?? processType}</div>
<div className="meta"><strong>Run ID:</strong> {runId}</div>
<div className="meta"><strong>Status:</strong> {run ? <span className={statusBadgeClass(run.status)}>{run.status}</span> : "-"}</div>
<div className="meta"><strong>Timestamp:</strong> {run ? formatDate(run.timestamp) : "-"}</div>
<div className="meta"><strong>Job:</strong> {run?.jobType ?? "-"}</div>
<div className="meta"><strong>Source:</strong> {run?.source ?? "-"}</div>
<div className="meta"><strong>Total:</strong> {formatNumber(run?.summary.totalProducts)}</div>
<div className="meta"><strong>FBA/FBM/SKIP:</strong> {formatNumber(run?.summary.fbaCount)}/{formatNumber(run?.summary.fbmCount)}/{formatNumber(run?.summary.skipCount)}</div>
<div className="meta"><strong>FBA/FBM/BUY/WATCH/SKIP:</strong> {formatNumber(run?.summary.fbaCount)}/{formatNumber(run?.summary.fbmCount)}/{formatNumber(run?.summary.buyCount)}/{formatNumber(run?.summary.watchCount)}/{formatNumber(run?.summary.skipCount)}</div>
</div>
<div style={{ marginTop: 10 }}>
<TinyBar fba={run?.summary.fbaCount ?? 0} fbm={run?.summary.fbmCount ?? 0} skip={run?.summary.skipCount ?? 0} />
@@ -647,6 +687,8 @@ function RunDetails({
<option value="">All verdicts</option>
<option value="FBA">FBA</option>
<option value="FBM">FBM</option>
<option value="BUY">BUY</option>
<option value="WATCH">WATCH</option>
<option value="SKIP">SKIP</option>
</select>
<select value={sellabilityStatus} onChange={(e) => { setPage(1); setSellabilityStatus(e.target.value); }}>
@@ -677,7 +719,7 @@ function RunDetails({
</div>
<div style={{ marginTop: 10 }}>
<a
href={`/api/runs/${processType}/${runId}/export.csv?q=${encodeURIComponent(search)}&verdict=${encodeURIComponent(verdict)}&sellabilityStatus=${encodeURIComponent(sellabilityStatus)}&amazonIsSeller=${encodeURIComponent(amazonSellerFilter)}&minConfidence=${encodeURIComponent(minConfidence)}&maxConfidence=${encodeURIComponent(maxConfidence)}&sort=${encodeURIComponent(buildSortValue(sort))}`}
href={`/api/runs/${runId}/export.csv?q=${encodeURIComponent(search)}&verdict=${encodeURIComponent(verdict)}&sellabilityStatus=${encodeURIComponent(sellabilityStatus)}&amazonIsSeller=${encodeURIComponent(amazonSellerFilter)}&minConfidence=${encodeURIComponent(minConfidence)}&maxConfidence=${encodeURIComponent(maxConfidence)}&sort=${encodeURIComponent(buildSortValue(sort))}`}
>
<button>Export filtered CSV</button>
</a>
@@ -692,8 +734,8 @@ function RunDetails({
<div className="anomaly-list" style={{ marginTop: 8 }}>
{anomalies.slice(0, 8).map((item) => (
<div key={`anom-${item.asin}-${item.fetched_at}`} className="anomaly-item">
<a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a>
<span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span>
{item.product_asin ? <a href={`/products/${item.product_asin}`}>{item.asin}</a> : item.asin}
{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}
<span>{detectAnomaly(item)}</span>
</div>
))}
@@ -729,8 +771,8 @@ function RunDetails({
) : results?.items.length ? (
results.items.map((item) => (
<tr key={`${item.asin}-${item.fetched_at}`}>
<td><a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a></td>
<td><span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span></td>
<td>{item.product_asin ? <a href={`/products/${item.product_asin}`}>{item.asin}</a> : item.asin}</td>
<td>{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}</td>
<td>{formatNumber(item.monthly_sold)}</td>
<td>{formatNumber(item.seller_count)}</td>
<td>{formatAmazonSeller(item.amazon_is_seller)}</td>
@@ -744,12 +786,14 @@ function RunDetails({
<td>{formatNumber(item.confidence)}</td>
<td className="reason-col" title={item.reasoning || undefined}>{item.reasoning || "-"}</td>
<td>
<button
onClick={() => reanalyzeAsin(item.asin)}
disabled={Boolean(reanalyzing[item.asin])}
>
{reanalyzing[item.asin] ? "Re-analyzing..." : "Re-analyze"}
</button>
{item.product_asin && run?.processType !== "supplier_upc" ? (
<button
onClick={() => reanalyzeItem(item)}
disabled={Boolean(reanalyzing[String(item.item_id)])}
>
{reanalyzing[String(item.item_id)] ? "Re-analyzing..." : "Re-analyze"}
</button>
) : "-"}
</td>
</tr>
))
@@ -813,12 +857,13 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
}, [search, activeVerdict, amazonSellerFilter, page, pageSize, sort]);
async function reanalyzeAsin(item: ProductListItem) {
const key = `${item.processType}:${item.runId}:${item.asin}`;
if (item.item_id == null) return;
const key = String(item.item_id);
if (reanalyzing[key]) return;
setReanalyzing((prev) => ({ ...prev, [key]: true }));
try {
const response = await fetch(
`/api/runs/${item.processType}/${item.runId}/asins/${encodeURIComponent(item.asin)}/reanalyze`,
`/api/run-items/${item.item_id}/reanalyze`,
{ method: "POST" },
);
if (!response.ok) {
@@ -854,6 +899,8 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
<option value="">All verdicts</option>
<option value="FBA">FBA</option>
<option value="FBM">FBM</option>
<option value="BUY">BUY</option>
<option value="WATCH">WATCH</option>
<option value="SKIP">SKIP</option>
</select>
<select
@@ -902,8 +949,8 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
) : items?.items.length ? (
items.items.map((item) => (
<tr key={`${item.processType}-${item.runId}-${item.asin}-${item.fetched_at}`}>
<td><a href={`http://amazon.com/dp/${item.asin}`} target="_blank" rel="noreferrer">{item.asin}</a></td>
<td><span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span></td>
<td><a href={`/products/${item.asin}`}>{item.asin}</a></td>
<td>{item.verdict ? <span className={verdictBadgeClass(item.verdict)}>{item.verdict}</span> : "-"}</td>
<td>{formatNumber(item.monthly_sold)}</td>
<td>{formatNumber(item.seller_count)}</td>
<td>{formatAmazonSeller(item.amazon_is_seller)}</td>
@@ -917,12 +964,14 @@ function ProductList({ verdict, onBack }: { verdict: VerdictFilter; onBack: () =
<td>{formatNumber(item.confidence)}</td>
<td className="reason-col" title={item.reasoning || undefined}>{item.reasoning || "-"}</td>
<td>
<button
onClick={() => reanalyzeAsin(item)}
disabled={Boolean(reanalyzing[`${item.processType}:${item.runId}:${item.asin}`])}
>
{reanalyzing[`${item.processType}:${item.runId}:${item.asin}`] ? "Re-analyzing..." : "Re-analyze"}
</button>
{item.item_id == null || item.processType === "supplier_upc" ? "-" : (
<button
onClick={() => reanalyzeAsin(item)}
disabled={Boolean(reanalyzing[String(item.item_id)])}
>
{reanalyzing[String(item.item_id)] ? "Re-analyzing..." : "Re-analyze"}
</button>
)}
</td>
</tr>
))
@@ -995,7 +1044,7 @@ function StalkerExplorer({
}, [search, sellerId, runId, minRatingCount, maxRatingCount, page, pageSize, sort, refreshTick]);
async function purgeStalkerData() {
const confirmed = window.confirm("Permanently delete all Stalker runs, sellers, and sellable products from the database?");
const confirmed = window.confirm("Permanently delete all Stalker runs and unreferenced seller records? Canonical products are retained.");
if (!confirmed) return;
setPurging(true);
@@ -1384,17 +1433,95 @@ function StalkerProductsExplorer({
);
}
function ProductDetails({
asin,
onBack,
}: {
asin: string;
onBack: () => void;
}) {
const [data, setData] = useState<ProductHistoryResponse | null>(null);
useEffect(() => {
let cancelled = false;
fetch(`/api/products/${encodeURIComponent(asin)}`)
.then((response) => response.json())
.then((payload: ProductHistoryResponse) => {
if (!cancelled) setData(payload);
});
return () => {
cancelled = true;
};
}, [asin]);
return (
<div className="page">
<button className="back" onClick={onBack}>Back</button>
<div className="card">
<h2>{data?.product.name ?? asin}</h2>
<div className="meta-grid" style={{ marginTop: 12 }}>
<div className="meta"><strong>ASIN:</strong> <a href={`https://amazon.com/dp/${asin}`} target="_blank" rel="noreferrer">{asin}</a></div>
<div className="meta"><strong>Brand:</strong> {data?.product.brand ?? "-"}</div>
<div className="meta"><strong>Category:</strong> {data?.product.category ?? "-"}</div>
<div className="meta"><strong>Last seen:</strong> {data ? formatDate(data.product.last_seen_at) : "-"}</div>
</div>
</div>
<div className="card">
<h3>Analysis History</h3>
<div className="table-wrap">
<table>
<thead><tr><th>Run</th><th>Decision</th><th>Confidence</th><th>Reasoning</th><th>Analyzed</th></tr></thead>
<tbody>
{data?.analyses.length ? data.analyses.map((analysis) => (
<tr key={analysis.id}>
<td>{analysis.run_id}</td>
<td>{analysis.decision}</td>
<td>{formatNumber(analysis.confidence)}</td>
<td>{analysis.reasoning ?? "-"}</td>
<td>{formatDate(analysis.analyzed_at)}</td>
</tr>
)) : <tr><td colSpan={5}>No analysis history</td></tr>}
</tbody>
</table>
</div>
</div>
<div className="card">
<h3>Observations</h3>
<div className="table-wrap">
<table>
<thead><tr><th>Run</th><th>Source</th><th>Price</th><th>Monthly Sold</th><th>Sales Rank</th><th>Sellability</th><th>Fetched</th></tr></thead>
<tbody>
{data?.observations.length ? data.observations.map((observation) => (
<tr key={observation.id}>
<td>{observation.run_id}</td>
<td>{observation.source}</td>
<td>{formatCurrency(observation.current_price)}</td>
<td>{formatNumber(observation.monthly_sold)}</td>
<td>{formatNumber(observation.sales_rank)}</td>
<td>{observation.sellability_status ?? "-"}</td>
<td>{formatDate(observation.fetched_at)}</td>
</tr>
)) : <tr><td colSpan={7}>No observations</td></tr>}
</tbody>
</table>
</div>
</div>
</div>
);
}
type AppRoute =
| { kind: "dashboard" }
| { kind: "run"; processType: ProcessType; runId: number }
| { kind: "products"; verdict: VerdictFilter }
| { kind: "product"; asin: string }
| { kind: "stalker" }
| { kind: "stalker-products" };
function parseRoute(pathname: string, search: string): AppRoute {
const runMatch = pathname.match(/^\/runs\/(lead_analysis|category_analysis)\/(\d+)$/);
const runMatch = pathname.match(/^\/runs\/(\d+)$/);
if (runMatch) {
return { kind: "run", processType: runMatch[1] as ProcessType, runId: Number(runMatch[2]) };
return { kind: "run", processType: "lead_analysis", runId: Number(runMatch[1]) };
}
if (pathname === "/products") {
@@ -1404,6 +1531,11 @@ function parseRoute(pathname: string, search: string): AppRoute {
return { kind: "products", verdict };
}
const productMatch = pathname.match(/^\/products\/([A-Z0-9]{10})$/i);
if (productMatch) {
return { kind: "product", asin: productMatch[1]!.toUpperCase() };
}
if (pathname === "/stalker") {
return { kind: "stalker" };
}
@@ -1425,7 +1557,7 @@ function App() {
}, []);
function openRun(run: Run) {
const path = `/runs/${run.processType}/${run.runId}`;
const path = `/runs/${run.runId}`;
history.pushState({}, "", path);
setRoute({ kind: "run", processType: run.processType, runId: run.runId });
}
@@ -1459,6 +1591,10 @@ function App() {
return <ProductList verdict={route.verdict} onBack={backToDashboard} />;
}
if (route.kind === "product") {
return <ProductDetails asin={route.asin} onBack={backToDashboard} />;
}
if (route.kind === "stalker") {
return <StalkerExplorer onBack={backToDashboard} onOpenProducts={openStalkerProducts} />;
}

View File

@@ -1,6 +1,11 @@
import { eq } from "drizzle-orm";
import { db } from "./db/index.ts";
import { runs, analysisResults } from "./db/schema.ts";
import { eq, sql } from "drizzle-orm";
import { analysisRunStats, runs } from "./db/schema.ts";
import {
persistLlmResults,
persistSupplierResults,
refreshRunStats,
} from "./db/persistence.ts";
import type { AnalysisResult, SupplierAnalysisResult } from "./types.ts";
import { mkdirSync } from "node:fs";
import path from "node:path";
@@ -13,18 +18,6 @@ export type RunCounts = {
skipCount: number;
};
function computeRunCountsFromResults(results: AnalysisResult[]): RunCounts {
const fbaCount = results.filter((r) => r.verdict.verdict === "FBA").length;
const fbmCount = results.filter((r) => r.verdict.verdict === "FBM").length;
const skipCount = results.filter((r) => r.verdict.verdict === "SKIP").length;
return {
totalProducts: results.length,
fbaCount,
fbmCount,
skipCount,
};
}
function buildRow(r: AnalysisResult) {
const price =
r.product.keepa?.currentPrice ??
@@ -91,9 +84,15 @@ export async function writeResultsToDb(
inputFile: string,
outputFile: string | undefined,
): Promise<void> {
const runCounts = computeRunCountsFromResults(results);
const runId = await startRunInDb(inputFile, outputFile, runCounts);
await appendResultsToRun(runId, results);
const runId = await startRunInDb(inputFile, outputFile);
try {
await appendResultsToRun(runId, results);
await refreshRunCountsInDb(runId);
await completeRunInDb(runId);
} catch (error) {
await failRunInDb(runId, error);
throw error;
}
console.log(`Results written to database for run_id: ${runId}`);
}
@@ -122,24 +121,28 @@ export async function startRunInDb(
fbmCount: 0,
skipCount: 0,
},
type: "lead_analysis" | "supplier_upc" = "lead_analysis",
): Promise<number> {
const [row] = await db
.insert(runs)
.values({
type: "lead_analysis",
type,
inputFile,
outputFile: outputFile ?? null,
status: "ok",
totalProducts: counts.totalProducts,
fbaCount: counts.fbaCount,
fbmCount: counts.fbmCount,
skipCount: counts.skipCount,
status: "running",
startedAt: new Date(),
completedAt: new Date(),
})
.returning({ id: runs.id });
if (!row) throw new Error("Failed to insert run record.");
await db.insert(analysisRunStats).values({
runId: row.id,
processedCount: counts.totalProducts,
analyzedCount: counts.totalProducts,
fbaCount: counts.fbaCount,
fbmCount: counts.fbmCount,
skipCount: counts.skipCount,
});
return row.id;
}
@@ -148,60 +151,11 @@ export async function appendResultsToRun(
results: AnalysisResult[],
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((r) => {
const row = buildRow(r);
return {
runId,
asin: row.ASIN,
productName: row.Name || null,
brand: row.Brand || null,
category: row.Category || null,
unitCost: (row["Unit Cost"] as number) ?? null,
currentPrice: (row["Current Price"] as number) || null,
avgPrice90d: (row["Avg Price 90d"] as number) || null,
avgPrice90dSheet: (row["Avg Price 90d (sheet)"] as number) || null,
sellingPriceSheet: (row["Selling Price (sheet)"] as number) || null,
salesRank: (row["Sales Rank"] as number) || null,
rankAvg90d: (row["Rank Avg 90d"] as number) || null,
sellerCount: (row.Sellers as number) || null,
amazonIsSeller:
row["Amazon Is Seller"] == null
? null
: Boolean(row["Amazon Is Seller"]),
amazonBuyboxSharePct90d:
(row["Amazon Buy Box Share 90d %"] as number) || null,
monthlySold: (row["Monthly Sold"] as number) || null,
rankDrops30d: (row["Rank Drops 30d"] as number) || null,
rankDrops90d: (row["Rank Drops 90d"] as number) || null,
fbaNetSheet: (row["FBA Net (sheet)"] as number) || null,
grossProfitDollar: (row["Gross Profit $"] as number) || null,
grossProfitPct: (row["Gross Profit %"] as number) || null,
netProfitSheet: (row["Net Profit (sheet)"] as number) || null,
roiSheet: (row["ROI (sheet)"] as number) || null,
moq: (row.MOQ as number) || null,
moqCost: (row["MOQ Cost"] as number) || null,
qtyAvailable: (row["Qty Available"] as number) || null,
supplier: row.Supplier || null,
sourceUrl: row["Source URL"] || null,
asinLink: row["ASIN Link"] || null,
promoCouponCode: row["Promo/Coupon Code"] || null,
notes: row.Notes || null,
leadDate: row["Lead Date"] || null,
fbaFee: row["FBA Fee"] ?? null,
fbmFee: row["FBM Fee"] ?? null,
referralPercent: row["Referral %"] ?? null,
canSell: row["Can Sell"],
sellabilityStatus: row.Sellability,
sellabilityReason: row["Sellability Reason"] || null,
verdict: row.Verdict,
confidence: row.Confidence ?? null,
reasoning: row.Reasoning,
fetchedAt: new Date(r.product.fetchedAt),
};
await persistLlmResults(runId, results, {
source: "lead_analysis",
metadataSource: "input",
preserveSourcingInput: true,
});
await db.insert(analysisResults).values(rows);
}
export async function appendSupplierResultsToRun(
@@ -209,92 +163,29 @@ export async function appendSupplierResultsToRun(
results: SupplierAnalysisResult[],
): Promise<void> {
if (results.length === 0) return;
const rows = results.map((result) => {
const keepa = result.keepa;
const spApi = result.spApi;
const asin = result.lookup.asin ?? result.record.asin ?? result.upc;
const category =
result.record.category ?? keepa?.categoryTree?.join(" > ") ?? null;
const canSell =
spApi?.canSell == null ? null : spApi.canSell ? "yes" : "no";
return {
runId,
asin,
productName: result.record.name,
brand: result.record.brand ?? null,
category,
unitCost: result.record.unitCost || null,
currentPrice: result.score.salePrice,
avgPrice90d: keepa?.avgPrice90 ?? null,
salesRank: keepa?.salesRank ?? null,
rankAvg90d: keepa?.salesRankAvg90 ?? null,
sellerCount: keepa?.sellerCount ?? null,
amazonIsSeller: keepa?.amazonIsSeller ?? null,
amazonBuyboxSharePct90d: keepa?.amazonBuyboxSharePct90d ?? null,
monthlySold: keepa?.monthlySold ?? null,
rankDrops30d: keepa?.salesRankDrops30 ?? null,
rankDrops90d: keepa?.salesRankDrops90 ?? null,
upc: result.upc,
fbaFee: result.score.fbaFee,
fbmFee: spApi?.fbmFee ?? null,
referralPercent: spApi?.referralFeePercent ?? null,
supplierScore: result.score.score,
supplierProfit: result.score.profit,
supplierMargin: result.score.margin,
supplierRoi: result.score.roi,
supplierReason: result.score.reason,
upcLookupStatus: result.lookup.status,
upcLookupReason: result.lookup.reason ?? null,
candidateAsins: result.lookup.candidateAsins.join(","),
canSell,
sellabilityStatus: spApi?.sellabilityStatus ?? null,
sellabilityReason: spApi?.sellabilityReason ?? null,
verdict: result.score.verdict,
confidence: result.score.score,
reasoning: result.score.reason,
fetchedAt: new Date(result.fetchedAt),
};
});
await db.insert(analysisResults).values(rows);
await persistSupplierResults(runId, results);
}
export async function refreshRunCountsInDb(runId: number): Promise<RunCounts> {
const [stats] = await db.execute(
sql<{
total: string;
fba: string | null;
fbm: string | null;
skip: string | null;
}>`SELECT
COUNT(*) AS total,
SUM(CASE WHEN verdict = 'FBA' THEN 1 ELSE 0 END) AS fba,
SUM(CASE WHEN verdict = 'FBM' THEN 1 ELSE 0 END) AS fbm,
SUM(CASE WHEN verdict = 'SKIP' THEN 1 ELSE 0 END) AS skip
FROM analysis_results
WHERE run_id = ${runId}`,
);
const counts: RunCounts = {
totalProducts: Number(stats?.total ?? 0),
fbaCount: Number(stats?.fba ?? 0),
fbmCount: Number(stats?.fbm ?? 0),
skipCount: Number(stats?.skip ?? 0),
};
return refreshRunStats(runId);
}
export async function completeRunInDb(runId: number): Promise<void> {
await db
.update(runs)
.set({
totalProducts: counts.totalProducts,
fbaCount: counts.fbaCount,
fbmCount: counts.fbmCount,
skipCount: counts.skipCount,
})
.set({ status: "completed", completedAt: new Date(), errorMessage: null })
.where(eq(runs.id, runId));
}
return counts;
export async function failRunInDb(
runId: number,
error: unknown,
): Promise<void> {
const errorMessage = error instanceof Error ? error.message : String(error);
await db
.update(runs)
.set({ status: "failed", completedAt: new Date(), errorMessage })
.where(eq(runs.id, runId));
}
export function printResults(results: AnalysisResult[]): void {