Files
ai-g3sb-backman2.0/backend/utils/__pycache__/fewshot_selector.cpython-312.pyc
T

186 lines
26 KiB
Plaintext
Raw Normal View History

2026-04-10 16:52:07 +08:00
Ë
XæÝiÃHã ó˜—dZddlZddlZddlmZddlmZmZmZddl m
2026-04-10 16:52:07 +08:00
Z
ddl Z ddl
Z
e
2026-04-10 16:52:07 +08:00
je«ZdZe
Gd„d««ZGd „d
«Zd efd „Zed
k(�réddlZej,d¬«Zej1dd¬«ej1dd¬«ej1ded¬«ej1ded¬«ej1ddd¬«ej5«Zeej8«Zej<rƒej?«Ze d«e d ed!›�«e d"ed#d$›�«e d%ed&›�«e d'«e!ed(jE«d)„¬*«dd+Z#e#D]\Z$Z%e d,e$›d-e%›�«ŒyejLrªejOejLejPejR¬.«Z*e d/ejL›d0�«e*D]\Z+e d1e+jX›d2e+jZ›d3e+j\›�«e d4e+j^›�«e d5e+j`›d6�«Œ^ye d7«yy)8uÚ
Few-shot示例选择器 - 基于�验数�集动�选择相关示例
与 Schema ��检索一致,使用 utils.embedding.get_embedder()(本地 Qwen3 或远程 OpenAI 兼容 API,
由 USE_LOCAL_EMBEDDING � OPENAI_* / MODELSCOPE_* / DASHSCOPE_* 等环境��决定)。
用法:
from utils.fewshot_selector import FewShotSelector
2026-04-10 16:52:07 +08:00
# Chroma 模�(FEWSHOT_USE_CHROMA=true):��传 JSONL,路径传 None 或 ""
selector = FewShotSelector(None)
# 或传统:FewShotSelector("data/experiences/all_samples.jsonl")
examples = selector.select(question="查询2024年1月的销售�", top_k=3)
# 在Prompt中使用
2026-04-10 16:52:07 +08:00
prompt = f"{examples}
当�问题:{question}
Schema:{schema}"
éN)ÚPath)ÚListÚDictÚOptional)Ú dataclassÚcó¬—eZdZUdZeed<eed<eeed<eed<eed<eeed<eeed<eed <e d
e
2026-04-10 16:52:07 +08:00
d dfd „«Z dd
e d efd„Z
d e
fd„Zy)ÚExperienceSampleuç»�验数æ�®æ ·æœ¬ÚqidÚ question_zhÚ question_enÚsqlÚ explanationÚratingÚtagsÚ
difficultyÚdataÚreturnc
ó—||jdd«|jdd«|jd«|jdd«|jdd«|jd«|jdg«|jd d
2026-04-10 16:52:07 +08:00
«¬ «S) Nr rr r
rrrrrÚmedium)r r r
2026-04-10 16:52:07 +08:00
rrrrr)Úget)Úclsrs úFC:\Users\24019\Desktop\backman-camel\backend\utils\fewshot_selector.pyÚ from_dictzExperienceSample.from_dict-s{€áØ—‘˜ Ó#ØŸ™ °Ó3ØŸ™ Ó/Ø—‘˜ Ó#ØŸ™ °Ó3Ø—8‘8˜HÓ%Ø—‘˜& "Ó%Ø—x‘x  ¨hÓ7ô 
ð
óÚinclude_explanationcó„—d|j›d|j›�}|r!|jr|d|jdd›�z
}|S)u转æ�¢ä¸ºfew-shotæ ¼å¼�õ 问题:u
SQL:
2026-04-10 16:52:07 +08:00
u
说明:NéÈ)r rr)ÚselfrÚresults rÚto_fewshot_formatz"ExperienceSample.to_fewshot_format:sN€à˜T×-Ñ-Ð.¨j¸¿¹¸
ÐCˆÙ  4×#3Ò#3Ø ˜  D×$4Ñ$4°T°cÐ$:Ð#;Ð<Ñ <ˆF؈
rcóŒ—|j|j|j|j|j|j
dœS)N)r Úquestionrrrr)r r rrrr©r s rÚto_dictzExperienceSample.to_dictAs8€à—8‘8Ø×(Ñ(Ø—8‘8Ø—k‘kØ—I‘IØŸ/™/ñ

ð
rN)T)Ú__name__Ú
__module__Ú __qualname__Ú__doc__ÚstrÚ__annotations__rÚintrÚ classmethodÚdictrÚboolr"r&©rrr
r
!s}…áØ ƒHØÓؘ#‘ÓØ ƒHØÓØ �S‰MÓØ
ˆs‰)ƒO؃Oàð

˜Tð
2026-04-10 16:52:07 +08:00

Ð&8ò

óð

ñ°TðÀSóð
2026-04-10 16:52:07 +08:00
˜ô
rr
cób—eZdZdZ ddeedeededeedeef
2026-04-10 16:52:07 +08:00
d„Zdd
„Zd „Z dd „Z
dd
„Z dede dee dee
ededee
ed e
efd„Z ddede dee dee
ededee
ed e
efd„Z ddede de d efd„Zd e
efd„Zd de
ede d efd„Zd efd„Zy)!ÚFewShotSelectoruž
Few-shot示例选择器
根�用户问题,从�验数�集中检索最相似的示例,
用于增强Prompt,��LLM生�质�。
NÚ samples_pathÚembedding_model_pathÚ use_cacheÚ
use_chromaÚchroma_persist_dircóð—|xsdj«}|r t|«nd|_g|_d|_d|_||_d|_d|_|�|n%tjdd«j«dv|_ ||_
|�|n'tjdt«j«|_|jr|j!«y|j#«|j%«y)uþ
Args:
samples_path: 样本 JSONL 路径;**空字符串**表示�读文件(仅当 ``FEWSHOT_USE_CHROMA=true``
2026-04-10 16:52:07 +08:00
且 Chroma 中已有数�时�用)。构建索引�请用 ``scripts/build_fewshot_chroma_index.py --samples``。
embedding_model_path: 本地 Embedding 模型目录;None 时用环境��
EMBEDDING_MODEL_PATH(仅 USE_LOCAL_EMBEDDING=true 时有效)
use_cache: 是�缓存样本��(按��维度分文件,�模型会自动�建)
use_chroma: 是�使用 Chroma �久化��库;None 时读环境�� FEWSHOT_USE_CHROMA
chroma_persist_dir: Chroma 目录;None 时用 FEWSHOT_CHROMA_PATH 或默认 chroma_fewshot
rNÚFEWSHOT_USE_CHROMAÚfalse)Ú1ÚtrueÚyesÚEMBEDDING_MODEL_PATH)Ústriprr4ÚsamplesÚ _embedderÚ
embeddingsr6Ú
cache_pathÚ
_chroma_storeÚosÚgetenvÚlowerr7Ú_chroma_persist_dirÚ_DEFAULT_LOCAL_EMBED_PATHÚ_embedding_model_pathÚ_init_chroma_modeÚ
_load_samplesÚ_init_embedder_and_vector_index)r r4r5r6r7r8Úsps rÚ__init__zFewShotSelector.__init__Tsé€ð$Ò ˜b×
'Ñ
'Ó
)ˆÙ8:¬D°¬HÀˆÔØ/1ˆŒ ؈ŒØ04ˆŒØ"ˆŒØ*.ˆŒØ!ˆÔðÐ%ñ
ä—‘Ð/°Ó9×?Ñ?ÓAÐEYÐYð
Œð
$6ˆÔ ð$Ð/ñ
!ä—‘Ð1Ô3LÓM×SÑSÓUð
Ô"ð �?Š?Ø × "Ñ "Õ $à × Ñ Ô Ø × 0Ñ 0Õ 2rrcó`—ddlm}ddlm}||j«|_||j
|j ¬«|_|jj«}|dkDr,tjd||jj«ná|j«|jrHtjd|j«|jj|jd¬«n}|j€tj!d
«n[|jj#«s!tj!d |j«n tj!d |j«d |_d |_tjd
|jj««y )u@Chroma 模å¼�:先连å�‘é‡�库;库é�žç©ºåˆ™ä¸�å†�读 JSONL。r©Ú get_embedder)ÚFewShotChromaStore)Ú persist_diru8[Few-shot] 已从å�‘é‡�库加载(Chroma %s æ�¡ï¼Œ%s)u?Few-shot Chroma 库为空,正从 JSONL 写入å�‘é‡�索引: %sF)Ú
force_rebuildNu{[Few-shot] Chroma 库为空且未é…�ç½® JSONL;请è¿�行 scripts/build_fewshot_chroma_index.py 或设置 FEWSHOT_DATA_PATHuA[Few-shot] Chroma 库为空且 JSONL ä¸�存在: %s;请先ç�Œåº“u8[Few-shot] Chroma 库为空且 JSONL 无有效样本: %su'[OK] Few-shot 使用 Chroma(%s æ�¡ï¼‰)Úutils.embeddingrSÚutils.fewshot_chroma_storerTrKrBrIrEÚcountÚloggerÚinforUrMrAr4Úbuild_from_samplesÚwarningÚexistsrCrD)r rSrTÚns rrLz!FewShotSelector._init_chroma_mode€sT€å0ÝAá% d×&@Ñ&@ÓAˆŒÙ/Ø �N‰NØ×0Ñ0ô
2026-04-10 16:52:07 +08:00
ˆÔð
× Ñ × $Ñ $Ó &ˆØ ˆqŠ5Ü �K‰KØJØØ×"Ñ"×.Ñ.õ
ð
× Ñ Ô Ø�|Š|Ü— ‘ ØUØ×%Ñ%ôð×"Ñ"×5Ñ5°d·l±lÐRWÐ5ÕXØ×"Ñ"Ð*Ü—‘ðXõð×&Ñ&×-Ñ-Ô/Ü—‘ØWØ×%Ñ%õô
—‘ØNØ×%Ñ%ôð
ˆŒØˆŒÜ� ‰ Ð=¸t×?QÑ?Q×?WÑ?WÓ?YÕZrcóÜ—|j€-|jrtjd«yt d«‚|jj «sE|jr!tj
d|j«ytd|j›�«‚tjd|j«|jjdd¬ «5}|D]S}tj|j««}|jjtj|««ŒU ddd«tjd
t!|j««y#1swYŒ3xYw) ud从 JSONL 加载样本到内存(é�ž Chroma 模å¼�必需;Chroma 空库时用于首次ç�Œåº“)。NuA[Few-shot] 未é…�ç½® JSONL 路径,è¿�行时仅从 Chroma 检索uG未指定样本 JSONL 路径且未å�¯ç”¨ Chroma(FEWSHOT_USE_CHROMA)uI[Few-shot] JSONL ä¸�存在 %s,跳过文件加载,仅从 Chroma 检索u样本文件ä¸�存在: u从 JSONL 加载样本: %sÚrzutf-8)Úencodingu"[OK] 自 JSONL 加载 %s 个样本)r4r7rZr[Ú
ValueErrorr^r]ÚFileNotFoundErrorÚopenÚjsonÚloadsr@rAÚappendr
rÚlen)r ÚfÚliners rrMzFewShotSelector._load_samples­s€à × Ñ Ð $Ø�ŠÜ— ‘ Ð_Ô`ØÜÐfÓgÐ gà× Ñ ×'Ñ'Ô)Ø�ŠÜ—‘Ø_Ø×%Ñ%ôðÜ#Ð&=¸d×>OÑ>OÐ=PÐ$QÓRÐ Rä� ‰ Ð0°$×2CÑ2CÔDØ
×
Ñ
×
2026-04-10 16:52:07 +08:00
#Ñ
# C°'Ð
#Ô
:¸aÛ�Ü—z‘z $§*¡*£,Ó/�Ø— ‘ ×#Ñ#Ô$4×$>Ñ$>¸tÓ$DÕEñ÷;ô
� ‰ Ð8¼#¸d¿l¹lÓ:KÕL÷ ;Ð
:ús
ÃAE"Å"E+có^—ddlm}||j«|_|j «y)u9é�ž Chroma:åˆ�始化 Embedder 与内存 numpy 索引。rrRN)rWrSrKrBÚ_build_numpy_index)r rSs rrNz/FewShotSelector._init_embedder_and_vector_indexÆs#€å0á% d×&@Ñ&@ÓAˆŒØ ×ÑÕ!rc ó*—|j€J‚|jjdgddd¬«}t|jd«}|jj
|jj ›d|›d�z |_|jr­|jj«r“ tj|j«|_ |jjd t|j«k(r?|jjd|k(r#tj!d
2026-04-10 16:52:07 +08:00
|j›�«y|jD�cgc]$}|j&xsd
j)«xsd‘Œ&}}|s-tj*d |ftj,¬«|_ ytj!dt|«›d�«|jj|t/dt|««dd¬«|_ |jrZ|j�Mtj0|j|j«tj!d|j›�«yyy#t"$r$}tj%d |›d �«Yd}~�ŒPd}~wwxYwcc}w)u?内存å�‘é‡� + å�¯é€‰ .npy 缓存(与历å�²è¡Œä¸ºä¸€è‡´ï¼‰ã€‚NÚ éTF©Ú
batch_sizeÚ normalizeÚ
show_progressz .fewshot_dimz.npyru#[OK] 加载 Few-shot å�‘é‡�缓存: uFew-shot 缓存加载失败: u,将é‡�新计算r)Údtypeu计算 u 个 Few-shot 样本å�‘é‡�...é u[OK] Few-shot å�‘é‡�已缓存: )rBÚencoder-Úshaper4ÚparentÚstemrDr6r^ÚnpÚloadrCrirArZr[Ú Exceptionr]r r@ÚemptyÚfloat32ÚminÚsave)r ÚprobeÚdimÚeÚsÚ questionss rrmz"FewShotSelector._build_numpy_indexÍs1€à�~‰~Ð)Ð)Ð)Ø—‘×%Ñ%Ø
ˆEØØØð &ó
ˆô �%—+‘+˜a‘.Ó!ˆà × Ñ × $Ñ $¨$×*;Ñ*;×*@Ñ*@Ð)AÀÈcÈUÐRVÐ'WÑ Wð
Œð �>Š>˜dŸo™o×4Ñ4Ô6ð
VÜ"$§'¡'¨$¯/©/Ó":�”à—O‘O×)Ñ)¨!Ñ,´°D·L±LÓ0AÒAØŸ™×-Ñ-¨aÑ0°CÒ7ä—K‘KÐ"EÀdÇoÁoÐEVÐ WÔXØð—\’\ó
á!�ð�]‰]Ò
2026-04-10 16:52:07 +08:00
˜b× 'Ñ 'Ó )Ò 0¨SÑ 0Ø!ð ð
ñÜ Ÿh™h¨¨3 x´r·z±zÔBˆDŒOØ ä� ‰ �gœc )›nÐ-Ð-JÐKÔLØŸ.™.×/Ñ/Ø Ü˜2œs 9›~Ó.ØØð 0ó
2026-04-10 16:52:07 +08:00
ˆŒð �>Š>˜dŸo™oÐ9Ü �G‰G�D—O‘O T§_¡_Ô 5Ü �K‰KÐ9¸$¿/¹/Ð9JÐKÕ Lð:ˆ>øô)ò
VÜ—‘Ð!>¸q¸cÐASÐT×UÒUûð
Vüò
2026-04-10 16:52:07 +08:00
sÂ&BI Å)JÉ J
É)JÊJ
r$Útop_kÚ
min_ratingÚ
required_tagsÚmax_difficultyÚ exclude_qidsc óv‡—ddlm}|j€J‚t|dzd«}|jj ||¬«} g}
| D]’\} } }
|| |
«Š|r‰j
|vrŒ!|r‰j �‰j |krŒ?|dk(r‰jdk7rŒT|dk(r‰jdk(rŒi|rtˆfd „|D««sŒ€|
j| ‰f«Œ”|
2026-04-10 16:52:07 +08:00
jd
„¬ «|
d|D��cgc]\}}|‘Œ }}}tjd |dd
›dt|«›d|›d|›d� «|Scc}}w)Nr)Úsample_from_chroma_metadataé é0)r‡ÚeasyrÚhardc3ó:•K—|]}|‰jv–—Œy­w©N©r)Ú.0Útagr…s €rÚ <genexpr>z1FewShotSelector._select_chroma.<locals>.<genexpr>søèø€Ð(PÁ-¸3¨°·±¬Á-ùóƒcó—|d S©Nrr1©Úxs rÚ<lambda>z0FewShotSelector._select_chroma.<locals>.<lambda>ó
€ q¨¡t¡er©Úkeyu Few-shot(Chroma)选择: 问题='éõ...' → 选中õ个示例 (top_k=ú
, min_rating=Ú))rXr�rEÚmaxÚ
search_rawr rrÚallrhÚsortrZr[ri)r r$r‡rˆr‰rŠr‹r�Ú
over_fetchÚrowsÚ
candidatesÚscoreÚmetaÚdocÚ_r…Úselecteds ` rÚ_select_chromazFewShotSelector._select_chromaüs]ø€õ Kà×!Ñ!Ð-Ð-Ð-ܘ ™ RÓ(ˆ
Ø×!Ñ!×,Ñ,¨X¸ZÐ,ÓHˆØ;=ˆ
Û $Ñ ˆE�4˜Ù+¨D°#Ó6ˆAÙ §¡¨Ñ 5ØÙ˜aŸh™hÐ2°q·x±xÀ*Ò7LØØ Ò'¨A¯L©L¸FÒ,BØØ Ò)¨a¯l©l¸fÒ.DØÙ¤SÓ(PÁ-Ó(PÔ%PØØ × Ñ ˜u a˜jÕ )ð!%ð �‰™OˆÔ,Ø",¨V¨eÑ"4Ô5Ñ"4™$˜!˜Q’AÐ"4ˆÑ5Ü� ‰ Ø.¨x¸¸¨}¨o¸_ÌSÐQYË]ÈOð\Ø�W˜M¨*¨°Qð
8ô
ðˆùó 6sÃ6 D5c
ó0‡
—|j�|j||||||«S|j� |j€tj d«gSt
2026-04-10 16:52:07 +08:00
|j«dk(rgS|jj|gddd¬«d}tj|j|«}g} tt||j««D]Œ\}
\} Š
|r‰
j|vrŒ|r‰
jr‰
2026-04-10 16:52:07 +08:00
j|krŒ8|dk(r‰
jdk7rŒM|d k(r‰
jd
2026-04-10 16:52:07 +08:00
k(rŒb|rt!ˆ
fd „|D««sŒy| j#|
| ‰
f«ŒŽ| j%d „¬
«| d|D� �
cgc]\} } }
|
‘Œ