Files
ai-g3sb-backman2.0/backend/utils/__pycache__/embedding.cpython-312.pyc
T

189 lines
22 KiB
Plaintext
Raw Normal View History

2026-04-10 16:52:07 +08:00
Ë
]ÚÝirGã ó`—UdZddlZddlmZmZmZmZddlZddl m
2026-04-10 16:52:07 +08:00
Z
 ddl m Z m
Z
ddlZdZddlZddlmZej(e«ZeGd „d
««Zd efd „ZGd
2026-04-10 16:52:07 +08:00
„d«Zddeded efd„ZGd„d«ZeZdaeee d< ddeedeeded efd„Z!d„Z"y#e$rdZYŒwxYw)u©
Embedding �装:兼容 OpenAI /v1/embeddings 的远程 API(默认),
2026-04-10 16:52:07 +08:00
或�选本地 HuggingFace 目录(USE_LOCAL_EMBEDDING=true + EMBEDDING_MODEL_PATH)。
éN)ÚUnionÚListÚOptionalÚAny)ÚPath)Ú AutoModelÚ
AutoTokenizerTF)Ú dataclasscóD—eZdZUdZeed<eed<eed<eed<eed<y)Ú_RemoteEmbeddingEnvuG从环境å�˜é‡�è§£æž�出的远程 OpenAI-Compatible Embedding é…�置。Úapi_keyÚbase_urlÚmodelÚ max_batchÚlabelN)Ú__name__Ú
__module__Ú __qualname__Ú__doc__ÚstrÚ__annotations__Úint©óúbackend/utils/embedding.pyr r s…áQà
ƒL؃MØ ƒJ؃NØ „Jrr ÚreturncóZ—tjdd«j«}tjdd«j«}tjdd«j«}tjd«xstjdd«j«}|r¸|xsd}tjd«xstjd d
«j«}tjd d «j«}|j«rt d
t |««nd}|s t
d«‚t||jd«||d¬«Stjdd«j«}|rð|jd«s|jd«r t
d«‚tjdd«j«xsd} tjdd«j«}
|
s t
d«‚tjdd«j«}|j«rt d
t |««nd}t|| jd«|
|d¬«S|s t
d«‚|s t
d «‚tjd!d«j«}|s t
d"«‚tjd#d$«j«}|j«rt d
t |««nd%}t||jd«||d&¬«S)'u`
优先 ModelScope(MODELSCOPE_*);未�置时回退 DashScope(DASHSCOPE_*)。
ÚMODELSCOPE_API_KEYÚÚMODELSCOPE_BASE_URLÚDASHSCOPE_API_KEYÚDASHSCOPE_BASE_URLÚDASHSCOPE_base_urlz&https://api-inference.modelscope.cn/v1ÚMODELSCOPE_EMBEDDING_MODELÚMODELSCOPE_MODELzQwen/Qwen3-Embedding-8BÚMODELSCOPE_EMBEDDING_MAX_BATCHÚ32éé u>未é…�ç½® MODELSCOPE_EMBEDDING_MODEL(或 MODELSCOPE_MODEL)Ú/Ú
ModelScope©r
rrrrÚOPENAI_API_KEYzhttp://zhttps://u‘OPENAI_API_KEY ä¸�能填写为 URL:请将网关地å�€å†™åˆ° OPENAI_BASE_URL(例如 http://host:9080/v1),密钥å�•独写在 OPENAI_API_KEYÚOPENAI_BASE_URLzhttps://api.openai.com/v1ÚOPENAI_EMBEDDING_MODELuB使用 OpenAI 兼容 Embedding 时请设置 OPENAI_EMBEDDING_MODELÚOPENAI_EMBEDDING_MAX_BATCHÚ100édÚOpenAIuÝ远程 Embedding 未é…�置:请设置 MODELSCOPE_API_KEY(å�Šå�¯é€‰ BASE_URL),或 OPENAI_API_KEY / OPENAI_EMBEDDING_MODEL(å�Šå�¯é€‰ OPENAI_BASE_URL),或 DASHSCOPE_API_KEY / DASHSCOPE_BASE_URL / DASHSCOPE_MODELu8未é…�ç½® DASHSCOPE_BASE_URL(或 DASHSCOPE_base_url)ÚDASHSCOPE_MODELu未é…�ç½® DASHSCOPE_MODELÚDASHSCOPE_EMBEDDING_MAX_BATCHÚ10é
Ú DashScope)
ÚosÚgetenvÚstripÚisdigitÚmaxrÚ
ValueErrorr ÚrstripÚ
startswith) Úms_keyÚms_baseÚds_keyÚds_baserrÚmbrÚoa_keyÚoa_baseÚoa_models rÚ_remote_embedding_from_envrI#sÉ€ô�Y‰YÐ+¨RÓ
0×
2026-04-10 16:52:07 +08:00
6Ñ
6Ó
8€FÜ�i‰iÐ-¨rÓ2×8Ñ8Ó:€GÜ
2026-04-10 16:52:07 +08:00
�Y‰YÐ*¨BÓ
/×
5Ñ
5Ó
2026-04-10 16:52:07 +08:00
7€Fä
� ‰ Ð&Ó'ÒN¬2¯9©9Ð5IÈ2Ó+Nß �eƒgð ñ
ØÒFÐFˆä �I‰IÐ2Ó 3ò
HÜ�y‰yÐ+Ð-FÓGß
2026-04-10 16:52:07 +08:00
‰%‹'ð ô�Y‰YÐ7¸Ó
>×
DÑ
DÓ
FˆØ')§z¡z¤|”C˜œ3˜r›7”O¸ˆ ÙÜÐ]Ó^Ð ^Ü"ØØ—_‘_ SÓ)ØØØô 
ð
ô�Y‰YÐ'¨Ó
,×
2026-04-10 16:52:07 +08:00
2Ñ
2Ó
4€FÙ
Ø × Ñ ˜YÔ '¨6×+<Ñ+<¸ZÔ+HÜðWóð
ô
2026-04-10 16:52:07 +08:00
�I‰IÐ'¨Ó ,× 2Ñ 2Ó 4Ò SÐ8Sð ô—9‘9Ð5°rÓ:×@Ñ@ÓBˆÙÜØTóð
ô�Y‰YÐ3°UÓ
;×
AÑ
AÓ
CˆØ')§z¡z¤|”C˜œ3˜r›7”O¸ˆ Ü"ØØ—^‘^ CÓ(ØØØô 
ð
ñ Üð
Kó
ð
ñ
ÜÐSÓTÐTÜ �I‰IÐ'¨Ó ,× 2Ñ 2Ó 4€EÙ ÜÐ4Ó5Ð5Ü �‰Ð2°DÓ 9× ?Ñ ?Ó A€BØ#%§:¡:¤<”�A”s˜2“w”°R€IÜ ØØ—‘ Ó$ØØØô  ðrc
óþ—eZdZdZ ddeedeedefd„Z ddeee efde
d ed
e
d ed e jf d
„Z
de jde jd e jfd„Z ddede ede
d e efd„Zy)ÚQwen3Embeddingu
本地 HuggingFace 格� Embedding 模型(Mean Pooling + L2,用于��检索)。
仅在 ``USE_LOCAL_EMBEDDING=true`` 时使用;路径由 ``model_path`` 或环境��
2026-04-10 16:52:07 +08:00
``EMBEDDING_MODEL_PATH`` 指定,**��内置默认目录**。
NÚ
model_pathÚdeviceÚuse_fp16cóî—ts td«‚|xsdj«xs$tjdd«j«}|s t d«‚t
|«}|j«std|›d�«‚|€"tjj«rdnd }||_ tjd
||« tj t#|«d ¬ «|_ t-j t#|«d ¬ «|_|j.j3«|j.j5|«|xr|d k7|_|j6r|j.j9«|j.j:j<|_tjd|j>›�«y#t&$rj}t#|«j)«}d
|vsd|vr=tj+d|«tj t#|«dd ¬«|_n‚Yd}~�ŒEd}~wwxYw#t
$r3}t#|«}d|j)«vsd|vr t1d«|‚‚d}~wwxYw)uS
�始化 embedding 模型
Args:
model_path: 本地模型目录;None 或空字符串时读 ``EMBEDDING_MODEL_PATH``
device: 推�设备('cpu', 'cuda', 'cuda:0'等),None则自动选择
use_fp16: 是�使用FP16混�精度(GPU�用时建议开�,速度更快)
ugtransformers 和 torch 未安装。请�行:
pip install transformers torch sentencepiece acceleraterÚEMBEDDING_MODEL_PATHuå·²å�¯ç”¨æœ¬åœ° Embedding(USE_LOCAL_EMBEDDING=true),但未设置有效模型路径。请在 .env 中设置 EMBEDDING_MODEL_PATH 指å�‘本地模型目录,或设置 USE_LOCAL_EMBEDDING=false 使用 OPENAI_* / MODELSCOPE_* / DASHSCOPE_* 远程接å�£ã€‚u)本地 Embedding 模型目录ä¸�存在:uH
请修正 EMBEDDING_MODEL_PATH,或改用 USE_LOCAL_EMBEDDING=false。NÚcudaÚcpuu0加载本地 Embedding 模型:%s,设备:%sT)Útrust_remote_codeÚ modelwrapperz
untagged enumuh快速 tokenizer è§£æž� tokenizer.json 失败(多为 tokenizers 过旧),改用 use_fast=False:%sF)Úuse_fastrSÚqwen3z$does not recognize this architectureuƒå½“å‰� transformers 版本ä¸�支æŒ� Qwen3(model_type=qwen3)。请å�‡çº§ï¼špip install "transformers>=4.51.0" "tokenizers>=0.21"u)[OK] 模型加载完æˆ�,嵌入维度:) Ú_TRANSFORMERS_AVAILABLEÚ ImportErrorr;r9r:r>rÚexistsÚFileNotFoundErrorÚtorchrQÚ is_availablerMÚloggerÚinfor Úfrom_pretrainedrÚ tokenizerÚ ExceptionÚlowerÚwarningrrÚ RuntimeErrorÚevalÚtorNÚhalfÚconfigÚ hidden_sizeÚ
2026-04-10 16:52:07 +08:00
embedding_dim)ÚselfrLrMrNÚresolvedÚeÚerrÚmsgs rÚ__init__zQwen3Embedding.__init__ysF€õ'ÜðJóð
ð
Ò$ "×+Ñ+Ó-Ò^´·±Ð;QÐSUÓ1V×1\Ñ1\Ó1^ˆÙÜðsóð
ô ˜(“^ˆ
Ø× Ñ Ô"Ü#Ø;¸J¸<ðHZðZóð
ð ˆ>Ü$Ÿz™z×6Ñ6Ô8‘V¸eˆFàˆŒ Ü� ‰ ÐFÈ
ÐTZÔ[ð Ü*×:Ñ:Ü�J“°4ôˆDŒNð" Ü"×2Ñ2Ü�J“°4ôˆDŒJð
�
‰
�‰ÔØ �
‰
�
2026-04-10 16:52:07 +08:00
‰
�fÔð!Ò4 V¨u¡_ˆŒ
Ø �=Š=Ø �J‰J�O‰OÔ ð"ŸZ™Z×.Ñ.×:Ñ:ˆÔÜ� ‰ Ð?À×@RÑ@RÐ?SÐTÕUøôMò Ü�a“&—,‘,“.ˆCØ Ñ$¨¸3Ñ(>Ü—‘ð1àôô
"/×!>Ñ!>ܘ
“O¨eÀtô"�•ðõ ûð ûô$ò Ü�a“&ˆCؘ#Ÿ)™)›+Ñ%Ð)OÐSVÑ)VÜ"ð\óððð
2026-04-10 16:52:07 +08:00
ûð ús1Ã%GÃ,%H8Ç H5Ç AH0È0H5È8 I4É.I/É/I4ÚtextsÚ
batch_sizeÚ normalizeÚ
max_lengthÚ
show_progressrcój—t|t«r|g}|s1tjd|jftj
¬«Sg}t
dt|«|«}|r ddlm}||d¬«}|D]ð} || | |z}
|j|
dd|d¬«j|j«} tj«5|jdi| ¤Ž} | j j#d ¬
«}
|jd k7r|
j%«}
|
j'«}
d d d «|r*tj(j+
d d¬
«}|
|dzz }
|j-
«Œòtj.|«j1tj
«S#t$rY�Œ3wxYw#1swYŒŠxYw)uâ
编�文本为��
Args:
texts: �个文本或文本列表
batch_size: 批处�大�(根�显存调整)
2026-04-10 16:52:07 +08:00
normalize: 是�L2归一化(余弦相似度必需)
max_length: 最大�列长度(模型支�8192,建议512-1024平衡速度与精度)
show_progress: 是�显示进度�(需安装tqdm)
2026-04-10 16:52:07 +08:00
Returns:
numpy数组,shape=(len(texts), embedding_dim)
r©Údtype©ÚtqdmÚ Embedding©ÚdescTÚpt)ÚpaddingÚ
truncationrtÚreturn_tensorsr()ÚdimrRN©ÚaxisÚkeepdims绽×Ùß|Û=r)Ú
isinstancerÚnpÚemptyrjÚfloat32ÚrangeÚlenrzrXr`rfrMr[Úno_gradrÚlast_hidden_stateÚmeanrRÚnumpyÚlinalgÚnormÚappendÚvstackÚastype)rkrqrrrsrtruÚall_embeddingsÚiteratorrzÚiÚbatchÚinputsÚoutputsÚ
embeddingsÚnormss rÚencodezQwen3Embedding.encodeÑs–€ô* �eœSÔ !Ø�GˆEáÜ—8‘8˜Q × 2Ñ 2Ð3¼2¿:¹:ÔFÐ Fàˆô˜œC ›J¨
2026-04-10 16:52:07 +08:00
Ó3ˆÙ ð
Ý%Ù ¨{Ô;�óˆAؘ!˜A 
™NÐ+ˆEð—^‘^ØØØØ%Ø#ð $ó÷ ‰b�—‘‹oð
ô—‘•Ø$˜$Ÿ*™*Ñ. vÑ.�ð%×6Ñ6×;Ñ;ÀÐ;ÓB�
2026-04-10 16:52:07 +08:00
ð—;‘; %Ò'Ø!+§¡Ó!1�JØ'×-Ñ-Ó/�
÷!ñäŸ ™ Ÿ™ z¸ÀD˜ÓI�Ø'¨5°5©=Ñ9�
à × !Ñ ! *Õ -ð;ô>�y‰y˜Ó(×/Ñ/´·
±
2026-04-10 16:52:07 +08:00
Ó;Ð;øôEò
Úð
ú÷ !�úsÁ"FÃAF)Æ F&Æ%F&Æ)F2 Úemb1Úemb2cóB—tj||j«S)u
计算两组embedding的余弦相似度
Args:
emb1: 第一组�� (n, dim)
emb2: 第二组�� (m, dim)
Returns:
相似度矩阵 (n, m),值域[-1, 1](若已归一化则为[0, 1])
©rˆÚdotÚT©rkrŸr s rÚ
similarityzQwen3Embedding.similaritys€ô �v‰v�d˜DŸF™FÓ#Ð#rÚqueryÚ documentsÚtop_kcó&—|j|gd¬«}|j|d¬«}|j||«d}tj|«ddd…d|}g}|D]/} |j t || «|| t
| «dœ«Œ1|S)u
便�方法:编�查询并检索最相似的文档
Args:
query: 查询文本
documents: 候选文档列表
top_k: 返回�K个结果
2026-04-10 16:52:07 +08:00
Returns:
[{"score": float, "document": str, "index": int}, ...]
T©rsrNéÿÿÿÿ©ÚscoreÚdocumentÚindex)ržr¦rˆÚargsortr“Úfloatr)
2026-04-10 16:52:07 +08:00
rkr§r¨r©Ú query_embÚdoc_embsÚscoresÚ top_indicesÚresultsÚidxs
rÚencode_and_searchz Qwen3Embedding.encode_and_search*s €ð"—K‘K  °4�KÓ8ˆ Ø—;‘;˜y°D�;Ó9ˆà—‘ ¨HÓ5°aÑ8ˆô—j‘j Ó(©¨2¨Ñ.¨v°Ð6ˆ àˆÛˆCØ �N‰Nܘv c™{Ó+Ø% c™NܘS›ñõ
2026-04-10 16:52:07 +08:00
ððˆr©NNF)r)Té F©é)rrrrrrÚboolrprrrrˆÚndarrayržr¦Údictr¹rrrrKrKqs„ñð%)Ø $Øñ VVà˜S‘MðVVð˜‘
ðVVðó VVðvØØØ#ñ