Oj:$SrSSKrSSKJrJr SSKJrJr SSKJr SSK J r SSK J r SS K J r SS K Jr \R "S 5r/S Qr"S S5rS\S\R*R,R.S\4SjrS\S\4Sjr"SS5rg)z)[Experimental] Text Only Local Tokenizer.N)AnyIterable)OptionalUnion)sentencepiece_model_pb2)_common)_local_tokenizer_loader) _transformers)typeszgoogle_genai.local_tokenizer)_parse_hex_byte_token_str_to_bytesLocalTokenizer_TextsAccumulatorc\rSrSrSrSSjrS\\4SjrS\\ RSS4Sjr S \ RSS4S jr S \ RSS4S jrS \ RS\ R4SjrS\\ RSS4SjrS\\ R$SS4SjrS\ R$SS4SjrS\ R*S\ R*4SjrS\ R.S\ R.4SjrS\\\4S\\\44SjrS\S\4SjrSrg)r'aAccumulates countable texts from `Content` and `Tool` objects. This class is responsible for traversing complex `Content` and `Tool` objects and extracting all the text content that should be included when calculating token counts. A key feature of this class is its ability to detect unsupported fields in `Content` objects. If a user provides a `Content` object with fields that this local tokenizer doesn't recognize (e.g., new fields added in a future API update), this class will log a warning. The detection mechanism for `Content` objects works by recursively building a "counted" version of the input object. This "counted" object only contains the data that was successfully processed and added to the text list for tokenization. After traversing the input, the original `Content` object is compared to the "counted" object. If they don't match, it signifies the presence of unsupported fields, and a warning is logged. returnNc/UlgN_textsselfs ^/var/www/html/leaf-doctor-v1/venv/lib/python3.13/site-packages/google/genai/local_tokenizer.py__init___TextsAccumulator.__init__;s DKcUR$rrrs r get_texts_TextsAccumulator.get_texts>s ;;rcontentsc8UHnURU5 M gr) add_content)rr!contents r add_contents_TextsAccumulator.add_contentsAs wrr$cx[R"/URS9nUR(GaLURGH;nURce[R"5nUR c UR b [S5eURbURUlURb,URUR5 URUl URb,URUR5 URUl URb6URUl URRUR5 URRU5 GM> UR!SS9UR!SS9:wa["R%SUSUS35 gg)N)partsrolez6LocalTokenizers do not support non-text content types.T) exclude_nonezHContent contains unsupported types for token counting. Supported fields z. Got .)r Contentr)r(Part file_data inline_data ValueErrorvideo_metadata function_calladd_function_callfunction_responseadd_function_responsetextrappend model_dumploggerwarning)rr$counted_contentpart counted_parts rr#_TextsAccumulator.add_contentEsmm"7<<@O}}}--$$$000zz| >> %)9)9)EF      *(,(;(;, %    )  !3!3 4'+'9'9, $  ! ! -  $ $T%;%; <+/+A+A, ( 99 "ii,  ++  TYY '$$\2% (t,0J0J1K1 nn $%VG9A 7rr2cUR(a%URRUR5 [R"URS9nUR (a"UR UR 5nX2lgg)z}Processes a function call and adds relevant text to the accumulator. Args: function_call: The function call to process. )nameN)r@rr7r FunctionCallargs_dict_traverse)rr2counted_function_call counted_argss rr3#_TextsAccumulator.add_function_calldsf  kk++,!..M4F4FG((););+ ''rschemac[R"5nUR(aURUlUR(aURUlURbURUlUR (a6UR RUR 5 UR UlUR(a6UR RUR5 URUlUR(a6UR RUR5 URUl UR(a6UR RUR5 URUl UR(aURUl UR(a!URUR5nX2l UR(a[0nURR5H5upVUR RU5 URU5nXtU'M7 XBlUR (a!UR#UR 5nXlU$)zProcesses a schema and adds relevant text to the accumulator. Args: schema: The schema to process. Returns: The new schema object with only countable fields. )r Schematypetitledefaultformatrr7raenumextendrequiredproperty_orderingitemsrc propertiesexample _any_traverse) rrfcounted_schemacounted_schema_itemsdkeyvalue counted_valuecounted_schema_examples rrc_TextsAccumulator.add_schemas\\^N {{"KKn ||#\\n ~~!%~~n }} kk'$mmn  kk++,#)#5#5n  {{ kk%"KKn  kk) &n )/)A)An& ||!__V\\:1  a))//1*# 3. #2#$ ~~#11&..A5 rrwc0nURR[UR555 UR 5Hup4UR U5X#'M U$)zProcesses a dict and adds relevant text to the accumulator. Args: d: The dict to process. Returns: The new dict object with only countable fields. )rrnlistkeysrqrt)rrw counted_dictrxvals rrC _TextsAccumulator._dict_traversesPLKKtAFFH~&GGI,,S1l rryc,[U[5(aURRU5 U$[U[5(aUR U5$[U[ 5(a!UVs/sHo RU5PM sn$U$s snf)zProcesses a value and adds relevant text to the accumulator. Args: value: The value to process. Returns: The new value with only countable fields. ) isinstancestrrr7dictrCr~rt)rryitems rrt_TextsAccumulator._any_traverses{% kk l E4   '' E4 38 954  &5 99 l:s2Br)rN)__name__ __module__ __qualname____firstlineno____doc__rrrrr r,r%r#rAr3rJrOrSrZrWr5r`rKrhrcrrrCrt__static_attributes__rrrr'sM& # 8EMM#: t 4> 0U-?-? 0D 0 5:: %** Xejj1d4 ()?)? @4 4 <$55 <  <("'";";(   ((+u||+ +Z d38n c3h rrtokenrircU[RRRR:Xa[ U5R SSS9$URSS5RS5$)Nrbig)length byteorderu▁ utf-8) r ModelProto SentencePieceTypeBYTEr to_bytesreplaceencode)rris rrrs\ $ / / = = B B G GG 5 ! * *!u * EE == $ + +G 44rc4[U5S:wa[SU35eURS5(aURS5(d[SU35e[ USSS5nUS :a[S U35eU$![a [S U35ef=f) zParses a hex byte string of the form '<0xXX>' and returns the integer value. Raises ValueError if the input is malformed or the byte value is invalid. zInvalid byte length: z<0x>zInvalid byte format: zInvalid hex value: zByte value out of range: )lenr0 startswithendswithint)rrs rr r s  Z1_ ,UG4 55   % s(;(; ,UG4 554 eAaj" C CZ 08 99 * 4 *5'2 334s A>>BcP\rSrSrSrS\4Sjr\R"S5SS.S\ \ R\ R4S \ \ RS \ R4S jj5r\R"S5S\ \ R\ R4S \ R"4S j5rS rg)ria[Experimental] Text Only Local Tokenizer. This class provides a local tokenizer for text only token counting. LIMITATIONS: - Only supports text based tokenization and no multimodal tokenization. - Forward compatibility depends on the open-source tokenizer models for future Gemini versions. - For token counting of tools and response schemas, the `LocalTokenizer` only supports `types.Tool` and `types.Schema` objects. Python functions or Pydantic models cannot be passed directly. model_namecd[R"U5UlSUlUR[R;a&[R "UR5Ulg[R"UR5Ul[R"UR5Ulgr) loaderget_tokenizer_name_tokenizer_name _model_protoGEMMA_TOKENIZER_TO_MODEL_NAMESget_huggingface_tokenizer _tokenizerload_model_protoget_sentencepiece)rrs rrLocalTokenizer.__init__$sz!44Z@DD vDDD889M9MNdo 11$2F2FGd001E1EFdorzThe SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.N)configr!rrc[R"U5n[5n[RR U=(d 05nUR U5 UR(aURUR5 UR(a@URR(a%URURR5 UR(a0UR [R"UR/55 URR[UR!555n[R""[%SU55S9$)aCounts the number of tokens in a given text. Args: contents: The contents to tokenize. config: The configuration for counting tokens. Returns: A `CountTokensResult` containing the total number of tokens. Usage: .. code-block:: python from google.genai import local_tokenizer tokenizer = local_tokenizer.LocalTokenizer(model_name='gemini-2.5-flash') result = tokenizer.count_tokens("What is your name?") print(result) # total_tokens=5 c38# UHn[U5v M g7fr)r).0tokenss r .LocalTokenizer.count_tokens..Vs?;V;s) total_tokens)t t_contentsrr CountTokensConfigmodel_validater%rQrSgeneration_configresponse_schemarcsystem_instructionrrr~rCountTokensResultsum)rr!rprocessed_contentstext_accumulator tokens_lists r count_tokensLocalTokenizer.count_tokens-s:h/(*  $ $ 3 3FLb AF!!"45 ||  . F$<$<$L$L!!&":":"J"JK   ##ALL&2K2K1L$MN//((.>.H.H.J)KLK  " "?;?? rc[R"U5n/n[5nUHUnURU5 UR(dM'URHnUR UR 5 M MW /nUR[R;aURR[UR555n[X5HupUR [R "U URR#U 5V s/sH4n U R%SS5RS5R%SS5PM6 sn U S95 M [R&"US9$URR)UR55n [X5HupUR [R "U R*Vs/sHoR,PM snU R*Vs/sHDn[/UR0UR2R*UR,R45PMF snU S95 M [R&"US9$s sn fs snfs snf)aComputes the tokens ids and string pieces in the input. Args: contents: The contents to tokenize. Returns: A `ComputeTokensResult` containing the token information. Usage: .. code-block:: python from google.genai import local_tokenizer tokenizer = local_tokenizer.LocalTokenizer(model_name='gemini-2.5-flash') result = tokenizer.compute_tokens("What is your name?") print(result) # tokens_info=[TokensInfo(token_ids=[279, 329, 1313, 2508, 13], tokens=[b' What', b' is', b' your', b' name', b'?'], role='user')] _rrs▁ ) token_idsrr)) tokens_info)rrrr#r(r7r)rrrrrr~rzipr TokensInfoconvert_ids_to_tokensrComputeTokensResultEncodeAsImmutableProtopiecesidrpiecerri)rr!rrolesrr$r token_infosrtoken_idr)r tokens_protos tokens_protors rcompute_tokensLocalTokenizer.compute_tokensYs#4h/ E(*%""7+ A ,,w|| $& K vDDD//((.>.H.H.J)KLi 1.(   " "&!F!Fx!P "QMM#s+VG_W_d34"Q    2 & &; ??OO::""$M"-7    /;/B/BC/Be/BC ,22  3e&kk4#4#4#;#;EHH#E#J#J 3    8  $ $ ==9$Ds;I*I/=A I4)rrr)rrrrrrrr experimental_warningrr ContentListUnionContentListUnionDictrCountTokensConfigOrDictrrrrrrrrrrs GG B9= &e,,e.H.HHI&u445 &   & &P BE>e,,e.H.HHIE>   E> E>rr)rloggingtypingrrrr sentencepiecerr r rr rr getLoggerr9__all__rrrrrbytesrrr rrrrrs 0 "1/    9 : MM`5 5-88FFKK5 5 3 3 ,L>L>r