Serialized Form

  • Package org.apache.tika.ml.chardetect

    • Class org.apache.tika.ml.chardetect.MojibusterEncodingDetector

      class MojibusterEncodingDetector extends Object implements Serializable

    • Class org.apache.tika.ml.chardetect.NaiveBayesBigramEncodingDetector

      class NaiveBayesBigramEncodingDetector extends Object implements Serializable

      • Serialized Fields

        • charsets
          Charset[] charsets
          Charset objects cached at load — one Charset.forName per class, ever.
        • cohorts
          NaiveBayesBigramEncodingDetector.Cohort[] cohorts
          Per-class cohort, parallel to NaiveBayesBigramEncodingDetector.labels.
        • gb18030ClassIdx
          int gb18030ClassIdx
        • idf8
          byte[] idf8
          Global per-bigram IDF = log((C+1)/(df_i+1)) baked in at training, quantized to int8 at load via idfScale = maxAbs(idf)/127. IDF is non-negative so int8 values land in [0, 127]. Zero means "bigram appears in every class, no signal" and is the hot-loop skip condition.
        • labels
          String[] labels
        • logP8
          byte[] logP8
          Bigram-major int8 logP layout. Quantized at load time via per-class scale scale[c] = maxAbs(class c's logP column) / 127. In-memory footprint: 65_536 × numClasses bytes ≈ 2.1 MB for 34 classes, 4× smaller than float32. The hot-loop accumulates raw int8 products and applies dequantization once at the end of the probe, CharSoup-style.
        • numClasses
          int numClasses
        • perClassDequant
          double[] perClassDequant
          Per-class dequantization constant folded from scale[c] * idfScale / logVocabSize[c]. Applied once per class at the end of the probe to convert int accumulator to the final log-score. Keeping log V(c) in the dequant constant preserves the B-3 per-class score normalization from the float-path at zero additional cost.

    • Class org.apache.tika.ml.chardetect.Utf16SpecialistEncodingDetector

      class Utf16SpecialistEncodingDetector extends Object implements Serializable

      • Serialized Fields