credsweeper.ml_model package

Subpackages

Submodules

credsweeper.ml_model.ml_validator module

class credsweeper.ml_model.ml_validator.MlValidator(threshold: float | ThresholdPreset, ml_config: None | str | Path = None, ml_model: None | str | Path = None, ml_providers: str | None = None)[source]

Bases: object

ML validation class

CHAR_INDEX = {'\x00': 0, '\t': 96, '\n': 97, '\x0b': 99, '\x0c': 100, '\r': 98, ' ': 95, '!': 63, '"': 64, '#': 65, '$': 66, '%': 67, '&': 68, "'": 69, '(': 70, ')': 71, '*': 72, '+': 73, ',': 74, '-': 75, '.': 76, '/': 77, '0': 1, '1': 2, '2': 3, '3': 4, '4': 5, '5': 6, '6': 7, '7': 8, '8': 9, '9': 10, ':': 78, ';': 79, '<': 80, '=': 81, '>': 82, '?': 83, '@': 84, 'A': 37, 'B': 38, 'C': 39, 'D': 40, 'E': 41, 'F': 42, 'G': 43, 'H': 44, 'I': 45, 'J': 46, 'K': 47, 'L': 48, 'M': 49, 'N': 50, 'O': 51, 'P': 52, 'Q': 53, 'R': 54, 'S': 55, 'T': 56, 'U': 57, 'V': 58, 'W': 59, 'X': 60, 'Y': 61, 'Z': 62, '[': 85, '\\': 86, ']': 87, '^': 88, '_': 89, '`': 90, 'a': 11, 'b': 12, 'c': 13, 'd': 14, 'e': 15, 'f': 16, 'g': 17, 'h': 18, 'i': 19, 'j': 20, 'k': 21, 'l': 22, 'm': 23, 'n': 24, 'o': 25, 'p': 26, 'q': 27, 'r': 28, 's': 29, 't': 30, 'u': 31, 'v': 32, 'w': 33, 'x': 34, 'y': 35, 'z': 36, '{': 91, '|': 92, '}': 93, '~': 94, 'ÿ': 101}
MAX_LEN = 160
NON_ASCII = 'ÿ'
NUM_CLASSES = 102
static encode(text: str, limit: int) → ndarray[source]

Encodes prepared text to array

static encode_line(text: str, position: int)[source]

Encodes line with balancing for position

static encode_value(text: str) → ndarray[source]

Encodes line with balancing for position

extract_common_features(candidates: List[Candidate]) → ndarray[source]

Extract features that are guaranteed to be the same for all candidates on the same line with same value.

extract_features(candidates: List[Candidate]) → ndarray[source]

extracts common and unique features from list of candidates

extract_unique_features(candidates: List[Candidate]) → ndarray[source]

Extract features that can be different between candidates. Join them with or operator.

get_group_features(candidates: List[Candidate]) → Tuple[ndarray, ndarray, ndarray, ndarray][source]

np.newaxis used to add new dimension if front, so input will be treated as a batch

validate_groups(group_list: List[Tuple[CandidateKey, List[Candidate]]], batch_size: int) → Tuple[ndarray, ndarray][source]

Use ml model on list of candidate groups.

Parameters:

  • group_list – List of tuples (value, group)

  • batch_size – ML model batch

Returns:

Boolean numpy array with decision based on the threshold, and numpy array with probability predicted by the model

Module contents