credsweeper.ml_model.features package

Submodules

credsweeper.ml_model.features.char_set module

class credsweeper.ml_model.features.char_set.CharSet(base: str)[source]

Bases: Feature

Feature is true when all characters of the value are from a set.

CHARS: Dict[Base, Set[str]] = {Base.ascii_lowercase: {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, Base.ascii_uppercase: {'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'}, Base.base16lower: {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}, Base.base16upper: {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}, Base.base32: {'2', '3', '4', '5', '6', '7', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'}, Base.base36: {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, Base.base64std: {'+', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '=', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, Base.base64url: {'-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '=', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, Base.digits: {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}}
extract(candidate: Candidate) → bool[source]

Abstract method of base class

credsweeper.ml_model.features.feature module

class credsweeper.ml_model.features.feature.Feature[source]

Bases: ABC

Base class for features.

any_word_in_(a_string: str) → bool[source]

Returns true if any words in a string

abstract extract(candidate: Candidate) → Any[source]

Abstract method of base class

property words: List[str]

getter

credsweeper.ml_model.features.file_extension module

class credsweeper.ml_model.features.file_extension.FileExtension(extensions: List[str])[source]

Bases: WordIn

Categorical feature of file type.

Parameters:

extensions – extension labels

extract(candidate: Candidate) → Any[source]

Abstract method of base class

credsweeper.ml_model.features.hartley_entropy module

class credsweeper.ml_model.features.hartley_entropy.HartleyEntropy(base: str, norm: bool = False)[source]

Bases: RenyiEntropy

Hartley entropy feature.

credsweeper.ml_model.features.has_html_tag module

class credsweeper.ml_model.features.has_html_tag.HasHtmlTag[source]

Bases: Feature

Feature is true if line has HTML tags (HTML file).

extract(candidate: Candidate) → bool[source]

Abstract method of base class

credsweeper.ml_model.features.is_secret_numeric module

class credsweeper.ml_model.features.is_secret_numeric.IsSecretNumeric[source]

Bases: Feature

Feature is true if candidate value is a numerical value.

extract(candidate: Candidate) → bool[source]

Abstract method of base class

credsweeper.ml_model.features.reny_entropy module

class credsweeper.ml_model.features.reny_entropy.RenyiEntropy(base: str, alpha: float, norm=False)[source]

Bases: Feature

Renyi entropy.

See next link for details: https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf

Parameters:

  • alpha – entropy parameter

  • norm – set True to normalize output probabilities

CHARS: Dict[Base, Chars] = {Base.base32: Chars.BASE32_CHARS, Base.base36: Chars.BASE36_CHARS, Base.base64: Chars.BASE64_CHARS, Base.hex: Chars.HEX_CHARS}
estimate_entropy(p_x: ndarray) → float[source]

Calculate Renyi entropy of ‘p_x’ sequence.

Function is based on definition of Renyi entropy for arbitrary probability distribution. Please see next link for details: https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf

extract(candidate: Candidate) → ndarray[source]

Abstract method of base class

get_probabilities(data: str) → ndarray[source]

Get list of alphabet’s characters presented in inputted string.

credsweeper.ml_model.features.rule_name module

class credsweeper.ml_model.features.rule_name.RuleName(rule_names: List[str])[source]

Bases: WordIn

Categorical feature that corresponds to rule name.

Parameters:

rule_names – rule name labels

extract(candidate: Candidate) → Any[source]

Abstract method of base class

credsweeper.ml_model.features.search_in_attribute module

class credsweeper.ml_model.features.search_in_attribute.SearchInAttribute(pattern: str, attribute: str)[source]

Bases: Feature

Abstract feature returns boolean for matched pattern in member of first LineData

extract(candidate: Candidate) → bool[source]

Returns boolean for first LineData member

credsweeper.ml_model.features.shannon_entropy module

class credsweeper.ml_model.features.shannon_entropy.ShannonEntropy(base: str, norm: bool = False)[source]

Bases: RenyiEntropy

Shannon entropy feature.

credsweeper.ml_model.features.word_in module

class credsweeper.ml_model.features.word_in.WordIn(words: List[str])[source]

Bases: Feature

Abstract feature returns array with all matched words in a string

property dimension: int

getter

property enumerated_words: List[Tuple[int, str]]

getter for speedup

abstract extract(candidate: Candidate) → Any[source]

Abstract method of base class

word_in_set(a_strings_set: Set[str]) → ndarray[source]

Returns array with words matches in a_strings_set

word_in_str(a_string: str) → ndarray[source]

Returns array with words included in a string

credsweeper.ml_model.features.word_in_line module

class credsweeper.ml_model.features.word_in_line.WordInLine(words: List[str])[source]

Bases: WordIn

Feature is true if line contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray[source]

Returns true if any words in first line

credsweeper.ml_model.features.word_in_path module

class credsweeper.ml_model.features.word_in_path.WordInPath(words: List[str])[source]

Bases: WordIn

Categorical feature that corresponds to words in path (POSIX, lowercase)

extract(candidate: Candidate) → Any[source]

Abstract method of base class

credsweeper.ml_model.features.word_in_value module

class credsweeper.ml_model.features.word_in_value.WordInValue(words: List[str])[source]

Bases: WordIn

Feature returns true if candidate value contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray[source]

Returns array of matching words for first line

credsweeper.ml_model.features.word_in_variable module

class credsweeper.ml_model.features.word_in_variable.WordInVariable(words: List[str])[source]

Bases: WordIn

Feature returns array of words matching in variable

extract(candidate: Candidate) → ndarray[source]

Returns array of matching words for first line

Module contents