credsweeper.ml_model.features package

Submodules

credsweeper.ml_model.features.char_set module

class credsweeper.ml_model.features.char_set.CharSet(base)[source]

Bases: Feature

Feature is true when all characters of the value are from a set.

CHARS: Dict[Base, Set[str]] = {<Base.digits: 'digits'>: {'2', '4', '9', '7', '8', '0', '6', '1', '3', '5'}, <Base.ascii_uppercase: 'ascii_uppercase'>: {'G', 'S', 'W', 'L', 'U', 'O', 'I', 'P', 'D', 'N', 'Q', 'Y', 'F', 'H', 'V', 'E', 'C', 'J', 'T', 'M', 'X', 'A', 'B', 'R', 'K', 'Z'}, <Base.ascii_lowercase: 'ascii_lowercase'>: {'q', 'a', 'u', 'k', 'o', 'z', 'v', 'n', 'd', 'p', 'r', 't', 'm', 'i', 's', 'l', 'x', 'j', 'y', 'w', 'h', 'e', 'b', 'f', 'c', 'g'}, <Base.base16upper: 'base16upper'>: {'2', '4', '9', 'F', '7', 'D', 'A', '8', 'B', 'E', 'C', '0', '6', '1', '3', '5'}, <Base.base16lower: 'base16lower'>: {'2', 'f', '4', '9', 'c', '7', 'e', '8', 'a', '0', '6', '1', '3', 'd', 'b', '5'}, <Base.base32: 'base32'>: {'G', 'S', 'W', 'L', 'U', 'O', 'I', 'P', '7', 'D', 'N', 'Q', '6', 'Y', '4', 'F', 'H', 'V', 'E', 'C', 'J', 'T', '3', 'M', '2', 'X', 'A', 'B', 'R', 'K', 'Z', '5'}, <Base.base36: 'base36'>: {'q', 'a', 'u', 'k', 'o', 'z', '0', 'v', 'n', 'd', '5', 'p', '9', 'r', '7', 't', 'm', '6', 'i', 's', 'l', 'x', 'j', 'y', 'w', '4', 'h', '1', 'e', '3', 'b', 'f', '2', 'c', '8', 'g'}, <Base.base64std: 'base64std'>: {'G', 'S', 'W', 'q', 'a', 'u', 'k', 'o', 'z', '0', 'L', 'U', 'v', 'O', 'd', 'n', 'I', '5', 'p', '9', 'P', 'r', '7', 'D', 't', 'm', 'N', 'Q', '6', 'Y', '+', 'i', 'l', 's', 'x', '/', 'j', 'y', 'F', 'w', '4', 'H', 'V', 'h', 'E', 'C', '=', 'J', '1', 'T', 'e', '3', 'M', 'b', 'f', '2', 'c', 'X', 'A', 'B', '8', 'R', 'K', 'Z', 'g'}, <Base.base64url: 'base64url'>: {'G', 'S', 'W', 'q', 'a', 'u', 'k', 'o', 'z', '0', 'L', 'U', 'v', 'O', 'd', 'n', 'I', '5', 'p', '9', 'P', 'r', '7', 'D', 't', 'm', 'N', 'Q', '6', 'Y', 'i', 'l', 's', 'x', 'j', 'y', 'F', 'w', '4', 'H', 'V', 'h', '-', 'E', 'C', '_', '=', 'J', '1', 'T', 'e', '3', 'M', 'b', 'f', '2', 'c', 'X', 'A', 'B', '8', 'R', 'K', 'Z', 'g'}}
extract(candidate)[source]

Abstract method of base class

Return type:

bool

credsweeper.ml_model.features.feature module

class credsweeper.ml_model.features.feature.Feature[source]

Bases: ABC

Base class for features.

any_word_in_(a_string)[source]

Returns true if any words in a string

Return type:

bool

abstract extract(candidate)[source]

Abstract method of base class

Return type:

Any

property words: List[str]

getter

credsweeper.ml_model.features.file_extension module

class credsweeper.ml_model.features.file_extension.FileExtension(extensions)[source]

Bases: WordIn

Categorical feature of file type.

Parameters:

extensions (List[str]) – extension labels

extract(candidate)[source]

Abstract method of base class

Return type:

Any

credsweeper.ml_model.features.hartley_entropy module

class credsweeper.ml_model.features.hartley_entropy.HartleyEntropy(base, norm=False)[source]

Bases: RenyiEntropy

Hartley entropy feature.

credsweeper.ml_model.features.has_html_tag module

class credsweeper.ml_model.features.has_html_tag.HasHtmlTag[source]

Bases: Feature

Feature is true if line has HTML tags (HTML file).

extract(candidate)[source]

Abstract method of base class

Return type:

bool

credsweeper.ml_model.features.is_secret_numeric module

class credsweeper.ml_model.features.is_secret_numeric.IsSecretNumeric[source]

Bases: Feature

Feature is true if candidate value is a numerical value.

extract(candidate)[source]

Abstract method of base class

Return type:

bool

credsweeper.ml_model.features.search_in_attribute module

class credsweeper.ml_model.features.search_in_attribute.SearchInAttribute(pattern, attribute)[source]

Bases: Feature

Abstract feature returns boolean for matched pattern in member of first LineData

extract(candidate)[source]

Returns boolean for first LineData member

Return type:

bool

credsweeper.ml_model.features.reny_entropy module

class credsweeper.ml_model.features.reny_entropy.RenyiEntropy(base, alpha, norm=False)[source]

Bases: Feature

Renyi entropy.

See next link for details: https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf

Parameters:

  • alpha (float) – entropy parameter

  • norm – set True to normalize output probabilities

CHARS: Dict[Base, Chars] = {<Base.base32: 'base32'>: <Chars.BASE32_CHARS: 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'>, <Base.base36: 'base36'>: <Chars.BASE36_CHARS: 'abcdefghijklmnopqrstuvwxyz1234567890'>, <Base.base64: 'base64'>: <Chars.BASE64_CHARS: 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/='>, <Base.hex: 'hex'>: <Chars.HEX_CHARS: '0123456789ABCDEFabcdef'>}
estimate_entropy(p_x)[source]

Calculate Renyi entropy of ‘p_x’ sequence.

Function is based on definition of Renyi entropy for arbitrary probability distribution. Please see next link for details: https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf

Return type:

float

extract(candidate)[source]

Abstract method of base class

Return type:

ndarray

get_probabilities(data)[source]

Get list of alphabet’s characters presented in inputted string.

Return type:

ndarray

credsweeper.ml_model.features.rule_name module

class credsweeper.ml_model.features.rule_name.RuleName(rule_names)[source]

Bases: WordIn

Categorical feature that corresponds to rule name.

Parameters:

rule_names (List[str]) – rule name labels

extract(candidate)[source]

Abstract method of base class

Return type:

Any

credsweeper.ml_model.features.shannon_entropy module

class credsweeper.ml_model.features.shannon_entropy.ShannonEntropy(base, norm=False)[source]

Bases: RenyiEntropy

Shannon entropy feature.

base: Base

credsweeper.ml_model.features.word_in module

class credsweeper.ml_model.features.word_in.WordIn(words)[source]

Bases: Feature

Abstract feature returns array with all matched words in a string

property dimension: int

getter

property enumerated_words: List[Tuple[int, str]]

getter for speedup

abstract extract(candidate)[source]

Abstract method of base class

Return type:

Any

word_in_set(a_strings_set)[source]

Returns array with words matches in a_strings_set

Return type:

ndarray

word_in_str(a_string)[source]

Returns array with words included in a string

Return type:

ndarray

credsweeper.ml_model.features.word_in_line module

class credsweeper.ml_model.features.word_in_line.WordInLine(words)[source]

Bases: WordIn

Feature is true if line contains at least one word from predefined list.

extract(candidate)[source]

Returns true if any words in first line

Return type:

ndarray

credsweeper.ml_model.features.word_in_path module

class credsweeper.ml_model.features.word_in_path.WordInPath(words)[source]

Bases: WordIn

Categorical feature that corresponds to words in path (POSIX, lowercase)

extract(candidate)[source]

Abstract method of base class

Return type:

Any

credsweeper.ml_model.features.word_in_value module

class credsweeper.ml_model.features.word_in_value.WordInValue(words)[source]

Bases: WordIn

Feature returns true if candidate value contains at least one word from predefined list.

extract(candidate)[source]

Returns array of matching words for first line

Return type:

ndarray

credsweeper.ml_model.features.word_in_variable module

class credsweeper.ml_model.features.word_in_variable.WordInVariable(words)[source]

Bases: WordIn

Feature returns array of words matching in variable

extract(candidate)[source]

Returns array of matching words for first line

Return type:

ndarray

Module contents