credsweeper.ml_model.features package

Submodules

credsweeper.ml_model.features.entropy_evaluation module

class credsweeper.ml_model.features.entropy_evaluation.EntropyEvaluation

Bases: Feature

Renyi, Shannon entropy evaluation with Hartley entropy normalization. Augmentation with possible set of chars (hex, base64, etc.) Analyse only begin of the value

See next link for details: https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf

CHAR_SET: List[Set[str]] = [{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F', 'a', 'b', 'c', 'd', 'e', 'f'}, {'-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}, {'-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}, {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}, {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}, {'2', '3', '4', '5', '6', '7', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'}, {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '=', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'+', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'+', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '=', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'}, {'!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~'}, {'\t', '\n', '\x0b', '\x0c', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~'}]

HUNK_SIZE = 256

LOG2_CACHE: Dict[int, float] = {4: 2.0, 5: 2.321928094887362, 6: 2.584962500721156, 7: 2.807354922057604, 8: 3.0, 9: 3.169925001442312, 10: 3.321928094887362, 11: 3.4594316186372973, 12: 3.584962500721156, 13: 3.700439718141092, 14: 3.807354922057604, 15: 3.9068905956085187, 16: 4.0, 17: 4.087462841250339, 18: 4.169925001442312, 19: 4.247927513443585, 20: 4.321928094887363, 21: 4.392317422778761, 22: 4.459431618637297, 23: 4.523561956057013, 24: 4.584962500721156, 25: 4.643856189774724, 26: 4.700439718141092, 27: 4.754887502163468, 28: 4.807354922057604, 29: 4.857980995127572, 30: 4.906890595608519, 31: 4.954196310386875, 32: 5.0, 33: 5.044394119358453, 34: 5.087462841250339, 35: 5.129283016944966, 36: 5.169925001442312, 37: 5.20945336562895, 38: 5.247927513443585, 39: 5.285402218862249, 40: 5.321928094887363, 41: 5.357552004618084, 42: 5.392317422778761, 43: 5.426264754702098, 44: 5.459431618637297, 45: 5.491853096329675, 46: 5.523561956057013, 47: 5.554588851677638, 48: 5.584962500721156, 49: 5.614709844115208, 50: 5.643856189774724, 51: 5.672425341971495, 52: 5.700439718141092, 53: 5.727920454563199, 54: 5.754887502163468, 55: 5.78135971352466, 56: 5.807354922057604, 57: 5.832890014164741, 58: 5.857980995127572, 59: 5.882643049361842, 60: 5.906890595608519, 61: 5.930737337562887, 62: 5.954196310386875, 63: 5.977279923499917, 64: 6.0, 65: 6.022367813028454, 66: 6.044394119358453, 67: 6.066089190457772, 68: 6.087462841250339, 69: 6.108524456778169, 70: 6.129283016944966, 71: 6.149747119504682, 72: 6.169925001442312, 73: 6.189824558880018, 74: 6.20945336562895, 75: 6.22881869049588, 76: 6.247927513443585, 77: 6.266786540694901, 78: 6.285402218862249, 79: 6.303780748177103, 80: 6.321928094887363, 81: 6.339850002884624, 82: 6.357552004618084, 83: 6.3750394313469245, 84: 6.392317422778761, 85: 6.409390936137702, 86: 6.426264754702098, 87: 6.442943495848728, 88: 6.459431618637297, 89: 6.475733430966398, 90: 6.491853096329675, 91: 6.507794640198696, 92: 6.523561956057013, 93: 6.539158811108031, 94: 6.554588851677638, 95: 6.569855608330948, 96: 6.584962500721156, 97: 6.599912842187128, 98: 6.614709844115208, 99: 6.6293566200796095, 100: 6.643856189774724, 101: 6.658211482751795, 102: 6.672425341971495, 103: 6.6865005271832185, 104: 6.700439718141092, 105: 6.714245517666122, 106: 6.727920454563199, 107: 6.741466986401147, 108: 6.754887502163468, 109: 6.768184324776926, 110: 6.78135971352466, 111: 6.794415866350106, 112: 6.807354922057604, 113: 6.820178962415188, 114: 6.832890014164741, 115: 6.845490050944375, 116: 6.857980995127572, 117: 6.870364719583405, 118: 6.882643049361842, 119: 6.894817763307944, 120: 6.906890595608519, 121: 6.918863237274595, 122: 6.930737337562887, 123: 6.94251450533924, 124: 6.954196310386875, 125: 6.965784284662087, 126: 6.977279923499917, 127: 6.9886846867721655, 128: 7.0, 129: 7.011227255423254, 130: 7.022367813028454, 131: 7.03342300153745, 132: 7.044394119358453, 133: 7.05528243550119, 134: 7.066089190457772, 135: 7.076815597050831, 136: 7.087462841250339, 137: 7.098032082960526, 138: 7.108524456778169, 139: 7.118941072723508, 140: 7.129283016944966, 141: 7.139551352398794, 142: 7.149747119504682, 143: 7.159871336778389, 144: 7.169925001442312, 145: 7.1799090900149345, 146: 7.189824558880018, 147: 7.199672344836364, 148: 7.20945336562895, 149: 7.219168520462161, 150: 7.22881869049588, 151: 7.2384047393250786, 152: 7.247927513443585, 153: 7.257387842692652, 154: 7.266786540694901, 155: 7.2761244052742375, 156: 7.285402218862249, 157: 7.294620748891627, 158: 7.303780748177103, 159: 7.312882955284356, 160: 7.321928094887363, 161: 7.330916878114617, 162: 7.339850002884624, 163: 7.348728154231077, 164: 7.357552004618084, 165: 7.366322214245816, 166: 7.3750394313469245, 167: 7.383704292474052, 168: 7.392317422778761, 169: 7.400879436282184, 170: 7.409390936137702, 171: 7.417852514885898, 172: 7.426264754702098, 173: 7.434628227636725, 174: 7.442943495848728, 175: 7.451211111832329, 176: 7.459431618637297, 177: 7.467605550082998, 178: 7.475733430966398, 179: 7.483815777264256, 180: 7.491853096329675, 181: 7.499845887083206, 182: 7.507794640198696, 183: 7.515699838284043, 184: 7.523561956057013, 185: 7.531381460516312, 186: 7.539158811108031, 187: 7.546894459887636, 188: 7.554588851677638, 189: 7.562242424221073, 190: 7.569855608330948, 191: 7.577428828035749, 192: 7.584962500721156, 193: 7.592457037268081, 194: 7.599912842187128, 195: 7.60733031374961, 196: 7.614709844115208, 197: 7.622051819456376, 198: 7.6293566200796095, 199: 7.636624620543649, 200: 7.643856189774724, 201: 7.651051691178929, 202: 7.658211482751795, 203: 7.6653359171851765, 204: 7.672425341971495, 205: 7.679480099505446, 206: 7.6865005271832185, 207: 7.693486957499325, 208: 7.700439718141092, 209: 7.7073591320808825, 210: 7.714245517666122, 211: 7.721099188707185, 212: 7.727920454563199, 213: 7.734709620225838, 214: 7.741466986401147, 215: 7.7481928495894605, 216: 7.754887502163468, 217: 7.7615512324444795, 218: 7.768184324776926, 219: 7.774787059601174, 220: 7.78135971352466, 221: 7.787902559391432, 222: 7.794415866350106, 223: 7.800899899920305, 224: 7.807354922057604, 225: 7.813781191217037, 226: 7.820178962415188, 227: 7.826548487290915, 228: 7.832890014164741, 229: 7.839203788096944, 230: 7.845490050944375, 231: 7.851749041416057, 232: 7.857980995127572, 233: 7.864186144654281, 234: 7.870364719583405, 235: 7.876516946564999, 236: 7.882643049361842, 237: 7.888743248898259, 238: 7.894817763307944, 239: 7.900866807980749, 240: 7.906890595608519, 241: 7.912889336229962, 242: 7.918863237274595, 243: 7.924812503605781, 244: 7.930737337562887, 245: 7.936637939002571, 246: 7.94251450533924, 247: 7.948367231584678, 248: 7.954196310386875, 249: 7.960001932068081, 250: 7.965784284662087, 251: 7.971543553950772, 252: 7.977279923499917, 253: 7.98299357469431, 254: 7.9886846867721655, 255: 7.994353436858858, 256: 8.0}

RESULT_SIZE = 17

extract(candidate: Candidate) → ndarray

Returns real entropy and possible sets of characters

credsweeper.ml_model.features.feature module

class credsweeper.ml_model.features.feature.Feature

Bases: ABC

Base class for features.

abstract extract(candidate: Candidate) → Any

Abstract method of base class

credsweeper.ml_model.features.file_extension module

class credsweeper.ml_model.features.file_extension.FileExtension(extensions: List[str])

Bases: WordIn

Categorical feature of file type.

Parameters:

extensions – extension labels

extract(candidate: Candidate) → Any

Abstract method of base class

credsweeper.ml_model.features.has_html_tag module

class credsweeper.ml_model.features.has_html_tag.HasHtmlTag

Bases: WordIn

Feature is true if line has HTML tags (HTML file).

HTML_WORDS = ['< img', '<img', '< script', '<script', '< p', '<p', '< link', '<link', '< meta', '<meta', '< a', '<a']

extract(candidate: Candidate) → float

Abstract method of base class

credsweeper.ml_model.features.is_secret_numeric module

class credsweeper.ml_model.features.is_secret_numeric.IsSecretNumeric

Bases: Feature

Feature is true if candidate value is a numerical value.

extract(candidate: Candidate) → float

Abstract method of base class

credsweeper.ml_model.features.length_of_attribute module

class credsweeper.ml_model.features.length_of_attribute.LengthOfAttribute(attribute: str)

Bases: Feature

Abstract class for obtain a normalized value of length with max size of hunk

extract(candidate: Candidate) → ndarray

Returns boolean for first LineData member

credsweeper.ml_model.features.morpheme_dense module

class credsweeper.ml_model.features.morpheme_dense.MorphemeDense

Bases: Feature

Feature calculates morphemes density for a value

extract(candidate: Candidate) → float

Abstract method of base class

credsweeper.ml_model.features.rule_name module

class credsweeper.ml_model.features.rule_name.RuleName(rule_names: List[str])

Bases: WordIn

Categorical feature that corresponds to rule name.

Parameters:

rule_names – rule name labels

extract(candidate: Candidate) → Any

Abstract method of base class

credsweeper.ml_model.features.rule_severity module

class credsweeper.ml_model.features.rule_severity.RuleSeverity

Bases: Feature

Categorical feature that corresponds to rule name.

extract(candidate: Candidate) → float

Abstract method of base class

credsweeper.ml_model.features.search_in_attribute module

class credsweeper.ml_model.features.search_in_attribute.SearchInAttribute(pattern: str, attribute: str)

Bases: Feature

Abstract feature returns boolean for matched pattern in member of first LineData

extract(candidate: Candidate) → float

Returns boolean for first LineData member

credsweeper.ml_model.features.word_in module

class credsweeper.ml_model.features.word_in.WordIn(words: List[str])

Bases: Feature

Abstract feature returns array with all matched words in a string

abstract extract(candidate: Candidate) → Any

Abstract method of base class

word_in_(iterable_data: str | List[str] | Set[str]) → ndarray

Returns array with words included in a string

property zero: ndarray

Returns zero filled array for case of empty input

credsweeper.ml_model.features.word_in_path module

class credsweeper.ml_model.features.word_in_path.WordInPath(words: List[str])

Bases: WordIn

Categorical feature that corresponds to words in path (POSIX, lowercase)

extract(candidate: Candidate) → Any

Abstract method of base class

credsweeper.ml_model.features.word_in_postamble module

class credsweeper.ml_model.features.word_in_postamble.WordInPostamble(words: List[str])

Bases: WordIn

Feature is true if line contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray

Returns true if any words in a part of line after value

credsweeper.ml_model.features.word_in_preamble module

class credsweeper.ml_model.features.word_in_preamble.WordInPreamble(words: List[str])

Bases: WordIn

Feature is true if line contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray

Returns true if any words in line before variable or value

credsweeper.ml_model.features.word_in_transition module

class credsweeper.ml_model.features.word_in_transition.WordInTransition(words: List[str])

Bases: WordIn

Feature is true if line contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray

Returns true if any words between variable and value

credsweeper.ml_model.features.word_in_value module

class credsweeper.ml_model.features.word_in_value.WordInValue(words: List[str])

Bases: WordIn

Feature returns true if candidate value contains at least one word from predefined list.

extract(candidate: Candidate) → ndarray

Returns array of matching words for first line

credsweeper.ml_model.features.word_in_variable module

class credsweeper.ml_model.features.word_in_variable.WordInVariable(words: List[str])

Bases: WordIn

Feature returns array of words matching in variable

extract(candidate: Candidate) → ndarray

Returns array of matching words for first line

Module contents