Credsweeper package

CredSweeper

class credsweeper.app.CredSweeper(rule_path=None, config_path=None, api_validation=False, json_filename=None, xlsx_filename=None, hashed=False, subtext=False, sort_output=False, use_filters=True, pool_count=1, ml_batch_size=None, ml_threshold=ThresholdPreset.medium, ml_config=None, ml_model=None, ml_providers=None, find_by_ext=False, depth=0, doc=False, severity=Severity.INFO, size_limit=None, exclude_lines=None, exclude_values=None, log_level=None)[source]

Bases: object

Advanced credential analyzer base class.

Parameters:

  • credential_manager – CredSweeper credential manager object

  • scanner – CredSweeper scanner object

  • pool_count (int) – number of pools used to run multiprocessing scanning

  • config – dictionary variable, stores analyzer features

  • json_filename (Union[None, str, Path]) – string variable, credential candidates export filename

class MlValidator(threshold, ml_config=None, ml_model=None, ml_providers=None)

Bases: object

ML validation class

CHAR_INDEX = {'\x00': 0, '\t': 96, '\n': 97, '\x0b': 99, '\x0c': 100, '\r': 98, ' ': 95, '!': 63, '"': 64, '#': 65, '$': 66, '%': 67, '&': 68, "'": 69, '(': 70, ')': 71, '*': 72, '+': 73, ',': 74, '-': 75, '.': 76, '/': 77, '0': 1, '1': 2, '2': 3, '3': 4, '4': 5, '5': 6, '6': 7, '7': 8, '8': 9, '9': 10, ':': 78, ';': 79, '<': 80, '=': 81, '>': 82, '?': 83, '@': 84, 'A': 37, 'B': 38, 'C': 39, 'D': 40, 'E': 41, 'F': 42, 'G': 43, 'H': 44, 'I': 45, 'J': 46, 'K': 47, 'L': 48, 'M': 49, 'N': 50, 'O': 51, 'P': 52, 'Q': 53, 'R': 54, 'S': 55, 'T': 56, 'U': 57, 'V': 58, 'W': 59, 'X': 60, 'Y': 61, 'Z': 62, '[': 85, '\\': 86, ']': 87, '^': 88, '_': 89, '`': 90, 'a': 11, 'b': 12, 'c': 13, 'd': 14, 'e': 15, 'f': 16, 'g': 17, 'h': 18, 'i': 19, 'j': 20, 'k': 21, 'l': 22, 'm': 23, 'n': 24, 'o': 25, 'p': 26, 'q': 27, 'r': 28, 's': 29, 't': 30, 'u': 31, 'v': 32, 'w': 33, 'x': 34, 'y': 35, 'z': 36, '{': 91, '|': 92, '}': 93, '~': 94, 'ÿ': 101}
MAX_LEN = 160
NON_ASCII = 'ÿ'
NUM_CLASSES = 102
static encode(text, limit)

Encodes prepared text to array

Return type:

ndarray

static encode_line(text, position)

Encodes line with balancing for position

static encode_value(text)

Encodes line with balancing for position

Return type:

ndarray

extract_common_features(candidates)

Extract features that are guaranteed to be the same for all candidates on the same line with same value.

Return type:

ndarray

extract_features(candidates)

extracts common and unique features from list of candidates

Return type:

ndarray

extract_unique_features(candidates)

Extract features that can be different between candidates. Join them with or operator.

Return type:

ndarray

get_group_features(candidates)

np.newaxis used to add new dimension if front, so input will be treated as a batch

Return type:

Tuple[ndarray, ndarray, ndarray, ndarray]

validate_groups(group_list, batch_size)

Use ml model on list of candidate groups.

Parameters:
Return type:

Tuple[ndarray, ndarray]

Returns:

Boolean numpy array with decision based on the threshold, and numpy array with probability predicted by the model

property config: Config

config getter

export_results()[source]

Save credential candidates to json file or print them to a console.

Return type:

None

file_scan(content_provider)[source]

Run scanning of file from ‘file_provider’.

Parameters:

content_provider (Union[DiffContentProvider, TextContentProvider]) – content provider object to scan

Return type:

List[Candidate]

Returns:

list of credential candidates from scanned file

files_scan(content_providers)[source]

Auxiliary method for scan one sequence

Return type:

List[Candidate]

property ml_validator: MlValidator

ml_validator getter

static pool_initializer(log_kwargs)[source]

Ignore SIGINT in child processes.

Return type:

None

post_processing()[source]

Machine learning validation for received credential candidates.

Return type:

None

run(content_provider)[source]

Run an analysis of ‘content_provider’ object.

Parameters:

content_provider (AbstractProvider) – path objects to scan

Return type:

int

scan(content_providers)[source]

Run scanning of files from an argument “content_providers”.

Parameters:

content_providers (Sequence[Union[DiffContentProvider, TextContentProvider]]) – file objects to scan

Return type:

None