Output

Problematic

How can we generate or work with outputs ?

Documentation

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the output related interfaces.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/special-thanks.html
Contributors:
https://pyfunceble.github.io/contributors.html
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/master/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Clean()

class PyFunceble.output.clean.Clean(clean_all=False, file_path=None)[source]

Provide the cleaning logic.

Note

By cleaning we mean the cleaning of the output directory.

Parameters:
  • list_to_test (list|None) – The list of domains we are testing.
  • clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.
  • file_path (str) –

    The path to the file we tested.

    Note

    This is only relevant if you use the MariaDB/MySQL database.

almost_everything(clean_all=False, file_path=False)[source]

Delete almost all discovered files.

Parameters:clean_all (bool) – Tell the subsystem if we have to clean everything instesd of almost everything.
classmethod databases_to_delete()[source]

Set the databases files to delete.

classmethod file_to_delete(all_files=False)[source]

Return the list of file to delete.

Constructor()

class PyFunceble.output.constructor.Constructor(production=False)[source]

Basically a backup/reconstructor of our output directory.

Parameters:production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode.
classmethod _create_directory(directory, loop=False)[source]

Creates the given directory if it does not exists.

Parameters:
  • directory (str) – The directory to create.
  • loop (bool) – Tell us if we are in the creation loop or not.
_get_structure()[source]

Get the structure we are going to work with.

Returns:The structure we have to work with.
Return type:dict
_restore_replace()[source]

Check if we need to replace “.gitignore” to “.keep”.

Returns:The replacement status.
Return type:bool
_update_structure_from_config(structure)[source]

Update the paths according to configs.

Parameters:structure (dict) – The read structure.
backup()[source]

Backup the developer state of output/ in order to make it restorable and portable for user.

delete_uneeded()[source]

Delete the directory which are not registered into our structure.

restore()[source]

Restore the ‘output/’ directory structure based on the dir_structure.json file.

Generate()

class PyFunceble.output.generate.Generate(subject, subject_type, status, source=None, expiration_date=None, http_status_code=None, whois_server='Unknown', filename=None, ip_validation=False, end=False)[source]

Generate different sort of files.

Parameters:
  • subject (str) – The subject we are working with.
  • subject_type (str) – The type of the subject.
  • status (str) – The catched status.
  • source (str) – The source of the given status.
  • expiration_date (str) – The expiration date of the domain (if catched).
  • http_status_code (str|int) – The HTTP status code.
  • whois_server (str) – The whois server.
  • filename (str) – The name of the file we are testing.
  • ip_validation (bool) – The IP validation check of the currently written subject.
_Generate___get_info_files_destinations(output_hosts, output_domains, output_json)

Given the output directory, this method return several paths.

Note

The given output directories have to be partially completed.

Indeed, we only do output % final_location.

Returns:The following paths:
(
    hosts_destination,
    plain_destination,
    json_destination,
    splited_destination
)
Return type:tuple
_Generate___info_files_authorization()

Provide the authorization for the generation of info files.

Basicaly here is what we check:

  • We are not testing as an imported module.

and

  • The hosts file generation is activated.

or

  • The plain list generation is activated.

or

  • The “api_file_generation” was set into the CONFIGURATION.
_analytic_host_file_directory()[source]

Return the analytic directory to write depending of the matched status.

_do_not_produce_file()[source]

Check if we are allowed to produce a file based from the given information.

Returns:The state of the production. True: We do not produce file. False: We do produce file.
Return type:bool
_prints_status_screen()[source]

Logic behind the printing (on screen) when generating status file.

analytic_file(new_status, old_status=None)[source]

Generate Analytic/* files based on the given old and new statuses.

Parameters:
  • new_status (str) – The new status of the domain.
  • old_status (str) – The old status of the domain.
complements_file()[source]

Generate complements files base on the current status.

info_files()[source]

Generate the hosts file, the plain list, the JSON file and the splitted files.

prints_status_file()[source]

Logic behind the printing (in file) when generating status file.

status_file(exclude_file_generation=False)[source]

Generate a file according to the domain status.

Parameters:exclude_file_generation (bool) – A shorthand to disable any file generation.
unified_file()[source]

Generate unified file. Understand by that that we use an unified table instead of a separate table for each status which could result into a misunderstanding.

Logs()

class PyFunceble.output.logs.Logs(output=None)[source]

Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:

Parameters:output (str) – A path to the JSON file we are going to write.
classmethod _get_content(file)[source]

Get and return the content of the given log file.

Parameters:file (str) – The file we have to get the content from.

:return The content of the given file. :rtype: dict

classmethod _write_content(content, file)[source]

Write the content into the given file.

Parameters:
  • content (str) – The dict to write.
  • file (str) – The file to write.
expiration_date(subject, extracted)[source]

Logs the extracted expiration date.

Parameters:
  • subject (str) – The currently tested subject.
  • extracted (str) – The extracted expiration date (from WHOIS record).
referer_not_found(subject, extension)[source]

Logs the case that the referer was not found.

Parameters:
  • subject (str) – The currently tested subject.
  • extension (str) – The extension of the domain we are testing.
whois(subject, record)[source]

Logs the WHOIS record if needed.

Parameters:
  • subject (str) – The currently tested subject.
  • record (str) – The record to log.

Percentage()

class PyFunceble.output.percentage.Percentage(domain_status=None, init=None)[source]

Calculation of the percentage of each status.

Parameters:
  • domain_status (str) – The status to increment.
  • init (dict) –

    The data from a previous session we are continuing.

    Warning

    We expect the numbers and not the percentages.

classmethod calculate()[source]

Calculate the percentage of each status.

count()[source]

Count the number of domain for each status.

log()[source]

Print on screen and on file the percentages for each status.

Prints()

class PyFunceble.output.prints.Prints(to_print, template, output_file=None, only_on_file=False)[source]

Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.

Parameters:
  • to_print (list) – The list of data to print.
  • template (str) –

    The template to use.

    Note

    Available templates:

    • Percentage
    • Less
    • HTTP
    • any of the official status.
  • output_file (str) – The path to the file to write.
  • only_on_file (bool) – Tell us if we only have to print on file and not on screen.
_Prints__get_print_size()

Provides the size of the element to print.

_colorify(data)[source]

Retun colored string.

Parameters:data (str) – The string to colorify.
Returns:A colored string.
Return type:str
_json_print()[source]

Management of the json template.

classmethod _size_from_header(header)[source]

Get the size of each columns from the header.

Parameters:header (dict) – The header template we have to get the size from.
Returns:The maximal size of the each data to print.
Return type:list
before_header()[source]

Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.

data()[source]

Management and input of data to the table.

Raises:
Exception

When self.data_to_print is not a list.

data_constructor(size)[source]

Construct the table of data according to given size.

Parameters:

size (list) – The maximal length of each string in the table.

Returns:

A dict with all information about the data and how to which what maximal size to print it.

Return type:

OrderedDict

Raises:
Exception

If the data and the size does not have the same length.

header(do_not_print=False)[source]

Management and creation of templates of header. Please consider as “header” the title of each columns.

Parameters:do_not_print (bool) – Tell us if we have to print the header or not.
classmethod header_constructor(data_to_print, header_separator='-', column_separator=' ')[source]

Construct header of the table according to template.

Parameters:
  • data_to_print (list) – The list of data to print into the header of the table.
  • header_separator (str) – The separator to use between the table header and our data.
  • colomn_separator (str) – The separator to use between each colomns.
Returns:

The data to print in a list format.

Return type:

list