Code Documentation¶
Note
This section will document every part (except the API section) of our code base.
Helpers¶
Problematic¶
How can we avoid writing the same thing every time?
Documentation¶
The tool to check the availability or syntax of domains, IPv4 or URL.
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
This submodule will provide the helpers.
- Author:
- Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
- Special thanks:
- https://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Contributors:
- http://pyfunceble.readthedocs.io/en/master/special-thanks.html
- Project link:
- https://github.com/funilrys/PyFunceble
- Project documentation:
- https://pyfunceble.readthedocs.io/en/master/
- Project homepage:
- https://funilrys.github.io/PyFunceble/
License:
MIT License
Copyright (c) 2017-2018 Nissar Chababy
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-
class
PyFunceble.helpers.
Command
(command)[source]¶ Shell command execution.
Parameters: command (str) – The command to execute
-
class
PyFunceble.helpers.
Dict
(main_dictionnary=None)[source]¶ Dictionary manipulations.
Parameters: main_dictionnary (dict) – The dict we are working with. -
classmethod
from_json
(data)[source]¶ Convert a JSON formatted string into a dictionary.
Parameters: data (str) – A JSON formatted string to convert to dict format. Returns: The dict representation of the JSON formatted string. Return type: dict
-
classmethod
from_yaml
(data)[source]¶ Convert a YAML formatted string into a dictionary.
Parameters: data (str) – A YAML formatted string to convert to dict format. Returns: The dict representation of the YAML formatted string. Return type: dict
-
merge
(to_merge, strict=True)[source]¶ Merge the content of to_merge into the given main dictionnary.
Parameters: - to_merge (dict) – The dictionnary to merge.
- strict (bool) –
Tell us if we have to strictly merge lists.
True
: We follow index :code`False`: We follow element (content)
Returns: The merged dict.
Return type: dict
-
remove_key
(key_to_remove)[source]¶ Remove a given key from a given dictionary.
Parameters: key_to_remove (list|str) – The key(s) to delete. Returns: The dict without the given key(s). Return type: dict|None
-
rename_key
(key_to_rename, strict=True)[source]¶ Rename the given keys from the given dictionary.
Parameters: - key_to_rename (dict) – The key(s) to rename.
Expected format:
{old:new}
- strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns: The well formatted dict.
Return type: dict|None
- key_to_rename (dict) – The key(s) to rename.
Expected format:
-
classmethod
-
class
PyFunceble.helpers.
Directory
(directory)[source]¶ Directory manipulation.
Parameters: directory (str) – A path to the directory to manipulate.
-
class
PyFunceble.helpers.
Download
(link, destination=None, return_data=False, verify_certificate=True)[source]¶ Download or return the content of the given link.
Parameters: - link (str) – The link to download.
- destination (str) – The location where we should save the downloaded content.
- return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
- verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
-
class
PyFunceble.helpers.
File
(file)[source]¶ File treatment/manipulations.
Parameters: file (str) – A path to the file to manipulate. -
copy
(destination)[source]¶ Copy the given file to the destination.
Parameters: destination (str) – The destination of the copy.
-
-
class
PyFunceble.helpers.
Hash
(file_path=None, algorithm='sha512', only_hash=False, data=None)[source]¶ Get and return the hash a file with the given algorithm.
Parameters: - file_path (str) – The path to the file we have to hash.
- algorithm (str) – The algorithm to use.
- only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.
Note
The original version can be found at https://git.io/vFQrK.
-
class
PyFunceble.helpers.
List
(main_list=None)[source]¶ List manipulation.
Parameters: main_list (list) – The list to manipulate. -
custom_format
(key_method, reverse=False)[source]¶ Return a well formatted list. With the key_method as a function/method to format the elements before sorting.
Parameters: - key_method (function|method) – A function or method to use to format the readed element before sorting.
- reverse (bool) – Tell us if we have to reverse the list.
Returns: A sorted list.
Return type: list
-
-
class
PyFunceble.helpers.
Regex
(data, regex, **args)[source]¶ A simple implementation ot the python.re package
Parameters: - data (str) – The data to check.
- regex (str) – The regex to match.
- group (int) – The group to return.
- rematch (bool) –
Allow to return the matched groups into a formatted list.
Note
This is an implementation of Bash
${BASH_REMATCH}
- replace_with (str) – The value to replace the matched regex with.
- occurences (int) – The number of occurence(s) to replace.
- return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
Download()
¶
-
class
PyFunceble.helpers.
Download
(link, destination=None, return_data=False, verify_certificate=True)[source] Download or return the content of the given link.
Parameters: - link (str) – The link to download.
- destination (str) – The location where we should save the downloaded content.
- return_data (bool) – Tell us if we need to return the page content or write its content into the given destination.
- verify_certificate (bool) – Tell us if we need to verify the SSL/TLS certificate.
-
text
()[source] Download the given link and return or save its
requests.text
at the given destination.Return type: mixed
Raises: Exception
If the status code is not
200
.
Command()
¶
-
class
PyFunceble.helpers.
Command
(command)[source] Shell command execution.
Parameters: command (str) – The command to execute -
_decode_output
(to_decode)[source] Decode the output of a shell command in order to be readable.
Parameters: to_decode – Output of a command to decode. Type: bytes Returns: The decoded output. Return type: str
-
execute
()[source] Execute the given command.
Returns: The output of the command. Return type: str
-
Regex()
¶
-
class
PyFunceble.helpers.
Regex
(data, regex, **args)[source] A simple implementation ot the python.re package
Parameters: - data (str) – The data to check.
- regex (str) – The regex to match.
- group (int) – The group to return.
- rematch (bool) –
Allow to return the matched groups into a formatted list.
Note
This is an implementation of Bash
${BASH_REMATCH}
- replace_with (str) – The value to replace the matched regex with.
- occurences (int) – The number of occurence(s) to replace.
- return_type – Tell us if we have to return the matched data or simply check if we matched (True) or not (False)
-
match
()[source] Used to get exploitable result of re.search
Returns: The data of the match status. Return type: mixed
-
matching_list
()[source] Return a list of the string which match the given regex.
-
not_matching_list
()[source] Return a list of string which don’t match the given regex.
-
replace
()[source] Used to replace a matched string with another.
Returns: The data after replacement. Return type: str
Dict()
¶
-
class
PyFunceble.helpers.
Dict
(main_dictionnary=None)[source] Dictionary manipulations.
Parameters: main_dictionnary (dict) – The dict we are working with. -
classmethod
from_json
(data)[source] Convert a JSON formatted string into a dictionary.
Parameters: data (str) – A JSON formatted string to convert to dict format. Returns: The dict representation of the JSON formatted string. Return type: dict
-
classmethod
from_yaml
(data)[source] Convert a YAML formatted string into a dictionary.
Parameters: data (str) – A YAML formatted string to convert to dict format. Returns: The dict representation of the YAML formatted string. Return type: dict
-
merge
(to_merge, strict=True)[source] Merge the content of to_merge into the given main dictionnary.
Parameters: - to_merge (dict) – The dictionnary to merge.
- strict (bool) –
Tell us if we have to strictly merge lists.
True
: We follow index :code`False`: We follow element (content)
Returns: The merged dict.
Return type: dict
-
remove_key
(key_to_remove)[source] Remove a given key from a given dictionary.
Parameters: key_to_remove (list|str) – The key(s) to delete. Returns: The dict without the given key(s). Return type: dict|None
-
rename_key
(key_to_rename, strict=True)[source] Rename the given keys from the given dictionary.
Parameters: - key_to_rename (dict) – The key(s) to rename.
Expected format:
{old:new}
- strict – Tell us if we have to rename the exact index or the index which looks like the given key(s)
Returns: The well formatted dict.
Return type: dict|None
- key_to_rename (dict) – The key(s) to rename.
Expected format:
-
to_json
(destination)[source] Save a dictionnary into a JSON file.
Parameters: destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
-
to_yaml
(destination, flow_style=False)[source] Save a dictionnary into a YAML file.
Parameters: destination (str) – A path to a file where we’re going to write the converted dict into a JSON format.
-
classmethod
List()
¶
-
class
PyFunceble.helpers.
List
(main_list=None)[source] List manipulation.
Parameters: main_list (list) – The list to manipulate. -
custom_format
(key_method, reverse=False)[source] Return a well formatted list. With the key_method as a function/method to format the elements before sorting.
Parameters: - key_method (function|method) – A function or method to use to format the readed element before sorting.
- reverse (bool) – Tell us if we have to reverse the list.
Returns: A sorted list.
Return type: list
-
format
()[source] Return a well formatted list. Basicaly, it’s sort a list and remove duplicate.
Returns: A sorted, without duplicate, list. Return type: list
-
merge
(to_merge, strict=True)[source] Merge to_merge into the given main list.
Parameters: - to_merge (list) – The list to merge.
- strict (bool) – Tell us if we have to respect index (True) or not (False).
Returns: The merged list.
Return type: list
-
Directory()
¶
-
class
PyFunceble.helpers.
Directory
(directory)[source] Directory manipulation.
Parameters: directory (str) – A path to the directory to manipulate. -
fix_path
(splited_path=None)[source] Fix the path of the given path.
Parameters: splited_path (list) – A list to convert to the right path. Returns: The fixed path. Return type: str
-
File()
¶
-
class
PyFunceble.helpers.
File
(file)[source] File treatment/manipulations.
Parameters: file (str) – A path to the file to manipulate. -
copy
(destination)[source] Copy the given file to the destination.
Parameters: destination (str) – The destination of the copy.
-
delete
()[source] Delete a given file path.
-
read
()[source] Read a given file path and return its content.
Returns: The content of the given file path. Return type: str
-
write
(data_to_write, overwrite=False)[source] Write or append data into the given file path.
Parameters: - data_to_write (str) – The data to write.
- overwrite (bool) – Tell us if we have to overwrite the content of the file we are working with.
-
Hash()
¶
-
class
PyFunceble.helpers.
Hash
(file_path=None, algorithm='sha512', only_hash=False, data=None)[source] Get and return the hash a file with the given algorithm.
Parameters: - file_path (str) – The path to the file we have to hash.
- algorithm (str) – The algorithm to use.
- only_hash – Tell us if we only have to return the desired algorithm instead of the dummy dict format.
Note
The original version can be found at https://git.io/vFQrK.
-
_hash_data
(algo)[source] Get hash of the given data.
Parameters: algo (str) – The algorithm to use.
-
_hash_file
(algo)[source] Get the hash of the given file
Parameters: algo (str) – The algorithm to use. Returns: The hexdigest of the data. Return type: str
-
get
()[source] Return the hash of the given file
AdBlock¶
Problematic¶
How can we efficiently decode AdBlock filter list?
Documentation¶
-
class
PyFunceble.adblock.
AdBlock
(list_from_file, aggressive=False)[source]¶ Provide the adblock decoding logic.
Parameters: list_from_file (list) – The file in list format. -
classmethod
_extract_base
(element)[source]¶ Extract the base of the given element.
Parameters: element (str|list) – The element we are working with.
-
classmethod
_format_decoded
(to_format, result=None)[source]¶ Format the exctracted adblock line before passing it to the system.
Parameters: - to_format (str) – The extracted line from the file.
- result (list) – A list of the result of this method.
Returns: The list of domains or IP to test.
Return type: list
-
_handle_options
(options)[source]¶ Handle the data from the options.
Parameters: options (list) – The list of options from the rule. Returns: The list of domains to return globally. Return type: list
-
classmethod
Auto-save¶
Note
Only Travis CI is actually supported.
Travis CI problematic¶
How can we bypass the default Travis CI timeout of 45 minutes?
Check¶
Problematic¶
How can we efficiently check the format of IP, domains, and URL?
Documentation¶
-
class
PyFunceble.check.
Check
(element=None)[source]¶ Provide a place to check several things around URL, IP or domain.
Parameters: element (str) – The element (URL, IP or domain) to check. -
is_domain_valid
(domain=None, subdomain_check=False)[source]¶ Check if the given domain is a valid.
Parameters: - domain (str) – The domain to validate.
- subdomain_check (bool) – Activate the subdomain checking.
Returns: The validity of the sub-domain.
Return type: bool
-
is_ip_range
(ip_to_check=None)[source]¶ Check if the given IP is a valid IPv4.
Parameters: ip_to_check (str) – The IP to test. Returns: The validity of the IP. Return type: bool Note
We only test IPv4 because for now we only them for now.
-
is_ip_valid
(ip_to_check=None)[source]¶ Check if the given IP is a valid IPv4.
Parameters: ip_to_check (str) – The IP to test. Returns: The validity of the IP. Return type: bool Note
We only test IPv4 because for now we only them for now.
-
is_subdomain
(domain=None)[source]¶ Check if the given subdomain is a subdomain.
Parameters: domain (str) – The domain to validate. Returns: The validity of the subdomain. Return type: bool
-
is_url_valid
(url=None, return_base=False, return_formatted=False)[source]¶ Check if the given URL is valid.
Parameters: - url (str) – The url to validate.
- return_base – Allow us the return of the url base (if URL formatted correctly).
- return_formatted (bool) – Allow us to get the URL converted to IDNA if the conversion is activated.
Returns: The validity of the URL or its base.
Return type: bool|str
-
Cleaning¶
Problematic¶
How can we clean the output/
directory so we do not have a collision between old and new files?
Documentation¶
-
class
PyFunceble.clean.
Clean
(list_to_test, clean_all=False)[source]¶ Provide the cleaning logic(s).
Parameters: - list_to_test (list|None) – The list of domains we are testing.
- clean_all (bool) – Tell the subsystem if we need to clean all. Which include, of course, the output directory but also all other file(s) generated by our system.
Configuration¶
Problematics¶
- How can we avoid the usage of
tool.py
? - How can we make personalization more simple?
Documentation¶
Load()
¶
-
class
PyFunceble.config.
Load
(path_to_config)[source]¶ Help us load the configuration(s) file(s).
Parameters: path_to_config (str) – The possible path to the configuration to load. -
classmethod
_install_directory_structure_file
()[source]¶ Download the latest version of dir_structure_production.json.
-
classmethod
Version()
¶
-
class
PyFunceble.config.
Version
(used=False)[source]¶ Compare the local with the upstream version.
Parameters: used (bool) – True: Version is configured for simple usage. False: Version compare local with upstream. -
classmethod
check_versions
(local, upstream)[source]¶ Compare the given versions.
Parameters: - local (list) – The local version converted by split_versions().
- upstream (list) – The upstream version converted by split_versions().
Returns: - True: local < upstream
- None: local == upstream
- False: local > upstream
Return type: bool|None
-
classmethod
check_versions_literally
(local, upstream)[source]¶ Compare the given versions literally.
Parameters: - local (str) – The local version converted by split_versions().
- upstream (str) – The upstream version converted by split_versions().
Returns: - True: local == upstream
- False: local != upstream
Return type: bool
-
classmethod
is_cloned
()[source]¶ Let us know if we are currently in the cloned version of PyFunceble which implicitly mean that we are in developement mode.
-
classmethod
right_url_from_version
(url)[source]¶ Convert the GitHub URL to the right one depending of the branch or version we are working with.
Parameters: url (str) – The URL to convert. Returns: The converted URL. Return type: str
-
classmethod
split_versions
(version, return_non_digits=False)[source]¶ Convert the versions to a shorter one.
Parameters: - version (str) – The version to split.
- return_non_digits (bool) – Activate the return of the non-digits parts of the splitted version.
Returns: The splitted version name/numbers.
Return type: list
-
classmethod
Core¶
Documentation¶
-
class
PyFunceble.core.
Core
(**args)[source]¶ Main entry to PyFunceble. Brain of the program. Also known as “put everything together to make the system works”.
Parameters: - domain_or_ip_to_test (str) – A domain or IP to test.
- file_path (str) – A path to a file to read and test.
- url_to_test (str) – A URL to test.
- url_file (str) – A path to a file which contains URL to test.
- link_to_test (str) – A link to a file to download and test.
- modulo_test – If set to True, it will tell the system that we are working as an exported module.
- modulo_test – bool
-
classmethod
_entry_management_url_download
(passed)[source]¶ Check if the given information is a URL. If it is the case, it download and update the location of file to test.
Parameters: passed (str) – The url passed to the system. Returns: The state of the check. Return type: bool
-
classmethod
_extract_domain_from_file
()[source]¶ Extract all non commented lines from the file we are testing.
Returns: The elements to test. Return type: list
-
_file_decision
(current, last, status=None)[source]¶ Manage the database, autosave and autocontinue systems for the case that we are reading a file.
Parameters: - current (str) – The currently tested element.
- last (str) – The last element of the list.
- status (str) – The status of the currently tested element.
-
classmethod
_format_domain
(extracted_domain)[source]¶ Format the extracted domain before passing it to the system.
Parameters: extracted_domain (str) – The extracted domain. Returns: The formatted domain or IP to test. Return type: str
-
classmethod
bypass
()[source]¶ Exit the script if
[PyFunceble skip]
is matched into the latest commit message.
-
classmethod
colorify_logo
(home=False)[source]¶ Print the colored logo based on global results.
Parameters: home (bool) – Tell us if we have to print the initial coloration.
-
domain
(domain=None, last_domain=None)[source]¶ Manage the case that we want to test only a domain.
Parameters: - domain (str) – The domain or IP to test.
- last_domain (str) – The last domain to test if we are testing a file.
- return_status (bool) – Tell us if we need to return the status.
-
file
()[source]¶ Manage the case that need to test each domain of a given file path.
Note
1 domain per line.
-
classmethod
switch
(variable, custom=False)[source]¶ Switch PyFunceble.CONFIGURATION variables to their opposite.
Parameters: - variable (str|bool) – The variable name to switch. The variable should be an index our configuration system. If we want to switch a bool variable, we should parse it here.
- custom (bool) – Let us know if have to switch the parsed variable instead of our configuration index.
Returns: The opposite of the configuration index or the given variable.
Return type: bool
Raises: Exception
When the configuration is not valid. In other words, if the PyFunceble.CONFIGURATION[variable_name] is not a bool.
-
test
(complete=False)[source]¶ Avoid confusion between self.domain which is called into __main__ and test() which should be called out of PyFunceble’s scope.
Parameters: complete (bool) – Activate the return of a dictionnary with signigican data about the test.
Returns: ACTIVE INACTIVE or INVALID.
Return type: str|list
Raises: Exception
When this method is called under
__name__ == '__main__'
Note
This method should never be called in a
__name__ == '__main__'
context.
Database¶
Problematics¶
- How can we continuously test
INACTIVE
andINVALID
domains or IP? - How can we reduce the number of whois requests over time?
Documentation¶
-
class
PyFunceble.database.
Inactive
[source]¶ Logic behind the generation and the usage of a database system. The main idea behind this is to provide an inactive-db.json and test all inactive domain which are into to it regularly
-
_add_to_test
(to_add)[source]¶ Add an element or a list of element into
PyFunceble.CONFIGURATION['inactive_db'][self.file_path]['to_test']
.Parameters: to_add (str|list) – The domain, IP or URL to add.
-
_merge
()[source]¶ Merge the real database with the older one which has already been set into
PyFunceble.CONFIGURATION["inactive_db"]
-
_reformat_historical_formating_error
()[source]¶ Format the old format so it can be merged into the newer format.
-
_timestamp
()[source]¶ Get the timestamp where we are going to save our current list.
Returns: The timestamp to append with the currently tested element. Return type: int|str
-
add
()[source]¶ Save the current :code.`PyFunceble.CONFIGURATION[‘to_test’]` into the current timestamp.
-
-
class
PyFunceble.database.
Whois
(expiration_date=None)[source]¶ Logic behind the whois database. Indeed, the idea is to implement #2.
Parameters: expiration_date (str) – The extracted expiration date. Check if we are authorized to work with our database.
Directory Structure¶
Problematic¶
How can we give make the output directory less annoying after an update?
Documentation¶
-
class
PyFunceble.directory_structure.
DirectoryStructure
(production=False)[source]¶ Basically a backup/reconstructor of our
output
directory.Parameters: production (bool) – Tell the subsystem if we are preparing for production which imply the execution of the backup insteam of of the “reconstructore” mode. -
classmethod
_create_directory
(directory, loop=False)[source]¶ Creates the given directory if it does not exists.
Parameters: - directory (str) – The directory to create.
- loop (bool) – Tell us if we are in the creation loop or not.
-
_get_structure
()[source]¶ Get the structure we are going to work with.
Returns: The structure we have to work with. Return type: dict
-
_restore_replace
()[source]¶ Check if we need to replace “.gitignore” to “.keep”.
Returns: The replacement status. Return type: bool
-
_update_structure_from_config
(structure)[source]¶ Update the paths according to configs.
Parameters: structure (dict) – The read structure.
-
classmethod
Execution Time¶
Problematic¶
How to monitor the execution time of the session?
Documentation¶
-
class
PyFunceble.execution_time.
ExecutionTime
(action='start', last=False)[source]¶ Set and return the exection time of the program.
Parameters: - action (str) – The action related the execution time. Can be start or stop.
- last (bool) – Tell the subsystem if we are at the very end of the file testing.
Check the execution authorization.
Returns: The authorization status. Return type: bool
-
classmethod
_calculate
(start=None, end=None)[source]¶ calculate the difference between starting and ending time.
Parameters: - start (int|str) – A starting time.
- stop (int|str) – A ending time.
Returns: A dict with following as index.
days
hours
minutes
seconds
as index.
Return type: dict
Expiration Date¶
Problematic¶
How can we get the expiration date of a given domain?
Documentation¶
-
class
PyFunceble.expiration_date.
ExpirationDate
[source]¶ Get, format and return the expiration date of a domain, if exist.
-
_cases_management
(regex_number, matched_result)[source]¶ A little internal helper of self.format. (Avoiding of nested loops)
Note
Please note that the second value of the case represent the groups in order
[day,month,year]
.This means that a
[2,1,0]
will be for example for a date in format2017-01-02
where01
is the month.Parameters: - regex_number (int) – The identifiant of the regex.
- matched_result (list) – The matched result to format.
Returns: A list representing the expiration date. The list can be “decoded” like
[day, month, year]
Return type: list|None
-
classmethod
_convert_1_to_2_digits
(number)[source]¶ Convert 1 digit number to two digits.
Parameters: number (str|int) – A number or a digit string. Returns: A 2 or more digit string. Return type: str
-
classmethod
_convert_or_shorten_month
(data)[source]¶ Convert a given month into our unified format.
Parameters: data (str) – The month to convert or shorten. Returns: The unified month name. Return type: str
-
_extract
()[source]¶ Extract the expiration date from the whois record.
Returns: The status of the domain. Return type: str
-
Generation¶
Problematic¶
How can we generate files which reflects the results of PyFunceble?
Documentation¶
-
class
PyFunceble.generate.
Generate
(domain_status, source=None, expiration_date=None)[source]¶ Generate different sort of files.
Parameters: - domain_status (str) – The domain status.
- source (str) – The source of the given status.
- expiration_date (str) – The expiration date of the domain (if catched).
-
_analytic_host_file_directory
()[source]¶ Return the analytic directory to write depending of the matched status.
-
classmethod
_handle_non_existant_index
()[source]¶ Handle and check that some configuration index exists.
-
analytic_file
(new_status, old_status)[source]¶ Generate
Analytic/*
files based on the given old and new statuses.Parameters: - new_status (str) – The new status of the domain.
- old_status (str) – The old status of the domain.
HTTP Code¶
Problematic¶
How can we get the HTTP status code of the given domain or IP?
IANA¶
Problematic¶
How can we get information from IANA?
Documentation¶
-
class
PyFunceble.iana.
IANA
[source]¶ Logic behind the update of iana-domains-db.json
-
_extensions
(block)[source]¶ Extract the extention from the given block. Plus get its referer.
Parameters: block (str) – The line from the IANA database.
-
Logs¶
Problematic¶
How can we efficiently generate and share logs?
Documentation¶
-
class
PyFunceble.logs.
Logs
(output=None)[source]¶ Provide a clean and unique way to work with logs. Indeed, it’s not good to have logs spread around the code :smile:
Parameters: output (str) – A path to the JSON file we are going to write. -
classmethod
_get_content
(file)[source]¶ Get and return the content of the given log file.
Parameters: file (str) – The file we have to get the content from. :return The content of the given file. :rtype: dict
-
classmethod
_write_content
(content, file)[source]¶ Write the content into the given file.
Parameters: - content (dict) – The dict to write.
- file (str) – The file to write.
-
expiration_date
(extracted)[source]¶ Logs the extracted expiration date.
Parameters: extracted (str) – The extracted expiration date (from WHOIS record).
-
classmethod
Lookup¶
Problematics¶
- How can we get information from WHOIS records?
- How can we check if a domain or IP have a valid pointer (nslookup)?
Documentation¶
-
class
PyFunceble.lookup.
Lookup
[source]¶ Can be used to NSLOOKUP or WHOIS lookup.
-
classmethod
whois
(whois_server, domain=None, timeout=None)[source]¶ Implementation of UNIX whois.
Parameters: - whois_server (str) – The WHOIS server to use to get the record.
- domain (str) – The domain to get the whois record from.
- timeout (int) – The timeout to apply to the request.
Returns: The whois record from the given whois server, if exist.
Return type: str|None
-
classmethod
Mining¶
Problematic¶
How can we get the list of domain or URL which link to the desired domain, IPv4 or URL?
Documentation¶
-
class
PyFunceble.mining.
Mining
[source]¶ Manage the minig subsystem.
-
_add
(to_add)[source]¶ Add the currently mined information to the mined “database”.
Parameters: to_add (dict) – The element to add.
-
list_of_mined
()[source]¶ Provide the list of mined so they can be added to the list queue.
Returns: The list of mined domains or URL. Return type: list
-
Percentage¶
Problematic¶
How can we calculate the percentage of each status?
Prints¶
Problematic¶
How can we print information on the screen and on file in a table format?
Documentation¶
-
class
PyFunceble.prints.
Prints
(to_print, template, output_file=None, only_on_file=False)[source]¶ Print data on screen and into a file if needed. Template Possibilities: Percentage, Less, HTTP and any status you want.
Parameters: - to_print – The list of data to print.
- template (str) –
The template to use.
Note
Available templates:
Percentage
Less
HTTP
- any of the official status.
- output_file (str) – The path to the file to write.
- only_on_file (bool) – Tell us if we only have to print on file and not on screen.
-
_before_header
()[source]¶ Print informations about PyFunceble and the date of generation of a file into a given path, if doesn’t exist.
-
_colorify
(data)[source]¶ Retun colored string.
Parameters: data (str) – The string to colorify. Returns: A colored string. Return type: str
-
_data_constructor
(size)[source]¶ Construct the table of data according to given size.
Parameters: size (list) – The maximal length of each string in the table.
Returns: A dict with all information about the data and how to which what maximal size to print it.
Return type: OrderedDict
Raises: Exception
If the data and the size does not have the same length.
-
classmethod
_header_constructor
(data_to_print, header_separator='-', column_separator=' ')[source]¶ Construct header of the table according to template.
Parameters: - data_to_print (list) – The list of data to print into the header of the table.
- header_separator (str) – The separator to use between the table header and our data.
- colomn_separator (str) – The separator to use between each colomns.
Returns: The data to print in a list format.
Return type: list
-
classmethod
_size_from_header
(header)[source]¶ Get the size of each columns from the header.
Parameters: header (dict) – The header template we have to get the size from. Returns: The maximal size of the each data to print. Return type: list
Production¶
Problematic¶
How can we efficiently prepare the repository for push/production?
Public Suffix¶
Problematic¶
How can we get the list of all possible or at least most used domain suffix?
Referer¶
Problematic¶
How can we efficiently get the whois server to call for whois record request?
Documentation¶
Sort¶
Problematic¶
How can we format the list to test (and the outputted information) in a format other than the alphabetical format?
Documentation¶
-
class
PyFunceble.sort.
Sort
[source]¶ Provide some sorting presets which we can parse to PyFunceble.helpers.List().custom_format().
-
classmethod
hierarchical
(element)[source]¶ The idea behind this method is to sort a list of domain hierarchicaly.
Parameters: element (str) – The element we are currently reading. Returns: The formatted element. Return type: str Note
For a domain like
aaa.bbb.ccc.tdl
.- A normal sorting is done in the following order:
aaa
bbb
ccc
tdl
- This method allow the sorting to be done in the following order:
tdl
ccc
bbb
aaa
-
classmethod
Status¶
Problematic¶
How can we efficiently manage the statuses in function of the test type?