PyFunceble.cli.processes.workers package

Submodules

PyFunceble.cli.processes.workers.base module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides the base of all our workers.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.base.WorkerBase(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: multiprocessing.context.Process

Provides the base of all our workers.

Parameters:
  • input_queue – The input queue to read.
  • output_queue – The output queue to write.
BREAKOFF = 2.0
MINING_WAIT_TIME = 60
STD_NAME = 'pyfunceble_base_worker'
accept_waiting_delay = None
add_to_input_queue(data: Any, *, worker_name: Optional[str] = None, destination_worker: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]

Adds the given data to the current queue.

Parameters:
  • data – The data to add into the queue.
  • destination_worker – The name of the worker which is supposed to read the message.
add_to_output_queue(data: Any, *, worker_name: Optional[str] = None, destination_worker: Optional[str] = None) → PyFunceble.cli.processes.workers.base.WorkerBase[source]

Adds the given data to the output queue queue.

Parameters:data – The data to add into the queue.
concurrent_worker_names = None
continuous_integration = None
db_session = None
exception

Provides the exception of the current worker.

exit_it = None
global_exit_event = None
input_queue = None
output_queue = None
run() → None[source]

Method to be run in sub-process; can be overridden in sub-class

send_stop_message = None
share_waiting_message(*, overall: bool = False, ignore_current_worker: bool = False, apply_breakoff: bool = False) → PyFunceble.cli.processes.workers.base.WorkerBase[source]

Shares a waiting message to both input and output queues. The idea is to keep our worker awake when necessary. This functionality is really important when running the mining worker because sometime the flow is so long, that we may need some time.

Parameters:
  • overall – Shares the message multiple time to specific destination in our current stack.
  • ignore_current_worker – Doesn’t share the message with ourself.
  • apply_breakoff – Activates the breakoff (sleep) before starting to share the message.
target(consumed: Any) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

terminate() → None[source]

Terminate our worker.

PyFunceble.cli.processes.workers.chancy_producer module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our producer worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.chancy_producer.ChancyProducerWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.producer.ProducerWorker

Provides our chancy producer worker.

Warning

This chancy producer does not provide any guarantee. The flow that keep PyFunceble safe are here unleashed.

USE AT YOUR OWN RISK. GOOD LUCK!

STD_NAME = 'pyfunceble_chancy_producer_worker'
target(consumed: Any) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

PyFunceble.cli.processes.workers.chancy_tester module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our chancy tester worker. The chancy tester worker is a worker that abstract from the standard tester worker. It just get rid of the walls between some of the component of our data workflow.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.chancy_tester.ChancyTesterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.tester.TesterWorker

Provides our chancy tester worker. The chancy worker breaks the walls between some of the core component of our data workflow.

Warning

This chancy tester does not provide any guarantee. The flow that keep PyFunceble safe are here unleashed.

USE AT YOUR OWN RISK. GOOD LUCK!

STD_NAME = 'pyfunceble_chancy_tester_worker'
target(consumed: dict) → Optional[Tuple[Any, ...]][source]

The actually wall destructor.

Parameters:consummed – The data that needs to be tested.

PyFunceble.cli.processes.workers.dir_files_sorter module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our directory files sorter worker. This is the description of a single directory file sorter worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.dir_files_sorter.DireFileSorterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.file_sorter_base.FileSorterWorkerBase

Provides our directory files sorter worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the sorting of the content of a submitted directory.

Expected (input) message:

{
    "directory": str,
    "remove_duplicates": bool,
    "write_header": bool
}

Expected (output) message:

None
STD_NAME = 'pyfunceble_dir_files_sorter_worker'
static get_files_to_sort(directory: str) → List[str][source]

Provides the list of files to sort.

Parameters:directory – The directory to start from.
target(consumed: Any) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

PyFunceble.cli.processes.workers.file_sorter module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our file sorter worker. This is the description of a file sorter worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.file_sorter.FileSorterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.file_sorter_base.FileSorterWorkerBase

Provides our file sorter worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the sorting of the content of a given file.

Expected (input) message:

{
    "file": str,
    "remove_duplicates": bool,
    "write_header": bool
}

Expected (output) message:

None
STD_NAME = 'pyfunceble_file_sorter_worker'
target(consumed: Any) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

PyFunceble.cli.processes.workers.file_sorter_base module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our file sorter worker base. This is the base of all our file sorter.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.file_sorter_base.FileSorterWorkerBase(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our the base of all our file sorters.

FILE_BUFFER_SIZE = 65536
MAX_LINES = 32000
classmethod process_file_sorting(file: str, remove_duplicates: bool = True, write_header: bool = True, sorting_key: Any = None) → None[source]

Process the sorting of the given file.

The idea is to split the file piece by piece and at the end join all sorted files. For that job, we create a temporary directory which will store the temporary files.

Parameters:
  • file – The file to sort.
  • remove_duplicates – Activates the deletion of duplicates.
  • write_header

    Activates the writing of the PyFunceble related header.

    Warning

    When this is set to True, we assume that the header itself was already given. Meaning that the first 2 commented lines will be excluded from the sorting and regenerated.

  • sorting_key

    The sorting key to apply while sorting.

    This is the lambda/function that goes into the key argument of the sorted function.

PyFunceble.cli.processes.workers.migrator module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our migrator (base) worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.migrator.MigratorWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our migrator (base) worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.

STD_NAME = 'pyfunceble_migrator_worker'
run() → None[source]

Method to be run in sub-process; can be overridden in sub-class

PyFunceble.cli.processes.workers.miner module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our miner worker. This is the description of a single miner worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.miner.MinerWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our miner worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the mining of dataset to test.

INACTIVE_STATUSES = ('INACTIVE', 'INVALID')
STD_NAME = 'pyfunceble_miner_worker'
static mine_from(subject: str) → Optional[List[str]][source]

Given the subject to work from, try to get the related subjects.

Parameters:subject – The URL to start from.
target(consumed: Tuple[dict, PyFunceble.checker.status_base.CheckerStatusBase]) → None[source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

url2netloc = None

PyFunceble.cli.processes.workers.producer module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our producer worker. This is the description of a single producer worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.producer.ProducerWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our producer worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the production of output to stdout or files.

INACTIVE_STATUSES = ('INACTIVE', 'INVALID')
STD_NAME = 'pyfunceble_producer_worker'
collection_query_tool = None
continue_dataset = None
counter = None
file_printer = None
header_already_printed = None
inactive_dataset = None
registrar_counter = None
run_continue_backup(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the backup or update of the auto-continue dataset storage.

run_counter(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the counter of the current file.

run_ignored_file_printer(test_dataset: dict, test_result: str) → None[source]

Runs the analytic behind the file printer.

Warning

Thie method assume that the givne dataset is ignored from the normal file printer.

run_inactive_backup(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the backup or update of the Inactive dataset storage.

The idea is that if the status is OK (active), we just remove it from the dataset storage. Otherwise, we just keep it in there :-)

run_status_file_printer(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the status file printer.

run_stdout_printer(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the stdout printer (if necessary).

Parameters:test_result – The rest result dataset.
run_whois_backup(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → None[source]

Runs the backup or update of the WHOIS record in our dataset storage.

Parameters:test_result – The test result.
should_we_block_status_file_printer(test_dataset: dict, test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]

Checks if we should block the file printer.

The reason behindn this is that we don’t want to generate an output when a subject was already into the inactive database.

static should_we_ignore(test_result: PyFunceble.checker.status_base.CheckerStatusBase) → bool[source]

Checks if we should ignore the given datasets.

Parameters:test_result – The test result to check.
static should_we_print_status_to_stdout(status: str) → bool[source]

Checks if we are allows to print the given status (to stdout).

Parameters:status – The status to check.
status_file_generator = None
stdout_printer = None
target(consumed: Any) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

whois_dataset = None

PyFunceble.cli.processes.workers.tester module

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides our tester worker. This is the description of a single tester worker.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
class PyFunceble.cli.processes.workers.tester.TesterWorker(input_queue: Optional[queue.Queue], output_queue: Optional[queue.Queue] = None, global_exit_event: Optional[multiprocessing.context.BaseContext.Event] = None, *, name: Optional[str] = None, daemon: Optional[bool] = None, continuous_integration: Optional[PyFunceble.cli.continuous_integration.base.ContinuousIntegrationBase] = None, configuration: Optional[dict] = None)[source]

Bases: PyFunceble.cli.processes.workers.base.WorkerBase

Provides our tester worker. The objective of this worker is to provides a single worker (or process if you prefer) which will be used to handle the tests.

STD_NAME = 'pyfunceble_tester_worker'
continue_dataset = None
inactive_dataset = None
initiated_testing_objects = {}
known_testing_objects = {}
static should_be_ignored(subject: str) → bool[source]

Checks if the given subject should be ignored.

target(consumed: dict) → Optional[Tuple[Any, ...]][source]

This the target that is run to process something. This method should return a result which will pu send to the output queue.

testing_object = None

Module contents

The tool to check the availability or syntax of domain, IP or URL.

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Provides all our standalone workers.

Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://pyfunceble.readthedocs.io/en/dev/
Project homepage:
https://pyfunceble.github.io/

License:

Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.