Update

Stable version

Using pip

From PyPi

$ pip3 install --user --upgrade PyFunceble

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

From GitHub

$ pip3 install --user --upgrade git+https://github.com/funilrys/PyFunceble.git@master#egg=PyFunceble

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

Using the AUR (for Arch Linux users)

With makepkg

$ curl https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble -o PKGBUILD
$ makepkg
$ sudo pacman -U python-pyfunceble*.tar.xz

With your favorite AUR helper

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -Syu python-pyfunceble

Using docker (hub)

The image description can be found at https://hub.docker.com/r/pyfunceble/pyfunceble

$ docker pull pyfunceble/pyfunceble

Pure Python method

Execute the following and enjoy PyFunceble!

$ cd PyFunceble
$ git checkout master && git fetch origin && git merge origin/master
$ python3 setup.py test
$ python3 setup.py install # Avoid this if you want to uninstall one day.
$ pip3 install --user --upgrade -e . # Prefer this method.

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

Development version

For development

$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev

Note

As you previously installed with

Only code/repository update is required.

For usage

Using pip

From PyPi
$ pip3 install --user --upgrade PyFunceble-dev

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

From GitHub
$ pip3 install --user --upgrade git+https://github.com/funilrys/PyFunceble.git@dev#egg=PyFunceble

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

Using the AUR (for Arch Linux users)

With makepkg
$ curl https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-pyfunceble-dev -o PKGBUILD
$ makepkg
$ sudo pacman -U python-pyfunceble-dev*.tar.xz
With your favorite AUR helper

Warning

We do not recommend any AUR helper but keep in mind that some AUR helpers are “better” than other. For more information about your current (or any other) AUR helper please report to the ArchWiki page.

$ yourFavoriteAurHelper -Syu python-pyfunceble-dev

Using docker (hub)

The image description can be found at https://hub.docker.com/r/pyfunceble/pyfunceble-dev

$ docker pull pyfunceble/pyfunceble-dev

Pure Python method

Execute the following and enjoy PyFunceble!

$ cd PyFunceble && git checkout dev
$ git fetch origin && git merge origin/dev
$ python3 setup.py test
$ python3 setup.py install # Avoid this if you want to uninstall one day.
$ pip3 install --user --upgrade -e . # Prefer this method.

Note

We recommend the --user flag which installs the required dependencies at the user level. More information about it can be found on pip documentation.

Warning

We do not recommend the --user flag when using PyFunceble into containers or CI engines.

Important information for pyfunceble >= 4.0.0

When you upgrade PyFunceble from <4.x to any version 4.x there will be:

  • a SQL conversion if you use the --database-type mysql or mariadb.

  • a JSON to CSV conversion if one of those files is found in your filesystem:

    • inactive_db.json
    • whois_db.json
  • The conversion of both SQL and json to CSV will take a “bit” of time as, it is done in a single process mode, to avoid any hick-ups instead of loading the entire file into memory. Loading the entire *.json file into memory can have severe consequences depending on the size of the source file.

    Note

    Once the job is done the json2csv shouldn’t appear again for this or later 4.x releases.

  • A workaround for waiting on the rather slow json2csv, you can delete

    • inactive_db.json
    • whois_db.json (Very Very Very bad idea in the long run…)

    However you will probably not benefit for by deleting the whois_db.json as this is the definitive slowest lookup process in the test flow, do to the limitation in available API call you can do to WHOIS servers before getting banned. Therefore we CAN NOT recommend deleting this file, rather than waiting for the conversion to finish.

  • The output directory structure have been altered to work with the ability to test more than one source at the time. Prior to version 4.0.0.ax the output hierarchy looked like output/domains/ACTIVE/list. In Pyfunceble version >=4.x this have been altered to include the source name and append to the folder structure. From this version it will therefor looks like output/{{ input_source_name }}/domains/ACTIVE list.

Note

As consequence of the time consuming conversion, we will advise you to run a simple pyfunceble command like:

pyfunceble -d mypdns.org

pyfunceble --database-type mariadb -d mypdns.org

How long time does it take

A few numbers to help you schedule your upgrade process.

We have tested the SQL conversion with the following specifications

  • 2 Xeon CPU x86_64 (8 cores) 2 GHz
  • 48 GB ram.
  • 1 SSD Kingston KC-600
  • Mariadb 10.5, default config
  • Non dedicated
The database contains
  • Roughly 265.000 records in the test tables
  • Approximately 1.000.000 records within the pyfunceble_whois table.

This process toke about 10 hours to complete.

Important information for >= 3.2.11

When you update from dev@<=3.2.10 or master@<=3.2.2 to newer release, there will be made a SQL conversion of the databases table layout. This can take up a sagnificent amount of time based on the size of the database.

The table layout converion is being made to:

  1. Minimize the total size
  2. Optimize the sql flow and minimizing the read/write to save disk I/O.
  3. Minimize the number of SQL queries being made

It have been seen taking days to convert these tables on very large installations.