searx-qt/docs/index.rst

********
Searx-Qt
********

Lightweight desktop application for Searx
#########################################

This is documentation for Searx-Qt version 0.3

.. image:: images/screenshot_0_3.png
    :alt: Searx-Qt v0.3 screenshot.
    :class: header

.. _index:

******
Index
******

- `Index <index_>`_

- `About <about_>`_
- - `Summary <about/summary_>`_
- - `Source <about/source_>`_
- - `License <about/license_>`_
- - `Dependencies <about/dependencies_>`_
- - `Translations <about/translations_>`_
- - `Contact <about/contact_>`_

- `Getting started <getting_started_>`_
- - `Install dependencies <getting_started/install_dependencies_>`_
- - `Installation <getting_started/installation_>`_

- `Usage <usage_>`_
- - `Profiles <usage/profiles_>`_
- - `Settings <usage/settings_>`_
- - `Instances <usage/instances_>`_
- - `Search <usage/search_>`_

- `Development <development_>`_
- - `Themes <development/themes_>`_
- - `Translations <development/translations_>`_

.. _about:

*****************
`About <about_>`_
*****************

.. _about/summary:

`Summary <about/summary_>`_
###########################

Searx-Qt is a lightweight desktop application that lets you search on
public Searx instances listed on `https://searx.space`.

Technically Searx-Qt is a client application that does magic with the Searx
API and Searx-Stats2 it's instances.json.

Since version `0.2` it is also possible to manage your own (private?)
instances with the use of a 'user' profile.


Searx
-----
Actual search requests will be made to a server running Searx software,
there are many public available. We call such servers 'Searx instances'
or in this document we may refer to just 'instance(s)'.

If you are not familiar with the Searx project; you can checkout their page
here: https://searx.github.io/searx/ or https://searx.me/

- API Docs: https://searx.github.io/searx/dev/search_api.html
- Source: https://github.com/searx/searx
- License: GPL3


Searx-Stats2
------------
The Searx-Stats2 project lists public Searx instances with statistics. The
original Searx-Stats2 instance is running at https://searx.space/. This is
where Searx-Qt will request a list with instances when the update button
is pressed.

- Source: https://github.com/searxng/searx-space
- License: GPL3


.. _about/source:

`Source <about/source_>`_
#########################
https://notabug.org/CYBERDEViL/searx-qt


.. _about/license:

`License <about/license_>`_
###########################
- GPL3 https://www.gnu.org/licenses/gpl-3.0.en.html

.. _about/dependencies:

`Dependencies <about/dependencies_>`_
#####################################

========  =========  =========  =====
 name      version    license    URL
========  =========  =========  =====
python    3          PSFL       https://docs.python.org/3/license.html
requests  -          Apache 2   http://docs.python-requests.org/en/master/
PyQt5     -          GPL3       https://www.riverbankcomputing.com/software/pyqt/intro
urllib3   -          MIT        https://urllib3.readthedocs.io/
========  =========  =========  =====


**Optional for socks proxy support**:

  ========  =========  =========  =====
   name      version    license    URL
  ========  =========  =========  =====
  pysocks   -          BSD        https://github.com/Anorov/PySocks
  ========  =========  =========  =====


**Building**:

  =========  =========  =========  =====
   name       version    license    URL
  =========  =========  =========  =====
  gettext    -          GPL        https://www.gnu.org/software/gettext/
  PyQt5-dev  Qt5        GPL3       https://www.riverbankcomputing.com/software/pyqt/intro
  =========  =========  =========  =====


.. _about/translations:

`Translations <about/translations_>`_
#####################################
The default language is English.

Since version `0.2` Searx-Qt is able (application-wise, not search results) to
be fully translated with the use of gettext and `.po .pot` files. However the
only translation available currently is for the Dutch language. If you like to
translate Searx-Qt in your language then you can find a example of how to do
that in the `development section <development_>`_. Please consider opening a
PR on ``https://notabug.org/CYBERDEViL/searx-qt`` after your translation has
finished.


.. _about/contact:

`Contact <about/contact_>`_
###########################
Please open a issue on ``https://notabug.org/cyberdevil/searx-qt``.


.. _getting_started:

**************************************
`Getting started <getting_started_>`_
**************************************

.. _getting_started/install_dependencies:

`Install dependencies <getting_started/install_dependencies_>`_
###############################################################

**Note:** ``python-requests`` is also dependent on ``python-urllib3`` ;
so ``python-urllib3`` will be installed with ``python-requests``
(No need to do a explicit install).

Debian / Ubuntu based
---------------------
Install required dependencies::

    # apt update
    # apt upgrade
    # apt install python3 python3-requests python3-pyqt5 gettext pyqt5-dev-tools

**Optional** for socks proxy support::

    # apt install python3-socks


Arch based
----------
Install required dependencies::

    # pacman -Syu python python-requests python-pyqt5 gettext qt5-tools

**Optional** for socks proxy support::

    # pacman -S python-pysocks


.. _getting_started/installation:

`Installation <getting_started/installation_>`_
###############################################

It is always recommended to let the package-manager of your system
do the installing of software, so your package-manager will keep
track of files installed. Only use ``setup.py`` directly if you
know what you are doing.

Since Searx-Qt isn't available in any GNU/Linux distribution (yet?); the
best option is to create a package for your distribution yourself from the
latest release. This will also mean that you have to manually update
Searx-Qt if there is a new version available.

**Note:** https://notabug.org/CYBERDEViL/searx-qt/releases

**Note:** noticed the ``#`` or ``$`` before every command? When there is a
``$`` before the command, it should be run as a regular user. ``#`` as root.

Debian based
------------
The steps below describe how to get a specific version of Searx-Qt; then
package and install it. This method is available from version
``0.1-beta2`` and up.

1) Make sure you have ``python3-stdeb`` and ``git`` installed::

    # apt install python3-stdeb git

2) Creating a working directory and ``cd`` in to it, you may
   change this to your own preference::

    $ mkdir ~/git
    $ cd ~/git

3) Cloning the repository and ``cd`` in to it::

    $ git clone "https://notabug.org/CYBERDEViL/searx-qt.git" "searx-qt"
    $ cd searx-qt

4) Checkout a specific version:

  **Note:** get a list with available tags (versions) with the
  ``git tag`` command.

  Below is a example to checkout version ``0.3-beta1``::

    $ git checkout 0.3-beta1

5) Create .deb::

    $ ./utils/gen_deb.sh

6) Install the created package::

    # dpkg -i ./deb_dist/python3-searx-qt_0.3-beta1-1_all.deb

Arch based
----------
For Arch based distributions there is a package available in the AUR;
https://aur.archlinux.org/packages/searx-qt/

1) Make sure you have ``git`` installed::

    # pacman -S git

2) Creating a working directory and ``cd`` in to it, you may change this
   to your own preference::

    $ mkdir ~/pkg
    $ cd ~/pkg

3) Getting the ``PKGBUILD`` from Arch AUR::

    $ git clone https://aur.archlinux.org/searx-qt.git
    $ cd searx-qt

4) Build and install Searx-Qt package::

    $ makepkg -si



.. _usage:

******************
`Usage <usage_>`_
******************

.. _usage/profiles:

`Profiles <usage/profiles_>`_
#############################

.. image:: images/profiles_window.png
    :alt: Profiles window
    :align: right

Profiles are useful when you want to have different settings and/or data without
to having to set it manually every-time. For example you can create a profile
named `Tor` which has different proxy and stats2 settings then you normal
profile.

There are two types of profiles:
 - `Stats2` profile
 - `User` profile

The profile type names maybe changed to something better, suggestions are
welcome.

Create a `Stats2` profile if you wish to get/update a list of Searx-instances
from a `Searx-Stats2` instance. For example the default `https://searx.space`.

Create a `User` profile if you wish to add/remove/update your own list with
Searx-instances.

 | **NOTE**: Profile types cannot be changed after the creation of the profile,
 |           but you can add multiple profiles of both types.

Creating new profile
--------------------
On first usage of `Searx-Qt` you will need to create a new profile. The `Add`
button (of the "Profile select" window) will open a dialog to do so.

There are profile settings presets (Web, Tor, i2p) which you can choose from.
The Tor preset sets the proxy to ``127.0.0.1:9050`` and
changes the Searx-Stats2 instance url to the onion address. The i2p preset
sets the proxy to ``127.0.0.1:4444``, it also adds some known
i2p instances of Searx.

    .. image:: images/profiles_new.png
        :alt: Create new profile dialog

Deleting a profile
------------------
I hope that it is self explanatory that the `Delete` button of the
"Profile select" window deletes the currently selected profile, it will ask
for confirmation before doing so.

It is not possible to delete a active profile (at-least it shouldn't ;-)).

.. _usage/settings:

`Settings <usage/settings_>`_
#############################


General
-------

.. image:: images/settings_general.png
    :align: right

Theme
=====

A ``Theme`` is a Searx-Qt specific ``stylesheet`` and the ``Base style`` is a
Qt theme/style.

The Searx-Qt specific ``Theme`` does override the ``Base style``.


CLI output level
================

The amount of CLI spam can be set here.

 - ``Info`` does print to ``stdout``.
 - ``Warning`` does print to ``stderr``.
 - ``Debug`` does print to ``stderr``.
 - ``Errror`` does print to ``stderr``.


Connection
----------

.. image:: images/settings_connection.png
    :align: right

Verify (SSL)
============
Request will fail on a invalid SSL/TLS certificate.

Leave checked if unsure.

See
https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification
for a more technical description.

Timeout
=======
Timeout in seconds for a single request.

Leave it at the default value of 10 seconds if unsure.

See https://requests.readthedocs.io/en/master/user/advanced/#timeouts for a
more technical description.

Proxy
=====
Here you can set a proxy that will be used for every connection Searx-Qt
makes. Leave the input field blank to use no proxy.

The 'Http' section can be used to proxy plain http:// requests to for
example instances without a certificate. The 'Https' section can be used
to proxy all https:// requests to instances (including fetching the
instances list data from https://searx.space) with a certificate. So you
can use a separate proxy for both protocols.

If you use a socks4 or socks5 proxy you probably want to make sure the
'Proxy DNS' checkbox is checked so DNS requests will also go through the
proxy. DNS proxy is not available for a http proxy type.

User-agents
===========
What user-agent string should Searx-Qt send?

After pressing the `Edit` button it will change to a `Save` button, you will
be able to edit the user-agent ?string(s) Searx-Qt will send. Some notes:

- One user-agent string per line.
- Set total blank to not send any user-agent string.

When the `Random` checkbox is checked and there are multiple user-agent
strings set then Searx-Qt will pick a random user-agent string from the list
for every request.


Searx-Stats2
------------
Here you can change the URL of the Searx-Stats2 instance you like to use
for fetching the instances data.

 | **NOTE**: This is only available for a `Stats2` profile type.

.. _usage/instances:


Guard
-------

Guard can put instances on a timeout or the blacklist when they are failing.

When the Guard rules are set properly, searches will be quicker over time
since failing instances are not used anymore. This reduces the chance of
requesting a search query to a instance that probably will fail again. Also
some instances block our request for whatever reason they have, and there is
no standard response so they all may (and many will) respond diffrently (Thus
we can not properly detect when we may use the API and when not).

 - It should be obvious that when ``Enable guard`` is checked \
   that Guard is enabled, when not checked Guard is disabled.

 - When Guard is enabled it is advised to also enable the ``Store log`` \
   option so that old failures can be evaluated against new failures after \
   Searx-Qt has been restarted.

 - Below the ``Store log`` option is a spinbox that defines for how long log \
   entries will be stored (in days).

 - A rule defines what a fail is and what should happen with the failing \
   instance.

.. image:: images/settings_guard.png
    :align: right

Rules
=====

A rule has the following variables:

 - ``Error Type``, the type of error.
 - ``Amount`` of fails.
 - ``Timeframe`` in which the ``Amount`` of fails have to occur.
 - ``Status``, the HTTP response code. (Only used for the ``WrongStatus`` \
   ``Error type``).
 - ``Destination``, what should happen to the instance? Should Guard put it on
   the blacklist or on a timeout?
 - ``Duration`` in minutes of the timeout. (Only used when ``Destionation`` \
   is set to ``Timeout``). When ``Timeout`` is used as ``Destination`` and \
   this is set to ``0`` minutes the instance will be on timeout until Searx-Qt
   has been restarted.

When ``Error type``, ``Amount``, ``Timeframe`` and ``Status`` are met the rule
will be triggered and the instance will be put on the ``Destination`` for
``Duration`` amount of time in minutes.

Log
===

Here you can see failed search requests. Failed search requests will only
be logged when Guard is enabled.

It logs as little as possible. The following is logged:

 - Date and time.
 - Instance url.
 - Error Type.
 - HTTP status code.
 - HTTP content/Error message (for debugging, may contain error message)




`Instances <usage/instances_>`_
###############################
.. image:: images/instances.png
    :align: right

A Searx instance is a server running the Searx project. Since we want to
preform searches to Searx instance(s) we need addresses of those
instance(s).

The interface to manage instances is on the right.

With `Stats2` profile type
--------------------------


When your profile is a `Stats2` type, the Searx-instances will be fetched
from ``https://searx.space/data/instances.json`` (or any other set in the
settings by your preference). The ``instances.json`` from ``search.space``
also contains a lot of other data about the instances it lists (which we can
use to filter instances based on our preferences).

When Searx-Qt is used for the first time you will need to update the
instances table. There is a 'Update' button between the Filter and the
Table that can be used for this. Searx-Qt will not update this automatically!

It maybe useful to update the instances data so now and then since public
instances appear, disappear and their stats change over time.


With `User` profile type
------------------------

If your profile is a `User` type you will have to add addresses of instances
manually.

This can be done by pressing the `Add instance` button right above
the instances table, a dialog will pop-up asking for the address to add
(without scheme).

The scheme (http:// or https://) can be selected from the combobox.

There is also a "Update data on add" checkbox, when this is checked
and `Add` is pressed it will automatically download data from
`http(s)://your-address/config`. Downloading/updating this data may also be
done later by right clicking on a (or multiple) Searx-instances in the table
and pressing `Update selected`.





.. _usage/instances/table:

Instances table
---------------
The instances table can be used to browse instances with their data that
remain after all filters. The table is also used to set the current
instance by left-clicking on one.

The currently used instance should also be visible bottom right in the
application it's status-bar.

Right-clicking in the table opens a context-menu from where you can do
the following:

- Whitelist/blacklist selected instance(s).
- Temporary blacklist.
- Copy selected instance(s) it's/their URL(s).
- Select All instances (CTRL+A should do the same).
- Hide or show columns.

If your profile is a `User` profile the context-menu will have the
following extra actions:

- Remove selected instance(s).
- Update selected instance(s).


Filter instances
----------------
When a filter is enabled and the instance it's value that is being
matched is unknown then it is excluded by default!

Network
=======
Filter instances on network type. Only instances that match one of the
checked network types remain.

Require ASN privacy
===================
Excludes instances that run their server at a known malicious network.
Like for example CloudFlare, Google, Akamai etc..

This does not give any guarantee, it only filters **known** privacy
violators!

For a full list of known malicious networks (technical):
https://github.com/dalf/searx-stats2/blob/master/searxstats/data/asn.py

Require IPv6
============
Exclude instances that don't have at least one IPv6 address.

Min version
===========
When a minimum version is set only instances that have the minimum set
or higher -version are included.

 - When ``Development`` is checked it will also include git versions.
 - When ``Unknown suffix`` is checked it will also include git versions with a
   unknown commit hash

Blacklist
=========
Here are the URLs of the instances that have been blacklisted, either manually
or automatically by Guard (when enabled).

There is a button right to each blacklist item to remove it from the
blacklist.

Hovering the remove button or the url of a blacklist item will show a tooltip
with some more info.

You can manually blacklist a instance by right clicking on a instance in
the instances table and click 'Add to blacklist'; multiple instances can
be blacklisted at once.

Blacklisted instances will be excluded from the table by default.

Whitelist
=========
Here are the URLs of the instances that have been manually whitelisted.
There is a button right to each whitelist item to remove it from the
whitelist.

You can manually whitelist a instance by right clicking on a instance in
the instances table and click 'Add to whitelist'; multiple instances can
be whitelisted at once.

Whitelisted instances will be in the table by default except when they are on
the timeout list.

Timeout
=======
This is a temporary blacklist. Instances manually put on a timeout will stay
here until Searx-Qt is restarted. When Guard is enabled it also may put
instances here depending on the set rules, those may persist after Searx-Qt is
restarted depending on the rule(s).

Hovering the remove button or the url of a timeout item will show a tooltip
with some more info.

.. _usage/search:

`Search <usage/search_>`_
#########################


.. _usage/search/bar:

Search bar
------------------------
.. image:: images/search_bar.png


.. _usage/search/bar/fallback:

Fallback
========
When checked it will pick a random instance from the instances table if a
search request fails one way or another and re-try the same request with
the freshly picked instance. There is a maximum amount of 10 tries (10
different instances to try the same request on).

What is fail?

- Connection errors including timeout.
- Wrong status code (not 200).
- No or malformed results returned.


.. _usage/search/random_every:

Random every
============
When checked it will automatically pick a random instance on a search request,
it will also hide the 'Random search button' because it makes it obsolete.

When not checked it will do search requests on the same instance unless the
request fails somehow and 'Fallback' is checked. Exception is when the
'Random search button' is used for the search request.


.. _usage/search/bar/random_search_button:

Random search button
====================
When pressed it will pick a random instance from the list and preform the
search request.


.. _usage/search/bar/reload_button:

Reload button
=============
When pressed it basically preforms a search request without 'Fallback'
whenever it is enabled or not, it also doesn't reset the page number. So
it can act as a reload button thus it's name, but it does more.

Note: When a search argument like the search query, instance URL,
categories/engines etc. has changed by user interaction it will do the
request with those changes, that isn't a real reload of the previous
request.

Dev-note: Probably this behavior should change or the name/icon should
change to something more fitting.


.. _usage/search/bar/search_button:

Search button
=============
Preform a search request on the currently selected instance.

Page number is reset, 'Fallback' and 'Random Every' options are honored.


.. _usage/search/bar/search_query_input:

Search query input
==================
The query you like to search for.

See https://searx.github.io/searx/user/search_syntax.html for what is
possible.

It will do a search request on ``enter`` key pressed, same behavior as
when the 'Search button' has been pressed.


.. _usage/search/options:

Search options
--------------

.. image:: images/search_options.png
    :alt: Search options bar

**NOTE**: Right clicking in (on the picture above) the dark area opens a
context-menu where you can manage what options you want to be visible or
not as shown in the image below.

.. image:: images/search_options_rmb.png
    :alt: Search options context menu
    :align: right


**NOTE**: Left-click (mouse) on the ``Categories`` or ``Engines`` label will
toggle the label collapsed/expanded state, to be able to reduce the height
when multiple options are selected.

**NOTE**: Right-click (mouse) on ``Categories`` or ``Engines`` label will open
a context menu with a option to uncheck all for convenience.


.. _usage/search/options/categories:

Categories
==========

.. image:: images/categories_menu.png
    :alt: Categories menu
    :align: right

A category is basically a collection of engines. When a category gets checked
then all the engines it represents will also be checked which in turn will
filter out all Searx-instances that don't have at least one of the checked
engines enabled.

Multiple categories may be selected.

The default (non-editable) categories will be compiled from categories listed
in engines. Besides default categories there is also a option to create custom
categories.

.. image:: images/custom_categories.png
    :alt: Custom categories window


.. _usage/search/options/engines:

Engines
=======
Here you can toggle what search engines should be enabled. It will
automatically filter out all instances from the instances table that doesn't
have at least one of the checked engines enabled. The checked engines will
be send with a search request to a Searx instance with the `enabled_engines`
param. You should only get results from engines that are checked.

If no engine is checked it means that it may return results of any engine
in the list.

The list with engines is created with data from the
`instances table <usage/instances/table_>`_, so only engines are listed that
are available from the instances table.


.. _usage/search/options/period:

Period
======
Search period you like results from. Options are ``Last day``,
``Last week``, ``Last month`` or ``Last year``.


.. _usage/search/options/language:

Language
========
If you want results in a specific language than you can select one here. The
set language will persist on restart.

Since Searx-Qt 0.3 there is a option to mark languages a favorites. Favorites
will appear on the top of the combobox list so you won't have to scroll.

Adding a language to favorite can be done by hovering the language and
pressing the spacebar on your keyboard. Removing a favorite works the same,
hover the favorite language and press the spacebar on your keyboard.

 | **NOTE**: Not all engines have language support and not all engines \
 |           honor the requested language. Searx-qt does not (yet?) act on \
 |           this.

 | **NOTE**: When ``Default language`` is set that means the default language
 |           of the instance.


Search results
--------------

Find text in results
====================
The find widget to search text inside the results can be opened/focused by the
keyboard shortcut ``Ctrl + F``.

Shortcuts that may be used while the find text input is activated:

    - ``Return`` to find the next match.
    - ``Shift + Return`` to find the previous match.
    - ``Escape`` to close.


.. _development:

*****************************
`Development <development_>`_
*****************************

**NOTE**: Make sure you are in the Searx-Qt source root (where utils/,
locales/, searxqt/ etc.. are).

**NOTE**: To run Searx-Qt without need to install::

    # Copy the executable from ./bin to cwd (searx-qt source root)
    cp bin/searx-qt ./

    # Start
    ./searx-qt

.. _development/themes:

`Themes <development/themes_>`_
###############################

Create new theme
----------------

A theme consists of icons, application css and search result/fail css.

Simple example to create a new theme from the default theme:

**NOTE**: Replace ``your_theme`` with the name of your new theme.

**NOTE**: For this example you should know basic CSS.

1. Setup structure for new theme.

    .. code-block::

      cp -r ./themes/default/ ./themes/your_theme/

2. Edit the application style ``./themes/your_theme/style.css``.
3. Edit the css used for failed search result message \
   ``./themes/your_theme/html_fail.css``.
4. Edit the css used for search results \
   ``./themes/your_theme/html_results.css``
5. Edit the icons (don't change their size) ``./themes/your_theme/icons/*.png``
6. Open ``./themes/your_theme/manifest.json`` and change the ``name`` \
   variable to the pretty name of your new theme.
7. See if your theme is listed, when not there is a error::

    python ./utils/themes_tool.py list

8. Compile the theme::

    python ./utils/themes_tool.py make your_theme

9. Open Searx-Qt, go to settings, change to your new theme (it should be \
   listed, else there is an error) and test it::

    ./searx-qt

10. Done? :-)

.. _development/translations:

`Translations <development/translations_>`_
###########################################

Searx-Qt will try to find a translation for your system locale and use that
when found.

To test translations the system locale should be installed for that language,
it doesn't have to be set for testing since we can easly override the ``LANG``
environment variable before executing Searx-Qt.

**NOTE**: The examples below are for a Dutch language translation, you
should replace ``nl_NL`` with the i18n locale ID of the language you whish to
translate.

Create new translation
----------------------

1. Setup structure for new language::

      # Update the searx-qt.pot template file.
      ./utils/locale_tool.sh -c

      # Create directory structure.
      mkdir -p ./locales/nl_NL/LC_MESSAGES/

      # Copy the template file to our new directory.
      cp ./locales/searx-qt.pot ./locales/nl_NL/LC_MESSAGES/searx-qt.po

      # Check if our new language is found. It should be listed.
      ./utils/locale_tool.sh --list

2. Start working on the translation.

   You can edit the ``./locales/nl_NL/LC_MESSAGES/searx-qt.po`` file with a
   text editor or a special editor for translations (that can handle ``.po``
   files like for example Poedit).

3. Compile the translation::

    ./utils/locale_tool.sh -m nl_NL

4. Test the translation::

    # Note: overriding XDG_DATA_HOME only for debugging translations! Themes
    #       will not work.
    XDG_DATA_HOME="$(pwd -P)/" LANG=nl_NL.UTF-8 ./searx-qt


Update existing translation
---------------------------

1. Update files::

    # Update the .pot template file.
    ./utils/locale_tool.sh -c

    # Update the translation it's .po file
    ./utils/locale_tool.sh -u nl_NL

2. Edit the translation:

   You can edit the ``./locales/nl_NL/LC_MESSAGES/searx-qt.po`` file with a
   text editor or a special editor for translations (that can handle ``.po``
   files like for example Poedit).

3. Compile the translation::

    ./utils/locale_tool.sh -m nl_NL

4. Test the translation::

    # Note: overriding XDG_DATA_HOME only for debugging translations! Themes
    #       will not work.
    XDG_DATA_HOME="$(pwd -P)/" LANG=nl_NL.UTF-8 ./searx-qt