Adapt both tracker-miner-fs and tracker-extract to store errors in
a location at ~/.cache. These errors can be then read eg. via CLI.

Move all cancellation handling closer to the TrackerExtractDecorator, so
we can pull files off persistence handling, since cancellation shouldn't
be considered an error on a future execution.

We still should exit ASAP, though.

Dummy files broke with the nie:DataObject/nie:InformationElement
split, as tracker-extract just did nothing while tracker-miner-fs
pre-filled the barebones nie:InformationElement.

We should at least fill in the rdf:types and the extra info to tie
the nie:InformationElement and nie:DataObject together, so return
a barebones TrackerResource to have that information successfully
attached.

Fixes extraction of games, ebooks, and other files using a dummy
extractor.

We just copy and free the mime_type variable, just don't do it.

If we stick longer than that for a single file, it would seem there's
some trouble processing it. As we're no longer guaranteed to stay in
control of the extraction thread, just exit the process. As the file
was being processed at the time of exiting, it would fall through the
retry/ignore paths we have in place for extractor crashes.

This timeout executes on the same thread/main context that called
tracker_extract_file(). It is guaranteed to trigger since we offload
extraction tasks to other threads.

This makes tracker-extract resilient to possible infinite loops that
might happen in 3rd party code via extractor modules.

Fixes: https://gitlab.gnome.org/GNOME/tracker-miners/-/issues/107

Make everything ready to use it in SPARQL production.

We don't offer all matches for a mimetype, only the best one. This
was mostly paranoia code in case modules failed. We don't expect
them to do so in practice, so this failure handling code is somewhat
pointless.

I'm not sure if most of these are useful. Statistics should be
shown by the CLI or other monitoring tools. I doubt users go
looking for them in the syslog/journal.

This fixes compile warnings which show up when we build with
-Dtracker_core=subproject. This is an example of the kind of
warning:

    [7/186] Compiling C object 'subprojects/tracker/src/tracker-store/7bb4675@@tracker-store@exe/tracker-config.c.o'.
    In file included from ../subprojects/tracker/src/tracker-store/tracker-config.c:21:
    subprojects/tracker/./config.h:25: warning: "GETTEXT_PACKAGE" redefined
       25 | #define GETTEXT_PACKAGE "tracker"
          |
    In file included from <command-line>:
    ./config.h:25: note: this is the location of the previous definition
       25 | #define GETTEXT_PACKAGE "tracker-miners"
          |

This fixes deprecation warnings about g_type_class_add_private()

The new G_ADD_PRIVATE() method has been around since GLib 2.38 (we
already depend on 2.40 at minimum).

See this blog post for more details:
https://www.bassi.io/articles/2013/06/21/the-king-is-dead/

Fixes https://gitlab.gnome.org/GNOME/tracker/issues/115

This depends on https://gitlab.gnome.org/GNOME/tracker/merge_requests/3

Sam Thursfield authored Oct 31, 2017 and Sam Thursfield committed Dec 16, 2017

We made a big compromise when splitting tracker core from tracker-miners
in that the common code that was needed by both parts would end up
duplicated. It's ugly but it works fine at the moment and allows us to
keep all of the common code private.

I had an issue when trying to embed tracker core into tracker-miners as
a Meson subproject though. Having two targets named tracker-common
caused confusion as duplicate targets aren't allowed, but they are not
quite equivalent so we can't just pick one or the other.

To work around this, I've renamed the copy in this repo to
tracker-miners-common. This only affects the target names, not the
actual function names.

https://bugzilla.gnome.org/show_bug.cgi?id=789725

The libmediaart dependency was disabled in commit 6a050686, it
doesn't make sense to drag this code around.

Those deal with plugins and potentially malicious content, make it
sure that any potential exploit is deprived of all tools that could
make it harmful.

https://bugzilla.gnome.org/show_bug.cgi?id=764786

It's basically unused, just use a private thread per-module so they're
easier to isolate.

https://bugzilla.gnome.org/show_bug.cgi?id=764786

This string is actually copied and freed, so definitely not const.

Sam Thursfield authored Mar 27, 2016 and Sam Thursfield committed Jul 14, 2016

For a long time, all the Tracker extractors have manually constructed a
SPARQL update command using TrackerSparqlBuilder to represent their
output. This commit changes all of them to use the TrackerResource
class instead, which makes the code a lot more concise and readable.

This introduces some API breaks in the internal libtracker-extract
library. This has been a private library since Tracker 0.16 or earlier,
so it's fine.

If the extractors only output SPARQL then they are only useful to
people who are using a SPARQL store. Now we can output a serialization
format like Turtle as well. This will hopefully make the extract modules
useful outside of Tracker itself.

I've tried to preserve the behaviour of the extractors as much as
possible, but there are two things that are now handled differently:

  * nao:Tag resources are given a fixed URI based on the tag label, such
    as <urn:tag:My_Tag>. Previously they were inserted as blank nodes,
    so tracker-store would give them unique IDs like <urn:uuid:1234...>

  * All extractors created nco:Contact resources for content publishers,
    but previously some would assign fixed URIs based on the name
    <urn:contact:James%20Joyce>, while others would insert them as blank
    nodes so they would be assigned unique IDs like <urn:uuid:1234...>.
    Now, all extractors create nco:Contact resources with fixed URIs
    based on the content creator's name.

https://bugzilla.gnome.org/show_bug.cgi?id=767472

Old output:

    tracker_mimetype_info_get_module: assertion 'info != NULL' failed
    No metadata or extractor modules found to handle this file

New output:

    file:///uri/of/file: No metadata or extractor modules found to handle this file

The message is now also printed to stderr rather than stdout, so that
stdout contains only the metadata itself in whatever output format the
caller chose.

Now that extraction is performed after the file got initially
inserted and has an URN, there's no reason we can't pass the
URN to the TrackerExtractInfo. This would make URNs available
to extract modules, which have some places where the SPARQL
produced would be clearer.

This actually opens the path for quite a cleanup, we might
pass a single TrackerSparqlBuilder to extract modules instead
of doing the postprocessing we currently perform.

There may be files where there's just nothing to be extracted, make
these fall through the success code paths. If a module fails for
any reason, it must return FALSE on the get_metadata func.

This will be necessary when we have an internal dummy function.

Extract modules might fail mid-processing, when the TrackerSparqlBuilder
has been partially written, we should bail out and move on to the next
extract module, as we can't guarantee its validity.

It doesn't make much sense to reuse TRACKER_DBUS_ERROR for these
purposes.

This uses a new API which is not compatible with previous versions.

https://bugzilla.gnome.org/show_bug.cgi?id=733863

This has not been compiled or used in years and likely doesn't produce an
ontology compatible output that we could use.

EXTERMINATE!

Also add i18n to command line use of tracker-extract.

It was hard to see what was going on with most logs as debug comments.

Also all thread messages are now debug and have been moved into the #ifdef for
other thread events

It's a bit easier to see what is going on now using -v 2 and there is less
mess with -v 3

If a tasks was iterated over multiple extraction modules, the task
ended up inserted on priv->running_tasks more than once, but only
removed once when the task finished, leaving dangling pointers in
priv->running_tasks.

If no extraction module was found for a file, the corresponding
TrackerExtractTask was being leaked.

This prevents a crash when reporting the statistics later, as there is a
NULL key that is expected to be a GModule.

Glib >= 2.30 already hard depend on libffi.

We require GLib 2.35.x, no need to check for previous versions.

Since 2.31, g_thread_create() is deprecated.

Since 2.31, g_mutex_new(), g_mutex_free(), g_cond_new(), g_cond_free() are
deprecated.

report_statistics() was locking on a mutex that was already freed,
so actually free it after this last use.