SyncEvolution 1.3.99.3 released

So far the development cycle for SyncEvolution 1.4 mostly focused on implementing the “PIM Manager” D-Bus API for IVI use cases [1]. The 1.3.99.3 pre-release starts to include more features and bug fixes again for syncing. For example, several ActiveSync improvements from Graham Cobb were included.

The remaining goal for 1.4, besides more testing of course, is to work out how to support Google CalDAV and CardDAV. I am in discussion with Google to get SyncEvolution whitelisted for use with these APIs – fingers crossed…

Upgrading from release 1.2.x

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Changes 1.3.2 -> 1.3.99.3

  • PIM Manager: add ReplaceSearch, always allow it

    The new ReplaceSearch is more flexible than RefineSearch. It can
    handle both tightening the search and relaxing it. The downside of it
    is the more expensive implementation (must check all contacts again,
    then find minimal set of change signals to update view).

    Previously, a search which had no filter set at all at the begining
    could not be refined. This limitation of the implementation gets
    removed by always using a FilteredView, even if the initial filter is
    empty.

  • PIM Manager: introduce CreateConfig()

    That SetPeer() allows modifying and creating a config leads to race
    conditions when multiple clients want to create a config. The new
    CreateConfig() avoids that by atomically checking that a config does
    not exist yet and creating it.

    SetPeer() is still available for backwards compatibility. It continues
    to be used for modifying an existing config in TestContacts.testSync
    to check the effect of the logging settings.

  • PIM Manager: fix double entries in filtered search with limit

    Stressing the FilteredView by using it in tests originally written
    for the FullView showed that the filling up a view may have
    used data while it was inconsistent internally, leading to
    contacts being present multiple times.

  • PIM Manager and sync: support location = GEO property (FDO #60373)

    Exposed as “location” -> (lat, long) in the D-Bus bindings.
    Reading, writing and updating are supported.

  • PIM Manager: support groups = CATEGORIES (FDO #60380)

    Allow reading and writing of groups (folks terminology), aka
    CATEGORIES in vCard.

  • PIM Manager: intelligent phone search in EDS (part of FDO #59571)

    If phone number search is enabled in EDS, then the direct search in
    EDS now uses the more accurate
    E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER comparison, with the E164
    formatted caller ID as value to compare against. This gives
    semantically correct results. The previous solution (now the
    fallback) had to use substring searches, which did not match if the
    contact’s phone number was not formatted according to E164 and
    which may have matched the wrong contacts if the trailing numbers
    are the same.

  • PIM Manager : use pre-computed normalized phone numbers from EDS (part of FDO #59571)

    When available, the pre-computed E164 number from EDS will be used
    instead of doing one libphonebook parser run for each telephone
    number while reading. Benchmarking showed that this parsing was the
    number one hotspot, so this is a considerable improvement.

  • PIM Manager: fix error messages

    Ensure and check that no unnecessary ERROR messages are printed.
    libfolks was used slightly incorrectly, leading to several
    harmless error messages (glib asserts). libphonenumber printed
    its error messages to stdout.

  • PIM Manager: fix memory leaks during writing of contacts

    Constructing the GValues created additional references instead
    of taking over ownership as intended.

  • D-Bus server: fix read-after-free bug when using syslog

    openlog() expects the string to remain valid. Must ensure that in
    LoggerSyslog by making a copy. Found with valgrind.

  • PIM Manager: make implementation of some of the D-Bus methods thread-safe

    The goal is to make it easier to extend syncevo-dbus-server
    with other IPC mechanisms, which then can call the native C++
    code directly.

    That code was not prepared to handle calls in threads other than the
    main one. Now this is checked when entering the methods and work is
    shifted to the main thread if necessary. In the meantime the calling
    thread waits for completion.

  • PIM Manager: check responsiveness (part of FDO #60851)

    Enhanced the testActive test so that it can detect when the D-Bus
    server stops responding for too long. One major reason for that was
    event processing in folks, which got improved as part of
    https://bugzilla.gnome.org/show_bug.cgi?id=694385

  • PIM Manager: adapt to gee 0.8

    Changed the code to compile with gee 0.8, as used by folks 0.9.x.
    Older versions of folks are no longer supported.

  • PBAP: support Bluez 5

    The new Bluez 5 API is the third supported API for doing PBAP
    transfers. It gets checked first, then the PBAB backend falls back to
    new-style obexd (file based, similar to Bluez 5, but not quite the
    same) and finally old-style obexd (data transfer via D-Bus).

    In contrast to previous APIs, Bluez 5 does not report the reason for a
    failed PBAP transfer. SyncEvolution then throws a generic “transfer
    failed” error with “reason unknown” as message.

  • command line: recover from slow sync with new sync modes

    The error message for an unexpected slow sync still mentioned
    the old and obsolete “refresh-from-client/server” sync modes.
    Better mention “refresh-from-local/remote”.

  • CalDAV: more workarounds for Google CalDAV + unique IDs

    Google became even more strict about checking REV. Tests which
    reused a UID after deleting the original item started to fail sometime
    since middle of December 2012.

  • CalDAV: work around Google server regression (undeclared namespace prefix in XML)

    Google CalDAV for a while (December 2012 till January 2013) sent
    invalid XML back when asked to include CardDAV properties in a
    PROPFIND. This got rejected in the XML parser, which prevents
    syncing calendar data:

    Neon error code 1: XML parse error at line 55: undeclared namespace prefix

    In the meantime Google fixed the issue in response to a bug report
    via email. But the workaround, only asking for the properties which
    are really needed, still makes sense and thus is kept.

  • WebDAV: don’t send Basic Auth via http proactively (FDO #57248)

    Sending basic authentication headers via http is insecure. Only do
    it proactively when the connection is encrypted and thus protects
    the information or when the server explicitly asks for it.

  • Nokia: always add TYPE=INTERNET to EMAIL (FDO #61784)

    Without the explicit TYPE=INTERNET, email addresses sent to a Nokia
    e51 were not shown by the phone and even got lost eventually (when
    syncing back).

    This commit ensures that the type is set for all emails sent to any
    Nokia phone, because there may be other phones which need it and
    phones which don’t, shouldn’t mind. This was spot-checked with a N97
    mini, which works fine with and without the INTERNET type.

    This behavior can be disabled again for specific Nokia phones by
    adding a remote rule which sets the addInternetEmail session variable
    to FALSE again.

    Non-Nokia phones can enable the feature in a similar way, by setting
    the variable to TRUE.

  • SyncML: config option for broken peers

    Some peers have problems with meta data (CtCap, old Nokia phones)
    and the sync mode extensions required for advertising the restart
    capability (Oracle Beehive). The default in SyncEvolution is to
    advertise the capability, so manual configuration is necessary when
    working with a peer that fails in that mode.

    Because the problem occurs when SyncEvolution contacts the peers
    before it gets the device information from the peer, dynamic rules
    based on the peer identifiers cannot be used. Instead the local config
    must already disable these extra features in advance.

    The “SyncMLVersion” property gets extended for this. Instead of just
    “SyncMLVersion = 1.0″ (as before) it now becomes possible to say
    “SyncMLVersion = 1.0, noctcap, norestart”.

    “noctcap” disables sending CtCap. “norestart” disables the sync mode
    extensions and thus doing multiple sync cycles in the same session
    (used between SyncEvolution instances in some cases to get client and
    server into sync in one session).

    Both keywords are case-insensitive. There’s no error checking for
    typos, so beware!

    The “SyncMLVersion” property was chosen because it was already in use
    for configuring SyncML compatibility aspects and adding a new property
    would have been harder.

  • ActiveSync: added support for specifying folder names

    Previously, the database field was interpreted as a Collection ID. This adds
    logic to allow the database to be interpreted as a folder path. The logic is:

    1) If the database is an empty string, pass it through (this is the most
    common case as it is interpreted as “use the default folder for the
    source type”).
    2) If the database matches a Collection ID, use the ID (this is the same as
    the previous behaviour).
    3) If the database matches a folder path name, with an optional leading “/”,
    use the Collection ID for the matching folder.
    4) Otherwise, force a FolderSync to get the latest folder changes from the
    server and repeat steps 2 and 3
    5) If still no match, throw an error.

  • ActiveSync: support for listing databases

    Now –print-databases scans folders on the ActiveSync server and
    shows suitable folders for the ActiveSync backends instead of the
    previous, hard-coded help text.

    Invoking –print-databases can be used as a workaround for
    “SyncFolder error: Invalid synchronization key” errors. A better
    solution would be to do that automatically, but there was no time
    to implement that. See FDO #61869 and “[SyncEvolution] Activesync server losing state”
    http://thread.gmane.org/gmane.comp.mobile.syncevolution/4295

  • command line: show backend error when listing databases fails

    The command line swallowed errors thrown by the backend while listing
    databases. Instead it just showed “: backend failed”. The goal
    was to not distract users who accidentally access a non-functional backend.
    But the result is that operations like –configure or –print-databases could
    fail without giving the user any hint about the root cause of the issue.

    Now the error explanation in all its gory details is included.

    For example, not having activesyncd running leads to:
    INFO] eas_contact: backend failed: fetching folder list:
    GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name
    org.meego.activesyncd was not provided by any .service files

    And running activesyncd without the necessary gconf keys shows up as:
    [INFO] eas_contact: backend failed: fetching folder list:
    GDBus.Error:org.meego.activesyncd.Error.AccountNotFound: Failed to find
    account [syncevolution@lists.intel.com]

  • Minor memory leak fix when using GDBus GIO: GDBusMethodInfo

    Also depends on a glib fix, see https://bugzilla.gnome.org/show_bug.cgi?id=695376

  • build fixes

    Avoid -lrt in make dependencies. Add missing pcre libs to
    syncevo-dbus-server. sqlite backend needs “#include ”
    (patch from Mario Kicherer).

1.3.99.1 => 1.3.99.2, 13.12.2012

  • PIM Manager searches for a caller ID (’phone’ search) in EDS
    directly while folks still starts up. No unification is done of
    these results. Intermediate results are replaced by the final ones
    from folks once those are ready.

  • PIM Manager: allow configuration of session directories (part of FDO #55921)

    Useful for moving the session directories to a temporary file system.
    They are essentially just useful for debugging when used as part of
    PIM Manager.

    • “logdir” – a directory in which directories are created with
      debug information about sync session
    • “maxsessions” – number of sessions that are allowed to exist
      after a sync (>= 0): 0 is special and means unlimited,
      1 for just the latest, etc.;
      old sessions are pruned heuristically (for example,
      keep sessions where something changed instead of
      some where nothing changed), so there is no hard
      guarantee that the last n sessions are present.
  • PIM Manager: write less data to disk (part of FDO #55921)

    Avoid writing config file changes to disk by enabling a new
    “ephemeral” mode for syncing via the PIM Manager. In this mode,
    config file changes are not flushed resp. discarded directly.
    This prevents writing to .ini files in ~/.config.

    The “synthesis” binfile client files are still written, but they get
    redirected into the session directory, which can (and should) be set
    to a temp file system and get deleted again quickly.

    Data dumps are turned off now in the configs created by the PIM
    Manager.

  • syncevo-dbus-server: use syslog instead of standard output by default

  • syncevo-dbus-server: command line options for controlling
    output and startup

    -d, –duration=seconds/’unlimited’ Shut down automatically
    when idle for this duration (default 300 seconds)
    -v, –verbosity=level Choose amount of output, 0 = no output,
    1 = errors, 2 = info, 3 = debug; default is 1.
    -o, –stdout Enable printing to stdout (result of operations)
    and stderr (errors/info/debug).
    -s, –no-syslog Disable printing to syslog.
    -p, –start-pim Activate the PIM Manager (= unified address book)
    immediately.

  • PIM Manager: store set of active address books persistently (FDO #56334)

    Together with storing the sort order persistently, this allows
    restarting the daemon and have it create the same unified address book
    again.

  • PIM Manager: remove colon from valid peer UID character set (FDO #56436)

    Using the UID as part of file names gets more problematic when
    allowing colons. Remove that character from the API and enforce
    the format in the source code.

  • PIM Manager API: introduce contact ID and use it for reading

    This makes it easier for a client to fully polulate its view with
    contact data. Previously it could happen that due to concurrent
    changes in the server, a client was returned data for the same
    contact multiple times. A client had to detect that and re-issue
    read requests.

  • PIM Manager API: optional ViewAgent.Quiescent() (FDO #56428)

    The callback is guaranteed to be invoked once when a search has
    finished sending its initial results, and not sooner. This makes it
    possible to check whether the current data contains some contact or
    not.

  • PIM Manager: limit number of search results (FDO #56142)

    A ‘limit’ search term with a number as parameter (formatted as string)
    can be added to a ‘phone’ or ‘any-contains’ search term to truncate the
    search results after a certain number of contacts. Example:
    Search([['any-contains', 'Joe'], ['limit', '10']])
    => return the first 10 Joes.

    As with any other search, the resulting view will be updated if
    contact data changes.

    The limit must not be changed in a RefineSearch(). A ‘limit’ term may
    (but doesn’t have to) be given. If it is given, its value must match
    the value set when creating the search. This limitation simplifies the
    implementation and its testing. The limitation could be removed if
    there is sufficient demand.

  • PIM Manager: fix refining a search

    Due to not mapping the local index in the view to the parent’s index,
    refining only worked in views where parent and child had the same
    index for the contacts in the search view.

  • PIM Manager: fix starting when done via search

    When the unified address book (= FullView) was not running yet at the
    time when a client wanted to search it, the unified address book was
    not started and thus the search never returned results.

  • PIM Manager: fix writing contact, support photo and notes

    folks and EDS do not support writing properties in parallel
    (https://bugzilla.gnome.org/show_bug.cgi?id=652659). Must serialize
    setting of modified properties.

  • PIM Manager: fix incorrect contact removal signals in filtered view

    The filtered view did not check whether a parent’s removed contact was
    really part of the view before sending a removal signal for it.

  • D-Bus: missing out parameters in D-Bus introspection XML (FDO #57292)

    The problem was in the C++ D-Bus binding. If the method that gets bound
    to D-Bus returns a value, that value was ignored in the signature:
    int foo() => no out parameter

    It works when the method was declared as having a retval:
    void foo (int &result) => integer out parameter

    This problem existed for both the libdbus and the GIO D-Bus
    bindings. In SyncEvolution it affected methods like GetVersions().

  • PIM Manager performance: pre-compute normalized telephone numbers

    Looking up by phone number spends most of its cycles in normalizing of
    the phone numbers in the unified address book. Instead of doing that
    work over and over again during the search, do it once while loading.

    Looking up a phone number only once does not gain from this change, it
    even gets slower (more memory intensive, less cache locality). Only
    searching multiple times becomes faster.

    Ultimately it would be best to store the normalized strings together
    with the telephone number inside EDS when the contact gets
    created. Work on that is in progress.

  • PIM Manager: improve performance of FullView sorting

    This fixes the hotspot during populating the FullView content: moving
    contacts around required copying IndividualData and thus copying
    complex C++ structs and strings. Storing pointers and moving those
    avoids that, with no lack of convenience thanks to boost::ptr_vector.

    Reordering also becomes faster, because the intermediate copy only
    needs to be of the pointers instead of the full content.

  • PIM Manager example: add benchmarking

    The new “checkpoints” split up the whole script run into pieces which
    are timed separately, with duration printed to stdout. In addition,
    tools like “perf” can be started for the duration of one phase.

  • EDS: fix creating databases

    –create-database was broken in combination with the final code in EDS
    3.6 because it passed NULL for the UID to e_source_new_with_uid(),
    which is considered an error by the implementation of that
    method. Must use e_source_new() if we don’t have a UID.

  • fixed some memory leaks, extended tests to cover new features and bugs

SyncEvolution 1.3.99.1, 25.10.2012

  • workarounds for warnings from g++ 4.5

  • engine: : local cache sync mode

    This patch introduces support for true one-way syncing (”caching”):
    the local datastore is meant to be an exact copy of the data on the
    remote side. The assumption is that no modifications are ever made
    locally outside of syncing. This is different from one-way sync modes,
    which allows local changes and only temporarily disables sending them
    to the remote side.

    Another goal of the new mode is to avoid data writes as much as
    possible.

    This new mode only works on the server side of a sync, where the
    engine has enough control over the data flow. Setting “sync” to:

    • “local-cache-incremental” will do an incremental sync (if possible)
      or a slow sync (otherwise). This is usually the right mode to use,
      and thus has “local-cache” as alias.
    • “local-cache-slow” will always do a slow sync. Useful for
      debugging or after (accidentally) making changes on the local side.
      An incremental sync will ignore such changes because they are not
      meant to happen, aren’t checked for to improve performance and
      thus will leave client and server out-of-sync!

    Both modes are recorded in the sync report of the local side. The
    target side is the client and records the normal “two-way” or “slow”
    sync modes.

    With the current SyncEvolution contact field list, first, middle and
    last name are used to find matches for contacts. For events, tasks
    and memos, time, summary and description are used.

  • HTTP proxy: useProxy=0 overrides http_* env variables

    Previously, if http_proxy was set, a proxy was used even if
    explicitly disabled. This prevented disabling the use of a proxy
    which only made sense in some cases, like accessing something
    that runs locally. Explicitly telling SyncEvolution to ignore
    http_proxy is necessary because it doesn’t support no_proxy.

  • WebDAV: auto-discovery fix

    With Google Contact + CardDAV the auto-discovery failed after
    finding the default address book, without reporting that result.

  • command line: implement –create/remove-database

    Creating a database is only possible with a chosen name. The UID is
    chosen automatically by the storage. Only implemented in the EDS
    backend.

  • file backend: sub-second mod time stamps

    Change tracking in the file backend used to be based on the
    modification time in seconds. When running many syncs quickly (as in
    testing), that can lead to changes not being detected when they happen
    within a second. Now the file backend also includes the sub-second part of the
    modification time stamp, if available.

    This change is relevant when upgrading SyncEvolution: most of the
    items will be considered “updated” once during the first sync after
    the upgrade (or a downgrade) because the revision strings get
    calculated differently.

  • D-Bus server: avoid progress outside of 0-100% range

    For example in the new TestLocalCache.testItemDelete100, the
    percentage value in the ProgressChanged signal become larger
    than 100 and then revert to 100 at the end of the sync.

    Seems the underlying calculation is faulty or simply inaccurate.
    This is not fixed. Instead the result is just clipped to the valid
    range.

  • code cleanup + improvements in testing

Source, Installation, Further information

http://syncevolution.org/blogs/pohly/2013/syncevolution-13993-released

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.