Skip to content

SyncEvolution 1.4.99.3 released

This release enhances CalDAV/CardDAV and PBAP syncing and fixes some
problems. The enhanced conflict handling introduced 1.4.99.2 was
unintentionally limited to syncs with EDS on the server side; it is
now also available for example in WebDAV<->SyncML bridge setups.

Details:

  • CardDAV: implement read-ahead

    Instead of downloading contacts one-by-one with GET, SyncEvolution now
    looks at contacts that are most likely going to be needed soon and
    gets all of them at once with addressbook-multiget REPORT.

    The number of contacts per REPORT is 50 by default, configurable by
    setting the SYNCEVOLUTION_CARDDAV_BATCH_SIZE env variable.

    This has two advantages:

    • It avoids round-trips to the server and thus speeds up a large
      download (100 small contacts with individual GETs took 28s on
      a fast connection, 3s with two REPORTs).
    • It reduces the overall number of requests. Google CardDAV is known
      to start issuing intermittent 401 authentication errors when the
      number of contacts retrieved via GET is too large. Perhaps this
      can be avoided with addressbook-multiget.
  • Google Calendar: remove child hack, improve alarm hack (FDO #63881)

    Google recently enhanced support for RECURRENCE-ID, so SyncEvolution
    no longer needs to replace the property when uploading a single
    detached event with RECURRENCE-ID. However, several things are still
    broken in the server, with no workaround in SyncEvolution:

    • Removing individual events gets ignored by the server;
      a full “wipe out server data” might work (untested).
    • When updating the parent event, all child events also get
      updated even though they were included unchanged in the data
      sent by SyncEvolution.
    • The RECURRENCE-ID of a child event of an all-day recurring event
      does not get stored properly.
    • The update hack seems to fail for complex meetings: uploading them
      once and then deleting them seems to make uploading them again
      impossible.

    All of these issues were reported to Google and are worked on there,
    so perhaps the situation will improve. In the meantime, syncing with
    Google CalDAV should better be limited to:

    • Downloading a Google calendar in one-way mode.
    • Two-way syncing of simple calendars without complex meeting
      serieses.

    While updating the Google workarounds, the alarm hack (sending a
    new event without alarms twice to avoid the automatic server side
    alarm) was simplified. Now the new event gets sent only once with a
    pseudo-alarm.

  • ephemeral sync: don’t write binfile client files (FDO #55921)

    When doing PBAP caching, we don’t want any meta data written because
    the next sync would not use it anyway. With the latest libsynthesis
    we can configure “/dev/null” as datadir for the client’s binfiles and
    libsynthesis will avoid writing them.

    The PIM manager uses this for PBAP syncing automatically. For testing
    it can be enabled by setting the SYNCEVOLUTION_EPHEMERAL env variable.

  • PBAP: avoid empty field filter

    Empty field filter is supposed to mean “return all supported
    fields”. This used to work and stopped working with Android phones
    after an update to 4.3 (seen on Galaxy S3); now the phone only
    returns the mandatory TEL, FN, N fields.

    The workaround is to replace the empty filter list with the list of
    known and supported properties. This means we only pull data we really
    need, but it also means we won’t get to see any additional properties
    that the phone might support.

  • PBAP: transfer in chunks (FDO #77272)

    If enabled via env variables, PullAll transfers will be limited to
    a certain numbers contacts at different offsets until all data got
    pulled. See PBAP README for details.

    When transfering in chunks, the enumeration of contacts for the engine
    no longer matches the PBAP enumeration. Debug output uses “offset #x”
    for PBAP and “ID y” for the engine.

  • PBAP: remove transfer via pipe

    Using a pipe was never fully supported by obexd (blocks
    obexd). Transfering in suitably sized chunks (FDO #77272) will be a
    more obexd friendly solution with a similar effect (not having to
    buffer the entire address book in memory).

  • engine: enable batching by default (FDO #52669)

    This reverts commit c435e937cd406e904c437eec51a32a6ec6163102.

    Commit 7b636720a in libsynthesis fixes an unitialized memory read in
    the asynchronous item update code path.

    Testing confirms that we can now used batched writes reliably with EDS
    (the only backend currently supporting asynchronous writes +
    batching), so this change enables it again also for local and
    SyncEvolution<->SyncEvolution sync (with asynchronous execution of
    contact add/update overlapped with SyncML message exchanges) and other
    SyncML syncs (with changes combined into batches and executed at the
    end of each message).

  • datatypes: fix contact caching

    Adding grouping to the contact datatype in 1.4.99.2 broke PBAP caching: when
    sending an empty URL, for example, during the sync, the parsed contact
    had different field arrays than the locally stored contact, because the
    latter was saved without the empty URL.

    This caused the field-based comparison to detect a difference even when
    the final, reencoded contact wasn’t different at all.

    To solve this, syncing now uses the same “don’t send empty properties”
    configuration as local storages. Testing shows that this resolves
    the difference for EDS.

    A more resilient solution would be to add a check based on the encoded
    data, but that’s more costly performance wise.

  • datatypes: fix vCard handling

    The new “preserve repeating properties during conflict resolution”
    feature from 1.4.99.2 was only active when using EDS as storage. The relevant
    merge script must be applied to all datatypes, not just the EDS
    flavor.

    The feature was also unintentionally active when running in
    caching mode. This caused two problems:

    • The cached item was updated even though only the
      ordering of repeating properties had been modified during
      merging.
    • The merged item was sent back to the client side, which
      was undesirable (caching is supposed to be one-way) or even
      impossible (PBAP is read-only, causing sync failures eith error 20030).

    We must check for caching mode and disable merging when it is active.
    We also must not tell the engine that we updated the photo property
    in the winning item, because then that item would get sent to the
    read-only side of the sync.

    Perhaps a better solution would be to actually tell the engine
    that the remote side is read-only when we activate caching mode.

  • datatypes: avoid PHOTO corruption during merge (FDO #77065)

    When handling an update/update conflict (both sides of the sync have an
    updated contact) and photo data was moved into a local file by EDS, the engine
    merged the file path and the photo data together and thus corrupted the photo.

    The engine does not know about the special role of the photo property.
    This needs to be handled by the merge script, and that script did not
    cover this particular situation. Now the loosing side is cleared,
    causing the engine to then copy the winning side over into the loosing
    one.

    Found by Renato Filho/Canonical when testing SyncEvolution for Ubuntu 14.04.

  • PBAP syncing: updated photo not always stored

    Because photo data was treated like a C string, changes after any
    embedded null byte were ignored during a comparison.

  • PIM: fix phone number normalization

    The parsed number always has a country code, whereas SyncEvolution expected it
    to be zero for strings without an explicit country code. This caused a caller
    ID lookup of numbers like “089788899″ in DE to find only telephone numbers in
    the current default country, instead of being more permissive and also finding
    “+189788899″. The corresponding unit test was broken and checked for the wrong
    result. Found while investigating an unrelated test failure when updating
    libphonenumber.

  • Various compiler problems and warnings fixed; compiles with
    –with-warnings=fatal on current Debian Testing and Ubuntu Trusty
    (FDO #79316).

Upgrading from releases <= 1.3.99.4:

If the value of “username/databaseUser/proxyUser” contains a colon,
the “user:” prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like “id:”, “goa:”, “signon:”, etc.).

The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is in the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. A backend
for Ubuntu Online Accounts was compiled on Ubuntu Saucy. The packages mentioned above are meta-packages which pull in suitable packages matching the distro during installation.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

Tizen Dev Conf: PIM in IVI – Contacts, your phone, your car

Here’s a copy of the slides that I presented at the Tizen Developer Conference 2014 in San Francisco. Compared to the previous talk at the Automotive Linux Summit in Edinburgh, you’ll find Tizen-specific instructions for getting started and information about the new OAuth2 (or, as a fallback, password based) Google Contact sync.

Attachment Size
TDC2014_contact_sync.pdf 1.21 MB

SyncEvolution 1.4.99.2 released

1.4.99.2 is a new development snapshot. It enhances interoperability with CardDAV servers and in particular Google Contacts considerably. Contact data gets converted to and from the format typically used by CardDAV servers, so now anniversary, spouse, manager, assistant and instant message information are exchanged properly.

Categories are not supported by Google CardDAV and thus still get lost.

Custom labels get stored in EDS as extensions and no longer get lost
when updating some other aspects of a contact. However, Evolution does
not show custom labels and removes them when editing a property which
has a custom label (BGO #730636).

Scanning for CardDAV/CalDAV resources was enhanced. It now finds
additional calendars with Google CalDAV and works with iCloud.
However, syncing with iCloud ran into a server bug (reported as
17001498 “CalDAV REPORT drops calendar data”) and needs further work.

The Ubuntu Online Accounts backend was added to syncevolution.org binaries, targeting Ubuntu Saucy and later.

Details:

  • vcard profile: avoid data loss during merging

    When resolving a merge conflict, repeating properties were taken
    wholesale from the winning side (for example, all email addresses). If
    a new email address had been added on the loosing side, it got lost.

    Arguably it is better to preserve as much data as possible during a
    conflict. SyncEvolution now does that in a merge script by checking
    which properties in the loosing side do not exist in the winning side
    and copying those entries.

    Typically only the main value (email address, phone number) is checked
    and not the additional meta data (like the type). Otherwise minor
    differences (for example, both sides have same email address, but with
    different types) would lead to duplicates.

    Only addresses are treated differently: for them all attributes
    (street, country, city, etc.) are compared, because there is no single
    main value.

  • engine: UID support in contact data

    Before, the UID property in a vCard was ignored by the engine.
    Backends were responsible for ensuring that the property is
    set if required by the underlying storage. This turned out to be
    handled incompletely in the WebDAV backend.

    This change moves this into the engine:

    • UID is now field. It does not get used for matching
      because the engine cannot rely on it being stored
      by both sides.
    • It gets parsed if present, but only generated if
      explicitly enabled (because that is the traditional
      behavior).
    • It is never shown in the DevInf’s CtCap
      because the Synthesis engine would always show it
      regardless whether a rule enabled the property.
      That’s because rules normally only get triggered
      after exchanging DevInf and thus DevInf has to
      be rule-independent. We don’t want it shown because
      then merging the incoming item during a local sync
      would use the incoming UID, even if it is empty.
    • Before writing, ensure that UID is set.

    When updating an existing item, the Synthesis engine reads
    the existing item, preserves the existing UID unless the peer
    claims to support UID, and then updates with the existing UID.

    This works for local sync (where SyncEvolution never claims
    to support UID when talking to the other side). It will break
    with peers which have UID in their CtCap although they
    rewrite the UID and backends whose underlying storage cannot
    handle UID changes during an update (for example, CardDAV).

  • CardDAV: use Apple/Google/CardDAV vCard flavor

    In principle, CardDAV servers support arbitrary vCard 3.0
    data. Extensions can be different and need to be preserved. However,
    when multiple different clients or the server’s Web UI interpret the
    vCards, they need to agree on the semantic of these vCard extensions.

    In practice, CardDAV was pushed by Apple and Apple clients are
    probably the most common clients of CardDAV services. When the Google
    Contacts Web UI creates or edits a contact, Google CardDAV will
    send that data using the vCard flavor used by Apple.

    Therefore it makes sense to exchange contacts with all CardDAV
    servers using that format. This format could be made configurable in
    SyncEvolution on a case-by-case basis; at the moment, it is
    hard-coded.

    During syncing, SyncEvolution takes care to translate between the
    vCard flavor used internally (based on Evolution) and the CardDAV
    vCard flavor. This mapping includes:

    X-AIM/JABBER/… <-> IMPP + X-SERVICE-TYPE

    Any IMPP property declared as X-SERVICE-TYPE=AIM will get
    mapped to X-AIM. Same for others. Some IMPP service types
    have no known X- property extension; they are stored in
    EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
    (for example, GaduGadu and Groupwise) are stored with
    X-SERVICE-TYPE values chosen by SyncEvolution so that
    Google CardDAV preserves them (GroupWise with mixed case
    got translated by Google into Groupwise, so the latter is used).

    Google always sends an X-ABLabel:Other for IMPP. This is ignored
    because the service type overrides it.

    The value itself also gets transformed during the mapping. IMPP uses
    an URI as value, with a chat protocol (like “aim” or “xmpp”) and
    some protocol specific identifier. For each X- extension the
    protocol is determined by the property name and the value is the
    protocol specific identifier without URL encoding.

    X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel

    The mapping is based on the X-ABLabel property attached to
    the X-ABRELATEDNAMES property. This depends on the English
    words “Spouse”, “Manager”, “Assistant” that Google CardDAV
    and Apple devices seem to use regardless of the configured
    language.

    As with IMPP, only the subset of related names which have
    a corresponding X- property extension get mapped. The rest
    is stored in EDS using the X-ABRELATEDNAMES property.

    X-ANNIVERSARY <-> X-ABDATE

    Same here, with X-ABLabel:Anniversary as the special case
    which gets mapped.

    X-ABLabel parameter <-> property

    CardDAV vCards have labels attached to arbitrary other properties
    (TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, …) via vCard group tags:
    item1.X-ABDATE:2010-01-01
    item1.X-ABLabel:Anniversary

    The advantage is that property values can contain arbitrary
    characters, including line breaks and double quotation marks,
    which is not possible in property parameters.

    Neither EDS nor KDE (judging from the lack of responses on the
    KDE-PIM mailing list) support custom labels. SyncEvolution could
    have used grouping as it is done in CardDAV, but grouping is not
    used much (not at all?) by the UIs working with the vCards in EDS
    and KDE. It seemed easier to use a new X-ABLabel parameter.

    Characters which cannot be stored in a parameter get converted
    (double space to single space, line break to space, etc.) during
    syncing. In practice, these characters don’t appear in X-ABLabel
    properties anyway because neither Apple nor Google UIs allow entering
    them for custom labels.

    The “Other” label is used by Google even in case where it adds no
    information. For example, all XMPP properties have an associated
    X-ABLabel=Other although the Web UI does not provide a means to edit
    or show such a label. Editing the text before the value in the UI
    changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
    other fields.

    Therefore the “Other” label is ignored by removing it during syncing.

    X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
    order of properties in the UI) gets stored in CardDAV. The only exception
    is Google CardDAV which got confused when an IMPP property had both
    X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
    X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
    of chat information can get lost when syncing with Google.

  • synccompare: support grouping and quoted parameter strings

    Grouped properties are sorted first according to the actual property
    name, then related properties are moved to the place where their group
    tag appears first. The first grouped property gets a “- ” prefix, all
    following ones are just indended with ” “. The actual group tag is not
    part of the normalized output, because its value is irrelevant:

    BDAY:19701230

    • EMAIL:john@custom.com
      X-ABLabel:custom-label2

      FN:Mr. John 1 Doe Sr.
    • IMPP;X-SERVICE-TYPE=AIM:aim:aim
      X-ABLabel:Other
    • X-ABDATE:19710101
      X-ABLabel:Anniversary

    Redundant tags (those set for only a single property, X-ABLabel:Other)
    get removed as part of normalizing an item.

  • WebDAV: use server’s order when listing collections

    When doing a recursive scan of the home set, preserve the order of
    entries as reported by the server and check the first one first. The
    server knows better which entries are more relevant for the user (and
    thus should be the default) or may have some other relevant
    order. Previously, SyncEvolution replaced that order with sorting by
    URL, which led to a predictable, but rather meaningless order.

    For example, Google lists the users own calendar first, followed by
    the shared calendars sorted alphabetical by their name. Now
    SyncEvolution picks the main calendar as default correctly when
    scanning from https://www.google.com/calendar/dav/.

  • WebDAV: improved database search (Google, Zimbra)

    Zimbra has a principal URL that also serves as home set. When using it
    as start URL, SyncEvolution only looked the URL once, without listing
    its content, and thus did not find the databases.

    When following the Zimbra principal URL indirectly, SyncEvolution did
    check all of the collections there recursively. Unfortunately that
    also includes many mail folders, causing the scan to abort after
    checking 1000 collections (an internal safe guard).

    The solution for both includes tracking what to do with a URL. For the
    initial URL, only meta data about the URL itself gets
    checked. Recursive scanning is only done for the home set. If that
    home set contains many collections, scanning is still slow and may run
    into the internal safe guard limit. This cannot be avoided because the
    CalDAV spec explicitly states that the home set may contain normal
    collections which contain other collections, so a client has to do the
    recursive scan.

    When looking at a specific calendar, Google CalDAV does not report
    what the current principal or the home set is and therefore
    SyncEvolution stopped after finding just the initial calendar. Now it
    detects the lack of meta information and adds all parents also as
    candidates that need to be looked at. The downside of this is that it
    doesn’t know anything about which parents are relevant, so it ends up
    checking https://www.google.com/calendar/ and
    https://www.google.com/.

    In both cases Basic Auth gets rejected with a temporary redirect to
    the Google login page, which is something that SyncEvolution must
    ignore immediately during scanning without applying the resend
    workaround for “temporary rejection of valid credentials” that can
    happen for valid Google CalDAV URLs.

  • WebDAV: enhanced database search (Google Calendar)

    Additional databases where not found for several
    reasons. SyncEvolution ignored all shared calendars
    (http://calendarserver.org/ns/shared) and Google marks the additional
    calendars that way. The other problem was that the check for leaf
    collections (= collections which cannot contain other desired
    collections) incorrectly excluded those collections instead of only
    preventing listing of their content.

    With this change,
    https://www.google.com/calendar/dav/?SyncEvolution=Google can be used
    as starting point for Google Calendar.

  • WebDAV: fix database scan on iCloud

    The calendar home set URL on iCloud (the one ending in /calendars/) is
    declared as containing calendar data. That was enough for
    SyncEvolution to accept it incorrectly as calendar. However, the home
    set only contains calendar data indirectly.

  • WebDAV: support redirects between hosts and DNS SRV lookup based on URL

    When finding a new URL, we must be prepared to reinitialize the Neon
    session with the new host settings.

    iCloud does not have .well-known support on its www.icloud.com
    server. To support lookup with a non-icloudd.com email address, we
    must do DNS SRV lookup when access to .well-known URLs fails. We do
    this without a www prefix on the host first, because that is what happens
    to work for icloud.com.

    With these changes it becomes possible to do database scans on Apple
    iCloud, using syncURL=https://www.icloud.com or
    syncURL=https://icloud.com. Giving the syncURL like this is only
    necessary for a username that does not end in @icloud.com. When
    the syncURL is not set, the domain for DNS SRV lookup is taken
    from the username.

  • WebDAV: more efficient item creation

    PUT has the disadvantage that a client needs to choose a name and then
    figure out what the real name on the server is. With Google CardDAV that
    requires sending another request and only works because the server happens
    to remember the original name (which is not guaranteed!).

    POST works for new items without a name and happens to be implemented
    by Google such that the response already includes all required
    information (new name and revision string).

    POST is checked for as described in RFC 5995 once before creating a new
    item. Servers which don’t support it continue to get a PUT.

  • WebDAV: send “User-Agent: SyncEvolution”

    Apple iCloud servers reject requests unless they contain a User-Agent
    header. The exact value doesn’t seem to matter. Making the string
    configurable might be better, but can still be done later when it
    is more certain whether and for what it is needed.

  • WebDAV: refactor and fix DNS SRV lookup

    The syncevo-webdav-lookup script was not packaged. It did not report
    “not found” DNS results correctly and the caller did not check for
    this either, so when looking up the information for a domain which
    does not have DNS SRV entries, SyncEvolution ended up retrying for
    while as if there had been a temporary lookup problem.

  • signon: make Accounts optional

    The new “signon” provider only depends on lib[g]signon-glib. It uses
    gSSO if found, else UOA. Instead of pulling parameters and the
    identity via libaccounts-glib, the user of SyncEvolution now has to
    ensure that the identity exists and pass all relevant parameters
    in the “signon:” username.

  • gSSO: adapt to gSSO >= 2.0

  • config templates: Funambol URLs

    Funambol turned of the URL redirect from my.funambol.com to
    onemedia.com. The Funambol template now uses the current URL. Users
    with existing Funambol configs must updated the syncURL property
    manually to https://onemediahub.com/sync

    Kudos to Daniel Clement for reporting the change.

  • command line: fix –update from directory

    The “–update ” operation was supposed to take the
    item luids from the file names inside the directory. That part
    had not been implemented, turning the operation accidentally
    into an “–import”.

    Also missing was the escaping/unescaping of luids. Now the
    same escaping is done as in command line output and command
    line parsing to make the luids safe for use as file name.

  • testing: added server-specific tests for CardDAV covering
    remote item formats and edit conflicts.

Upgrading from releases <= 1.3.99.4:

If the value of “username/databaseUser/proxyUser” contains a colon,
the “user:” prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like “id:”, “goa:”, “signon:”, etc.).

The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is in the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. A backend
for Ubuntu Online Accounts was compiled on Ubuntu Saucy. The packages mentioned above are meta-packages which pull in suitable packages matching the distro during installation.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

SyncEvolution 1.4.1 released

The first bug fix release in the 1.4 series addresses some issues which
occurred on some systems. Several issues with Akonadi were fixed.

Details:

  • EDS: only load one backend plugin of each kind

    SyncEvolution was meant to load the syncecal or syncebook shared object
    which uses the most recent libraries (libical, libecal/libebook) on
    the system and then stop loooking for alternatives. Due to a string
    handling bug the check for already backends always found nothing,
    leading to multiple conflicting backends loaded on some systems (for
    example, those with libical0 and libical1 installed).

    If that happened, the backend became unusable.

  • ical: workaround for libical 1.0 builtin timezone change

    libical 1.0 started to return VTIMEZONE definitions with multiple
    absolute transition times instead of RRULEs. This causes problems when
    exchanging data with peers (see
    https://sourceforge.net/p/freeassociation/bugs/95/).

    In SyncEvolution, this affected sending an event using New Zealand
    time in vCalendar 1.0 format to a phone, because the internal,
    out-dated definition of the time zone in libsynthesis was used as
    fallback when loading RRULE-based timezone definitions from libical
    failed (see “[SyncEvolution] Some events showing wrong time on
    phone”). It might also affect exchanging data with CalDAV peers (not
    tested).

    The workaround is to include the original code from libical.

  • dbus-session.sh: create XDG_RUNTIME_DIR

    More recent distros (for example, Ubuntu Saucy) rely on
    XDG_RUNTIME_DIR. Each time dbus-session.sh runs, it must
    ensure that the runtime dir exists and is empty.

    This was a problem when trying to run activesyncd + SyncEvolution
    on a headless Ubuntu Saucy server (see FDO #76273).

  • Akonadi: support KDE Notes, enhanced “database” check

    The KDE Notes resources store items under a different MIME type than the one
    used in AKonadi (see “[Kde-pim] note format”). SyncEvolution use the same type
    as Akonadi and thus did not find existing KDE Notes resources.

    To support both while KDE and Akonadi transition to the same type,
    SyncEvolution now looks for notes resources using both MIME types and accepts
    both kinds of items when reading. When writing, SyncEvolution picks the MIME
    type that is supported by the resource, which hopefully avoids confusing the
    KDE app using the resource (untested).

    As a positive side effect, the “database” value used for opening a resource is
    now checked more thoroughly. Non-existent resources and the type mismatches
    like pointing a “kde-contacts” backend to a calendar resource are now detected
    early.

  • Akonadi: ensure that UID is set (FDO #74342)

    Akonadi resources do not enforce iCalendar 2.0 semantic like
    “each VEVENT must have a UID” (see “[Kde-pim] iCalendar semantic”).
    When receiving an event from a peer which itself does not enforce
    that semantic (Funambol, vCalendar 1.0 based phones), then we
    need to generate a UID, otherwise KOrganizer will ignore the
    imported event.

  • Akonadi: avoid threading problem in HTTP server mode (FDO #75672)

    When used as storage in a server, Akonadi got called in a background thread
    that gets created to handle slow initialization of sources and preventing
    ensuing timeouts in HTTP clients (probably not needed for Akonadi itself,
    but may still be useful when combining it with other sources).

    Akonadi cannot be used like that, leading to false “Akonadi not running”
    errors or (if one got past that check) failing item operations.

  • autotools: Add QtCore include path to KDEPIM_CFLAGS (FDO #75670)

    This fixes an issue where configure fails to find Akonadi when
    test programs do not compile because QString is not found.

  • Enhanced testing again: faster execution, less false negatives
    under load. Re-enabled testing of Akonadi.

Upgrading from releases <= 1.3.99.4:

If the value of “username/databaseUser/proxyUser” contains a colon,
the “user:” prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like “id:”, “goa:”, “signon:”, etc.).

The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is in the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “stable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. The packages mentioned above are meta-packages which pull in suitable packages matching the distro during installation.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

SyncEvolution 1.4 released

The 1.4 release of SyncEvolution replaces 1.3.2 as the stable, supported release.

1.4 is the first stable version with the in-vehicle infotainment (IVI) PIM Manager included. GENIVI Diagnostic Log and Trace (DLT) is also supported. For more information about this aspect of SyncEvolution, see the PBAP and PIM entries in the 1.3.99 release notes and these Automotive Linux Summit slides.

The biggest change for normal Linux users is Google CalDAV/CardDAV
authentication with OAuth2. These are the open protocol that Google
currently supports and thus the recommended way of syncing with
Google, replacing ActiveSync and SyncML (both no longer available to
all Google customers).

Support for Google CardDAV is new. Like Evolution, SyncEvolution does
not yet support some of the advanced features of the server, in
particular custom labels for phone numbers, emails and
addresses. Likewise, some client properties are not supported by the
server: CALURI, CATEGORIES, FBURL, GEO and ROLE are not supported. Of
ORG, only the first two components are supported. Currently,
properties not supported by one side get lost in a full roundtrip
sync. Instant Messaging information is supported by both sides with
different vCard extensions; the server stores these extensions without
showing the information, while SyncEvolution drops the data sent by
the server.

SyncEvolution depends on external components for OAuth2. It can be
compiled to use gSSO or GNOME Online Accounts. The latter is
enabled in binaries from syncevolution.org. GNOME Online Accounts >=
3.10 works out of the box for CalDAV and CardDAV. 3.8 is guaranteed to
work for CalDAV and may also work for CardDAV, if the Linux
distribution ships a patched version (like Debian Testing/Jessie does). If it
does not, then GNOME Online Accounts 3.8 binary can be patched to also
support CardDAV, see the GOA README. Anything older than 3.8 does not
work. Support for Ubuntu Online Accounts is available when compiling
from source. For setup instructions see the GOA and signon READMEs.

Binary packages of 1.4 on syncevolution.org have enhanced support for
recent distros. They now work with EDS >= 3.6 and < 3.6. Distros
with libical1 like Ubuntu Saucy are also supported.

The HTTP server became better at handling message resends when the
server is slow with processing a message. The server is able to keep a
sync session alive while loading the initial data set by sending
acknowledgment replies before the client times out.

Some issues in CalDAV, WebDAV and SyncML were fixed.

Graham R. Cobb contributed several patches for enhancing ActiveSync
support and making it work with Exchange 2010. Guido Günther provided
some patches addressing problems when compiling SyncEvolution for
Maemo.

Details:

  • D-Bus server: support DLT (FDO #66769)

    Diagnostic Log and Trace (DLT) manages a sequence of log messages,
    with remote controllable level of detail. SyncEvolution optionally
    (can be chosen at compile time and again at runtime) uses DLT
    instead of its own syncevolution-log.html files. See README-DLT.rst
    for more information.

    To use the feature, configure SyncEvolution with
    “–enable-dbus-server=–dlt –no-syslog”

  • D-Bus server: fix abort when mixing auto-sync and manual operations (FDO #73562)

    When enabling auto-sync for a config and then accessing or syncing the
    config manually via the command line tool, the server would abort at
    the time when the auto-sync was originally scheduled.

  • D-Bus server: accept WBXML with charset in incoming connections

    A user reported via email that the Nokia 515 sends
    ‘application/vnd.syncml+wbxml; charset=UTF-8′ as type of its messages
    this tripped up the syncevo-http-server, leading to:

    [ERROR] syncevo-dbus-server: /org/syncevolution/Server: message type
    ‘application/vnd.syncml+wbxml; charset=UTF-8′ not supported for starting
    a sync

  • D-Bus server: command line options for controlling output and startup

    The system log is used by default now. New command line options can be
    used to change this:

  -d, --duration=seconds/'unlimited'    Shut down automatically
                                        when idle for this duration (default 300 seconds)
  -v, --verbosity=level                 Choose amount of output, 0 = no output,
                                        1 = errors, 2 = info, 3 = debug; default is 1.
  --dbus-verbosity=level                Choose amount of output via D-Bus signals, 0 = no output,
                                        1 = errors, 2 = info, 3 = debug; default is 2.
  -o, --stdout                          Enable printing to stdout (result of operations)
                                        and stderr (errors/info/debug).
  -s, --no-syslog                       Disable printing to syslog.
  -p, --start-pim                       Activate the PIM Manager (= unified address book)
                                        immediately.

  • D-Bus: missing out parameters in D-Bus introspection XML (FDO #57292)

    The problem was in the C++ D-Bus binding. If the method that gets bound
    to D-Bus returns a value, that value was ignored in the signature:
    int foo() => no out parameter

    It works when the method was declared as having a retval:
    void foo (int &result) => integer out parameter

    This problem existed for both the libdbus and the GIO D-Bus
    bindings. In SyncEvolution it affected methods like GetVersions().

  • D-Bus server: avoid progress outside of 0-100% range

    For example in the new TestLocalCache.testItemDelete100, the
    percentage value in the ProgressChanged signal become larger
    than 100 and then revert to 100 at the end of the sync.

    Seems the underlying calculation is faulty or simply inaccurate.
    This is not fixed. Instead the result is just clipped to the valid
    range.

  • sync: less verbose output, shorter runtime

    For each incoming change, one INFO line with “received x[/out of y]”
    was printed, immediately followed by another line with total counts
    “added x, updated y, removed z”. For each outgoing change, a “sent
    x[/out of y]” was printed.

    In addition, these changes were forwarded to the D-Bus server where a
    “percent complete” was calculated and broadcasted to clients. All of
    that caused a very high overhead for every single change, even if the
    actual logging was off. The syncevo-dbus-server was constantly
    consuming CPU time during a sync when it should have been mostly idle.

    To avoid this overhead, the updated received/sent numbers that come
    from the Synthesis engine are now cached and only processed when done
    with a SyncML message or some other event happens (whatever happens
    first).

    To keep the implementation simple, the “added x, updated y, removed z”
    information is ignored completely and no longer appears in the output.

  • command line: implement –create/remove-database

    Creating a database is only possible with a chosen name. The UID is
    chosen automatically by the storage. Only implemented in the EDS
    backend.

  • command line: execute –export and –print-items while the source is still reading

    Instead of reading all item IDs, then iterating over them, process
    each new ID as soon as it is available. With sources that support
    incremental reading (only the PBAP source at the moment) that provides
    output sooner and is a bit more memory efficient.

  • command line: recover from slow sync with new sync modes

    The error message for an unexpected slow sync still mentioned
    the old and obsolete “refresh-from-client/server” sync modes.
    Better mention “refresh-from-local/remote”.

  • command line: show backend error when listing databases fails

    The command line swallowed errors thrown by the backend while listing
    databases. Instead it just showed “: backend failed”. The goal
    was to not distract users who accidentally access a non-functional backend.
    But the result is that operations like –configure or –print-databases could
    fail without giving the user any hint about the root cause of the issue.

    Now the error explanation in all its gory details is included.

    For example, not having activesyncd running leads to:
    INFO] eas_contact: backend failed: fetching folder list:
    GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name
    org.meego.activesyncd was not provided by any .service files

    And running activesyncd without the necessary gconf keys shows up as:
    [INFO] eas_contact: backend failed: fetching folder list:
    GDBus.Error:org.meego.activesyncd.Error.AccountNotFound: Failed to find
    account [syncevolution@lists.intel.com]

  • password handling: fix usage of GNOME Keyring and KWallet (FDO #66110)

    When clients like the GTK sync-ui stored a password, it was always
    stored as plain text in the config.ini file by the
    syncevo-dbus-server. The necessary code for redirecting the password
    storage in a keyring (GNOME or KWallet) simply wasn’t called in that
    case.

    The command line tool, even when using the D-Bus server to run the
    operation, had the necessary code active and thus was not affected.
    Now all SyncEvolution components use the same default: use safe
    password storage if either GNOME Keyring or KWallet were enabled
    during compilation, don’t use it if not.

    Fixing this revealed other problems, like not being able to store
    certain passwords that lacked the necessary lookup criteria (like
    syncURL and/or username). To address this, the lookup criteria where
    extended and a new check was added to avoid accidentally removing
    other passwords. As a result, it may be possible that SyncEvolution
    no longer finds passwords that were stored with older versions of
    SyncEvolution. In such a case the passwords must be set again.

  • GNOME: clean up keyring access and require libgnome-keyring >= 2.20

    The updated error messages now always include information about the
    password and libgnome-keyring error texts.

    A workaround is used for the “Error communicating with
    gnome-keyring-daemon” problem that started to appear fairly
    frequently in the automated testing once the keyring was actually
    used. The problem shows up with some additional debug messages:

    Gkr: received an invalid, unencryptable, or non-utf8 secret
    Gkr: call to daemon returned an invalid response: (null).(null)()

    It seems that sometimes setting up a session with GNOME keyring
    fails such that all further communication leads to decoding problem.

    There is an internal method to reset the session, but it cannot be
    called directly. As a workaround, fake the death of the GNOME
    keyring daemon and thus trigger a reconnect when retrying the GNOME
    keyring access. This is done by sending a D-Bus message, which will
    also affect other clients of GNOME keyring, but hopefully without
    user-visible effects.

  • config: enhanced password handling

    It is possible to configure a plain username/password combination
    once in SyncEvolution and then use references to it in other
    configurations, instead of having to set (and update) the
    credentials in different places. This is useful in particular with
    WebDAV, where credentials had to be repeated several times (target
    config, in each database when used as part of SyncML) or when using
    a service which requires several configs (Google via SyncML and
    CalDAV).

    To use this, create a sync config for a normal peer or a dedicated
    config just for the credentials, with “username/password/syncURL”
    set. The “syncURL” must be set to something identifying the peer if
    GNOME Keyring is used for the password storage.

    Then set “username”, “databaseUser” and “proxyUser” properties to
    “id:” and all read and write access
    to those properties will be redirected by SyncEvolution into that
    other configuration. This even works in the GTK UI.

    For user names which contain colons, the new “user:” format
    must be used. Strings without colons are assumed to be normal user
    names, so most old configurations should continue to work.

  • signon: new backend using libgsignond-glib + libaccounts-glib

    The code works with gSSO (https://01.org/gsso) and Ubuntu Online
    Accounts.

  • GOA: get OAuth2 tokens out of GNOME Online Accounts

    “username = goa:…” selects an account in GOA and retrieves the
    OAuth2 token from that.

  • WebDAV: support OAuth2

    If given an authentication configuration which can handle OAuth2,
    then OAuth2 is used instead of plain username/password
    authentication.

  • WebDAV: support Google CardDAV, break Yahoo

    Google CardDAV has one peculiarity: it renames new contacts during PUT without
    returning the new path to the client. See also
    http://lists.calconnect.org/pipermail/caldeveloper-l/2013-July/000524.html

    SyncEvolution already had a workaround for that (PROPGET on old path, extract
    new path from response) which happened to work. This workaround was originally
    added for Yahoo, which sometimes merges contacts into existing ones. In
    contrast to Yahoo, Google really seems to create new items.

    Without some server specific hacks, the client cannot tell what happened.
    Because Google is currently supported and Yahoo is not, let’s change the
    hard-coded behavior to “renamed items are new”.

  • WebDAV: started testing with owndrive.com = OwnCloud

  • WebDAV: avoid segfault during collection lookup

    Avoid referencing pathProps->second when the set of paths that
    PROPFINDs returns is empty. Apparently this can happen in combination
    with Calypso.

  • CalDAV: more workarounds for Google CalDAV + unique IDs

    Google became even more strict about checking REV. Tests which
    reused a UID after deleting the original item started to fail sometime
    since middle of December 2012.

  • CalDAV: work around Google server regression (undeclared namespace prefix in XML)

    Google CalDAV for a while (December 2012 till January 2013) sent
    invalid XML back when asked to include CardDAV properties in a
    PROPFIND. This got rejected in the XML parser, which prevents
    syncing calendar data:

    Neon error code 1: XML parse error at line 55: undeclared namespace prefix

    In the meantime Google fixed the issue in response to a bug report
    via email. But the workaround, only asking for the properties which
    are really needed, still makes sense and thus is kept.

  • WebDAV: auto-discovery fix

    With Google Contact + CardDAV the auto-discovery failed after
    finding the default address book, without reporting that result.

  • WebDAV: don’t send Basic Auth via http proactively (FDO #57248)

    Sending basic authentication headers via http is insecure. Only do
    it proactively when the connection is encrypted and thus protects
    the information or when the server explicitly asks for it.

  • file backend: sub-second mod time stamps

    Change tracking in the file backend used to be based on the
    modification time in seconds. When running many syncs quickly (as in
    testing), that can lead to changes not being detected when they happen
    within a second. Now the file backend also includes the sub-second part of the
    modification time stamp, if available.

    This change is relevant when upgrading SyncEvolution: most of the
    items will be considered “updated” once during the first sync after
    the upgrade (or a downgrade) because the revision strings get
    calculated differently.

  • GTK UI: fixed two crashes – running a sync with no service selected
    and a 64 bit pointer problem recently discovered by Tino Keitel when
    compiling the Debian package with -fPIE.

  • packaging: compatible with EDS up to and including 3.10 and both
    libical.so.0 and libical.so.1

    The binary packages now contain different versions of syncecal.so
    and syncebooks.so to cover different combinations of EDS and libical.

  • libical: compatibiliy mode for libical.so.0 and libical.so.1

    libical 1.0 broke the ABI, leading to libical.so.1. The only relevant change
    for SyncEvolution is the renumbering of ICAL_*_PROPERTY enum values. We can
    adapt to that change at runtime, which allows us to compile once with
    libical.so.0, then patch executables or use dynamic loading to run with the
    more recent libical.so.1 if we add 1 to the known constants.

  • packaging: fix rpm (FDO #73347)

    After installing the syncevolution.org rpm on OpenSUSE,
    SyncEvolution was not starting because its shared libraries were not
    found unless “ldconfig” was called manually. Now the package does
    that automatically.

  • packaging: fix description

    The syncevolution-bundle description of both rpm and deb
    packagesaccidentally used the same description as
    syncevolution-evolution.

  • glib: fix double-free of source tags

    glib 2.39.0 (aka GNOME 3.10) as found in Ubuntu Trusty introduces
    warnings when g_source_remove() is passed an unknown tag. SyncEvolution
    did this in two cases: in both, the source callback returned false and thus
    caused the source to be removed by the caller. In that case, the explicit
    g_source_remove() is redundant and must be avoided.

    Such a call is faulty and might accidentally remove a new source with the same
    tag (unlikely though, given that tags seem to get assigned incrementally).

    The only noticable effect were additional error messages with different
    numbers:

    [ERROR] GLib: Source ID 9 was not found when attempting to remove it

  • EDS: fix compile problem with boost and EDS > 3.36

    This fixes the following problem, seen with Boost 1.53.0 on altlinux
    when compiling for EDS >= 3.6:

    /usr/include/boost/smart_ptr/shared_ptr.hpp: In instantiation of ‘typename boost::detail::sp_array_access::type boost::shared_ptr::operator const [with T = char*; typename boost::detail::sp_array_access::type = void; std::ptrdiff_t = long int]‘:
    src/backends/evolution/EvolutionSyncSource.cpp:163:38: required from here
    /usr/include/boost/smart_ptr/shared_ptr.hpp:663:22: error: return-statement with a value, in function returning ‘void’ [-fpermissive]
    make[2]: *** [src/backends/evolution/src_backends_evolution_syncecal_la-EvolutionSyncSource.lo]

  • EDS contacts: avoid unnecessary DB writes during slow sync

    Traditionally, contacts were modified shortly before writing into EDS
    to match with Evolution expectations (must have N, only one CELL TEL,
    VOICE flag must be set). During a slow sync, the engine compare the
    modified contacts with the unmodified, incoming one. This led to
    mismatches and/or merge operations which end up not changing anything
    in the DB because the only difference would be removed again before
    writing.

  • EDS contacts: read-ahead cache

    Performance is improved by requesting multiple contacts at once and
    overlapping reading with processing. On a fast system (SSD, CPU fast
    enough to not be the limiting factor), testpim.py’s testSync takes 8
    seconds for a “match” sync where 1000 contacts get loaded and compared
    against the same set of contacts. Read-ahead with only 1 contact per
    query speeds that up to 6.7s due to overlapping IO and
    processing. Read-ahead with the default 50 contacts per query takes
    5.5s. It does not get much faster with larger queries.

  • PBAP: add support for obexd 0.47, 0.48 and Bluez 5

    obexd 0.48 is almost the same as obexd 0.47, except that it dropped
    the SetFilter and SetFormat methods in favor of passing a Bluex 5-style
    filter parameter to PullAll.

  • PBAP: various enhancements for efficient caching of contacts

  • HTTP server: handle message resends

    If a client gave up waiting for the server’s response and resent its message
    while the server was still processing the message, syncing failed with
    “protocol error: already processing a message” raised by the
    syncevo-dbus-server because it wasn’t prepared to handle that situation.

    The right place to handle this is inside the syncevo-http-server, because it
    depends on the protocol (HTTP in this case) whether resending is valid or
    not. It handles that now by tracking the message that is currently in
    processing and matching it against each new message. If it matches, the new
    request replaces the obsolete one without sending the message again to
    syncevo-dbus-server. When syncevo-dbus-server replies to the old message, the
    reply is used to finish the newer request.

  • engine: prevent timeouts in HTTP server mode

    HTTP SyncML clients give up after a certain timeout (SyncEvolution
    after RetryDuration = 5 minutes by default, Nokia e51 after 15
    minutes) when the server fails to respond.

    This can happen with SyncEvolution as server when it uses a slow
    storage with many items, for example via WebDAV. In the case of slow
    session startup, multithreading is now used to run the storage
    initializing in parallel to sending regular “keep-alive” SyncML
    replies to the client.

    By default, these replies are sent every 2 minutes. This can be
    configured with another extensions of the SyncMLVersion property:
    SyncMLVersion = REQUESTMAXTIME=5m

    Other modes do not use multithreading by default, but it can be
    enabled by setting REQUESTMAXTIME explicitly. It can be disabled
    by setting the time to zero.

    The new feature depends on a libsynthesis with multithreading enabled
    and glib >= 2.32.0, which is necessary to make SyncEvolution itself
    thread-safe. With an older glib, multithreading is disabled, but can
    be enabled as a stop-gap measure by setting REQUESTMAXTIME explicitly.

  • Various testing and stability enhancements. SyncEvolution had to
    be made thread-safe for the HTTP timeout prevention.

  • Nokia: always add TYPE=INTERNET to EMAIL (FDO #61784)

    Without the explicit TYPE=INTERNET, email addresses sent to a Nokia
    e51 were not shown by the phone and even got lost eventually (when
    syncing back).

    This commit ensures that the type is set for all emails sent to any
    Nokia phone, because there may be other phones which need it and
    phones which don’t, shouldn’t mind. This was spot-checked with a N97
    mini, which works fine with and without the INTERNET type.

    This behavior can be disabled again for specific Nokia phones by
    adding a remote rule which sets the addInternetEmail session variable
    to FALSE again.

    Non-Nokia phones can enable the feature in a similar way, by setting
    the variable to TRUE.

  • SyncML: config option for broken peers

    Some peers have problems with meta data (CtCap, old Nokia phones)
    and the sync mode extensions required for advertising the restart
    capability (Oracle Beehive). The default in SyncEvolution is to
    advertise the capability, so manual configuration is necessary when
    working with a peer that fails in that mode.

    Because the problem occurs when SyncEvolution contacts the peers
    before it gets the device information from the peer, dynamic rules
    based on the peer identifiers cannot be used. Instead the local config
    must already disable these extra features in advance.

    The “SyncMLVersion” property gets extended for this. Instead of just
    “SyncMLVersion = 1.0″ (as before) it now becomes possible to say
    “SyncMLVersion = 1.0, noctcap, norestart”.

    “noctcap” disables sending CtCap. “norestart” disables the sync mode
    extensions and thus doing multiple sync cycles in the same session
    (used between SyncEvolution instances in some cases to get client and
    server into sync in one session).

    Both keywords are case-insensitive. There’s no error checking for
    typos, so beware!

    The “SyncMLVersion” property was chosen because it was already in use
    for configuring SyncML compatibility aspects and adding a new property
    would have been harder.

  • ActiveSync: added support for specifying folder names

    Previously, the database field was interpreted as a Collection ID. This adds
    logic to allow the database to be interpreted as a folder path. The logic is:

    1) If the database is an empty string, pass it through (this is the most
    common case as it is interpreted as “use the default folder for the
    source type”).
    2) If the database matches a Collection ID, use the ID (this is the same as
    the previous behaviour).
    3) If the database matches a folder path name, with an optional leading “/”,
    use the Collection ID for the matching folder.
    4) Otherwise, force a FolderSync to get the latest folder changes from the
    server and repeat steps 2 and 3
    5) If still no match, throw an error.

  • ActiveSync: support for listing databases

    Now –print-databases scans folders on the ActiveSync server and
    shows suitable folders for the ActiveSync backends instead of the
    previous, hard-coded help text.

    Invoking –print-databases can be used as a workaround for
    “SyncFolder error: Invalid synchronization key” errors. A better
    solution would be to do that automatically, but there was no time
    to implement that. See FDO #61869 and “[SyncEvolution] Activesync server losing state”
    http://thread.gmane.org/gmane.comp.mobile.syncevolution/4295

  • SyncML: workarounds for broken peers

    Some peers have problems with meta data (CtCap, old Nokia phones) and
    the sync mode extensions required for advertising the restart
    capability (Oracle Beehive).

    Because the problem occurs when SyncEvolution contacts the peers
    before it gets the device information from the peer, dynamic rules
    based on the peer identifiers cannot be used. Instead the local config
    must already disable these extra features in advance.

    The “SyncMLVersion” property gets extended for this. Instead of just
    “SyncMLVersion = 1.0″ (as before) it now becomes possible to say
    “SyncMLVersion = 1.0, noctcap, norestart”.

    “noctcap” disables sending CtCap. “norestart” disables the sync mode
    extensions and thus doing multiple sync cycles in the same session
    (used between SyncEvolution instances in some cases to get client and
    server into sync in one session).

    Both keywords are case-insensitive. There’s no error checking for
    typos, so beware!

    The “SyncMLVersion” property was chosen because it was already in use
    for configuring SyncML compatibility aspects and adding a new property
    would have been harder.

  • engine: local cache sync mode

    This patch introduces support for true one-way syncing (”caching”):
    the local datastore is meant to be an exact copy of the data on the
    remote side. The assumption is that no modifications are ever made
    locally outside of syncing. This is different from one-way sync modes,
    which allows local changes and only temporarily disables sending them
    to the remote side.

    Another goal of the new mode is to avoid data writes as much as
    possible.

    This new mode only works on the server side of a sync, where the
    engine has enough control over the data flow. Setting “sync” to:

    • “local-cache-incremental” will do an incremental sync (if possible)
      or a slow sync (otherwise). This is usually the right mode to use,
      and thus has “local-cache” as alias.
    • “local-cache-slow” will always do a slow sync. Useful for
      debugging or after (accidentally) making changes on the local side.
      An incremental sync will ignore such changes because they are not
      meant to happen, aren’t checked for to improve performance and
      thus will leave client and server out-of-sync!

    Both modes are recorded in the sync report of the local side. The
    target side is the client and records the normal “two-way” or “slow”
    sync modes.

    With the current SyncEvolution contact field list, first, middle and
    last name are used to find matches for contacts. For events, tasks
    and memos, time, summary and description are used.

  • Minor memory leak fix when using GDBus GIO: GDBusMethodInfo

    Also depends on a glib fix, see BGO #695376

  • build fixes

    Avoid -lrt in make dependencies. Add missing pcre libs to
    syncevo-dbus-server. sqlite backend needs “#include ”
    (patch from Mario Kicherer).

  • autotools: fix temp file vulnerability during compilation (CVE-2014-1639)

    We must use the temporary file that was created for us securily, not
    a temp file named after that file. This caused a temp file vulnerability
    and the real temporary files were not deleted by the script.

  • workarounds for warnings from g++ 4.5

Upgrading from releases <= 1.3.99.4:

If the value of “username/databaseUser/proxyUser” contains a colon,
the “user:” prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like “id:”, “goa:”, “signon:”, etc.).

The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

SyncEvolution 1.3.99.7 -> 1.4

Compared to the pre-release, 1.4 mostly just enhanced the testing.
Compatibility with GNOME 3.10 and a glib-related issue that existed
almost forever without causing obvious problems were
fixed. syncevolution.org binaries now finally work with distros using
libical.so.1 (for example, Ubuntu Saucy and Trusty).

Details:

  • autotools: fix temp file vulnerability during compilation (CVE-2014-1639)

    We must use the temporary file that was created for us securily, not
    a temp file named after that file. This caused a temp file vulnerability
    and the real temporary files were not deleted by the script.

  • glib: fix double-free of source tags

    glib 2.39.0 (aka GNOME 3.10) as found in Ubuntu Trusty introduces
    warnings when g_source_remove() is passed an unknown tag. SyncEvolution
    did this in two cases: in both, the source callback returned false and thus
    caused the source to be removed by the caller. In that case, the explicit
    g_source_remove() is redundant and must be avoided.

    Such a call is faulty and might accidentally remove a new source with the same
    tag (unlikely though, given that tags seem to get assigned incrementally).

    The only noticable effect were additional error messages with different
    numbers:

    [ERROR] GLib: Source ID 9 was not found when attempting to remove it

  • libical: compatibiliy mode for libical.so.0 and libical.so.1

    libical 1.0 broke the ABI, leading to libical.so.1. The only relevant change
    for SyncEvolution is the renumbering of ICAL_*_PROPERTY enum values. We can
    adapt to that change at runtime, which allows us to compile once with
    libical.so.0, then patch executables or use dynamic loading to run with the
    more recent libical.so.1 if we add 1 to the known constants.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “stable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. The packages mentioned above are meta-packages which pull in suitable packages matching the distro during installation.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

SyncEvolution 1.3.99.7 released

This the final release candidate for 1.4. No further changes planned unless
new problems are found.

Details:

  • SSO: support Ubuntu Online Accounts

    When compiling from source on recent Ubuntu it becomes possible to
    use Ubuntu Online Accounts for authenticating against Google’s
    CalDAV and CardDAV servers.

  • D-Bus server: fix abort when mixing auto-sync and manual operations (FDO #73562)

    When enabling auto-sync for a config and then accessing or syncing the
    config manually via the command line tool, the server would abort at
    the time when the auto-sync was originally scheduled.

  • D-Bus server: accept WBXML with charset in incoming connections

    A user reported via email that the Nokia 515 sends
    ‘application/vnd.syncml+wbxml; charset=UTF-8′ as type of its messages.
    This tripped up the syncevo-http-server, leading to:

    [ERROR] syncevo-dbus-server: /org/syncevolution/Server: message type ‘application/vnd.syncml+wbxml; charset=UTf-8′ not supported for starting a sync

    We need to strip the ‘; charset=UTF-8′ suffix also when checking for
    WBXML.

  • packaging: compatible with EDS up to and including 3.10

    The packages now contain three versions of syncecal.so:

    • one for EDS < 3.6
    • one for EDS >= 3.6 < 3.10
    • one for EDS >= 3.10 with the libecal-1.2 soname patched

    The third flavor became necessary because EDS 3.10 accidentally
    changed the soname. The API and ABI actually is the same.

    Package meta-data was fixed to reflect the extended range of
    compatible EDS libraries, so syncevolution-evolution can be installed
    again with recent EDS.

  • packaging: update syncevolution-kde dependencies

    kdebase-runtime became kde-runtime in Debian Wheezy. Accept both
    as prerequisite of syncevolution-kde to allow installation on
    newer distros without pulling in the transitional kdebase-runtime
    package.

  • packaging: fix rpm (FDO #73347)

    After installing the syncevolution.org rpm on OpenSUSE,
    SyncEvolution was not starting because its shared libraries were not
    found unless “ldconfig” was called manually. Now the package does
    that automatically.

  • packaging: fix description

    The syncevolution-bundle description of both rpm and deb
    packagesaccidentally used the same description as
    syncevolution-evolution.

  • test improvements, integration of cppcheck and clang’s scan-build

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

SyncEvolution 1.3.99.6 released

This update focuses on SyncEvolution in IVI again. It adds support for
GENIVI Diagnostic Log and Trace (DLT) and enhances searching in the
unified address book.

The biggest change for normal Linux desktop users is enhanced support
for recent distros. Binaries on syncevolution.org now work with EDS >=
3.6 and < 3.6. Distros with libical1 like Ubuntu Saucy are also
supported. Automated testing was updated to cover these newer
platforms more thoroughly.

The binaries support Google CalDAV in combination with GNOME Online Accounts (GOA) >= 3.8 and and Google CardDAV with GOA >= 3.10. To use CardDAV with GOA 3.8, one has to patch or recompile GOA.

Support for Google CalDAV/CardDAV with Ubuntu Online Accounts requires recompilation of SyncEvolution with one additional patch.

Details:

  • GNOME Online Accounts: fix D-Bus problem in syncevolution.org binaries

    Support was included in syncevolution.org binaries, but was not
    tested and did not actually work due to some issue accessing
    the D-Bus session.

  • libsynthesis: partial fix batching of items

    The batching of contact writes introduced with SyncEvolution
    1.3.99.4 caused problems with non-SyncEvolution SyncML peers when
    syncing contacts stored in EDS >= 3.6. EDS < 3.6 was not affected.

    That part is fixed. However, even in SyncEvolution<->SyncEvolution
    syncs another crash was found. This will require more investigation.
    Clearly the feature is not ready yet for general sync, so for now
    it is disabled by default and only enabled in the simpler PBAP
    sync.

  • libsynthesis: avoid redundant (and sometimes slow) getaddrbyname() (FDO #70771)

    The network lookup of the hostname can be slow (10 second delay when
    not connected) and shouldn’t be necessary anyway, so disable it.

  • PIM: accent-insensitive and transliterated search (FDO #56524)

    Accent-insensitive search ignores accents, using the same code as in
    EDS. Transliterated search ignores foreign scripts by transliterating
    search term and contact properties to Latin first. That one is using
    ICU directly in the same way as EDS, but doesn’t use the EDS
    ETransliterator class to avoid extra string copying.

    This commit changes the default behavior such that searching is by
    default most permissive (case- and accent-insensitive, does
    transliteration). Flags exist to restore more restrictive matching.

  • PIM: relax phone number matching

    Previously, the current default country was used to turn phone numbers
    without an explicit country code into full E164 numbers, which then
    had to match the search term when doing a caller ID lookup.

    This was inconsistent with EDS, where a weaker
    EQUALS_NATIONAL_PHONE_NUMBER was done. The difference is that a
    comparison between a number with country code matches one without if
    the national number of the same, regardless of the current default
    country. This is better because it reduces the influence of the hard
    to guess default country on matching.

    Another advantage of this change is the lower memory consumption and
    faster comparison, because strings are now stored in 4 + 8 byte
    numbers instead of strings of varying length.

  • PIM: fix incorrect write into pim-manager.ini (FDO #70772)

    Removing a peer accidentally wrote the updated list of active
    address books into the “sort” property of pim-manager.ini, which
    then prevented starting the PIM Manager.

  • PIM: ignore broken sort order in config (FDO #70772)

    Failure to set the sort order from pim-manager.ini should not
    prevent the startup of the PIM Manager because the client cannot
    really diagnose and fix the problem. It is better to try again with
    the default sort order.

  • PIM: adapt to locale changes at runtime (FDO #66618)

    Listen to signals from localed D-Bus system service and update all
    internal state which depends on the current locale. This state includes:

    • pre-computed data in all loaded contacts
    • filtering (for example, case sensitivity is locale dependent)
    • the sort order

    This feature can be controlled by setting the SYNCEVOLUTION_LOCALED
    env variable:

    • “session” – use a localed instance on the D-Bus session bus instead
      of the system instance. This was originally meant for
      testing, but might also be useful for per-user setting changes.
    • “none” – disables the feature
  • PIM: fix sync.py + multiple peers

    Due to overwriting a variable, configuring multiple different
    peers did not work.

  • D-Bus server: support DLT (FDO #66769)

    Diagnostic Log and Trace (DLT) manages a sequence of log messages,
    with remote controllable level of detail. SyncEvolution optionally
    (can be chosen at compile time and again at runtime) uses DLT
    instead of its own syncevolution-log.html files. See README-DLT.rst
    for more information.

    To use the feature, configure SyncEvolution with
    “–enable-dbus-server=–dlt –no-syslog”

  • EDS: enhanced compatibility mode

    SyncEvolution compiled for EDS < 3.6 can now also load EDS backends
    compiled for EDS >= 3.6. The packaging for syncevolution.org uses
    that to bundle EDS backends compiled on different distros in the
    same package.

  • EDS: SYNCEVOLUTION_EBOOK_QUERY env variable

    Setting the SYNCEVOLUTION_EBOOK_QUERY env variable to a valid EBook
    query string limits the results to contacts matching that
    query. Useful only in combination with –print-items or
    –export. Only implemented for EDS >= 3.6.

  • EDS: fix compile problem with boost and EDS > 3.36

    This fixes the following problem, seen with Boost 1.53.0 on altlinux
    when compiling for EDS >= 3.6:

    /usr/include/boost/smart_ptr/shared_ptr.hpp: In instantiation of ‘typename boost::detail::sp_array_access::type boost::shared_ptr::operator const [with T = char*; typename boost::detail::sp_array_access::type = void; std::ptrdiff_t = long int]‘:
    src/backends/evolution/EvolutionSyncSource.cpp:163:38: required from here
    /usr/include/boost/smart_ptr/shared_ptr.hpp:663:22: error: return-statement with a value, in function returning ‘void’ [-fpermissive]
    make[2]: *** [src/backends/evolution/src_backends_evolution_syncecal_la-EvolutionSyncSource.lo]

  • PBAP: add support for obexd 0.48

    obexd 0.48 is almost the same as obexd 0.47, except that it dropped
    the SetFilter and SetFormat methods in favor of passing a Bluex 5-style
    filter parameter to PullAll.

    SyncEvolution now supports 4, in words, four different obexd
    APIs. Sigh.

    This feature was originally announced for SyncEvolution 1.3.99.5,
    but not actually included in the code yet.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

PIM – it’s all about the contacts

I just presented the work done on PIM in Tizen IVI 3.0 at the Linux Foundation’s Automotive Linux Summit.

The work is based on a considerably enhanced Evolution Data Server, libphonenumber, folks, and of course SyncEvolution. If you want to know more about this, the slides are attached.

Attachment Size
als-edinburgh-2013.pdf 1.09 MB

SyncEvolution 1.3.99.5 released

SyncEvolution now supports Google CalDAV/CardDAV with OAuth2
authentication. These are the open protocol that Google currently
supports and thus the recommended way of syncing with Google,
replacing ActiveSync and SyncML (both no longer available to all
Google customers).

Support for Google CardDAV is new. Because of a vCard encoding issue
on the server side, spaces in long notes may get removed. Like
Evolution, SyncEvolution does not yet support some of the advanced
features of the server, in particular custom labels for phone numbers,
emails and addresses. Likewise, some client properties are not
supported by the server: CALURI, CATEGORIES, FBURL, GEO and ROLE are
not supported. Of ORG, only the first two components are supported.
Currently, properties not supported by one side get lost in a full
roundtrip sync.

SyncEvolution depends on external components for OAuth2. It can be
compiled to use gSSO or GNOME Online Accounts. GNOME Online
Accounts >= 3.10 works out of the box for CalDAV and CardDAV, 3.8 only
for CardDAV (but the GNOME Online Accounts binary can be patched to
also support CalDAV), anything older than 3.8 does not
work. Support for Ubuntu Online Accounts should not be hard to add,
but is not available yet.

Evolution >= 3.6 is not supported by the binaries on syncevolution.org. On systems with a more recent Evolution, SyncEvolution must be compiled from source.

Details:

  • GTK UI: fixed two crashes – running a sync with no service selected
    and a 64 bit pointer problem recently discovered by Tino Keitel when
    compiling the Debian package with -fPIE.

  • password handling: fix usage of GNOME Keyring and KWallet (FDO #66110)

    When clients like the GTK sync-ui stored a password, it was always
    stored as plain text in the config.ini file by the
    syncevo-dbus-server. The necessary code for redirecting the password
    storage in a keyring (GNOME or KWallet) simply wasn’t called in that
    case.

    The command line tool, even when using the D-Bus server to run the
    operation, had the necessary code active and thus was not affected.
    Now all SyncEvolution components use the same default: use safe
    password storage if either GNOME Keyring or KWallet were enabled
    during compilation, don’t use it if not.

    Fixing this revealed other problems, like not being able to store
    certain passwords that lacked the necessary lookup criteria (like
    syncURL and/or username). To address this, the lookup criteria where
    extended and a new check was added to avoid accidentally removing
    other passwords. As a result, it may be possible that SyncEvolution
    no longer finds passwords that were stored with older versions of
    SyncEvolution. In such a case the passwords must be set again.

  • GNOME: clean up keyring access and require libgnome-keyring >= 2.20

    The updated error messages now always include information about the
    password and libgnome-keyring error texts.

    A workaround is used for the “Error communicating with
    gnome-keyring-daemon” problem that started to appear fairly
    frequently in the automated testing once the keyring was actually
    used. The problem shows up with some additional debug messages:

    Gkr: received an invalid, unencryptable, or non-utf8 secret
    Gkr: call to daemon returned an invalid response: (null).(null)()

    It seems that sometimes setting up a session with GNOME keyring
    fails such that all further communication leads to decoding problem.

    There is an internal method to reset the session, but it cannot be
    called directly. As a workaround, fake the death of the GNOME
    keyring daemon and thus trigger a reconnect when retrying the GNOME
    keyring access. This is done by sending a D-Bus message, which will
    also affect other clients of GNOME keyring, but hopefully without
    user-visible effects.

  • config: enhanced password handling

    It is possible to configure a plain username/password combination
    once in SyncEvolution and then use references to it in other
    configurations, instead of having to set (and update) the
    credentials in different places. This is useful in particular with
    WebDAV, where credentials had to be repeated several times (target
    config, in each database when used as part of SyncML) or when using
    a service which requires several configs (Google via SyncML and
    CalDAV).

    To use this, create a sync config for a normal peer or a dedicated
    config just for the credentials, with “username/password/syncURL”
    set. The “syncURL” must be set to something identifying the peer if
    GNOME Keyring is used for the password storage.

    Then set “username”, “databaseUser” and “proxyUser” properties to
    “id:” and all read and write access
    to those properties will be redirected by SyncEvolution into that
    other configuration. This even works in the GTK UI.

    For user names which contain colons, the new “user:” format
    must be used. Strings without colons are assumed to be normal user
    names, so most old configurations should continue to work.

  • signon: new backend using libgsignond-glib + libaccounts-glib

    The code works with gSSO (https://01.org/gsso). With some tweaks to
    the configure check and some ifdefs it probably could be made to work
    with Ubuntu Online Accounts.

    The code depends on an account accessible via libaccounts-glib which
    has a provider and and (optionally) services enabled for that
    provider. It is not necessary that the account already has a signon
    identity ID, the backend will create that for the provider (and thus
    shared between all services) if necessary.

    Therefore it is possible to use the ag-tool to create and enable the
    account and services. Provider and service templates are in the next
    commit.

  • WebDAV: support OAuth2

    If given an authentication configuration which can handle OAuth2,
    then OAuth2 is used instead of plain username/password
    authentication.

  • WebDAV: support Google CardDAV, break Yahoo

    Google CardDAV has one peculiarity: it renames new contacts during PUT without
    returning the new path to the client. See also
    http://lists.calconnect.org/pipermail/caldeveloper-l/2013-July/000524.html

    SyncEvolution already had a workaround for that (PROPGET on old path, extract
    new path from response) which happened to work. This workaround was originally
    added for Yahoo, which sometimes merges contacts into existing ones. In
    contrast to Yahoo, Google really seems to create new items.

    Without some server specific hacks, the client cannot tell what happened.
    Because Google is currently supported and Yahoo is not, let’s change the
    hard-coded behavior to “renamed items are new”.

  • WebDAV: started testing with owndrive.com = OwnCloud

  • GOA: get OAuth2 tokens out of GNOME Online Accounts

    “username = goa:…” selects an account in GOA and retrieves the
    OAuth2 token from that.

    The implementation uses the GOA D-Bus API directly, because our C++
    D-Bus bindings are easier to use and this avoids an additional library
    dependency.

  • PIM: fix UID usage in sync.py example

    Using the underscore in the UID has been wrong all along, it only
    happened to work because UID sanity checking was missing. After adding
    it, the example broke.

    Now simply remove the colon. It makes the UID less readable, but it
    doesn’t have to be, and ensures that file names and database names
    contain the UID as-is.

  • PIM: if busy, don’t shut down

    While there are sessions pending or active, the server should not shut down.
    It did that while executing a long-running PIM Manager SyncPeer() operations,
    by default after 10 minutes.

    This was not a problem elsewhere because other operations are associated with
    a client, whose presence also prevents shutdowns. Perhaps PIM Manager should
    also track the caller and treat it like a client.

  • PBAP: do not end Bluez5 transfer prematurely

    A transfer was marked as finished prematurely when encountering the
    “active” Status value, which can happen for longer transfers.

  • PBAP: add support for obexd 0.48

    obexd 0.48 is almost the same as obexd 0.47, except that it dropped
    the SetFilter and SetFormat methods in favor of passing a Bluex 5-style
    filter parameter to PullAll.

    SyncEvolution now supports 4, in words, four different obexd
    APIs. Sigh.

  • updated tests

Upgrading from releases <= 1.3.99.4:

If the value of “username/databaseUser/proxyUser” contains a colon,
the “user:” prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like “id:”, “goa:”, “signon:”, etc.).

The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.

Upgrading from release 1.2.x:

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution –configure \
syncFormat=text/x-vcard \
mobical addressbook

The Funambol template explicitly enables usage of the
“refresh-from-server” sync mode to avoid getting throttled with 417
‘retry later’ errors. The same must be added to existing configs
manually:
syncevolution –configure \
enableRefreshSync=TRUE \
funambol

Upgrading from releases before 1.2:

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.

SyncEvolution 1.3.99.4 released

The focus of this development snapshot is enhanced performance of
syncing. With EDS, contacts get added, updated or loaded with batch
operations, which led to 4x runtime improvements when importing PBAP
address book for the first time. Removing unnecessary work from any
following PBAP sync resulted in a 6x improvement.

These improvements also benefit non-PBAP syncing and could in theory
work with any SyncML peer. In practice, batching of items is currently
limited to SyncEvolution as peer.

The PBAP backend itself was rewritten such that data gets transferred
from a phone in parallel to processing the already transferred
data. The effect is that on a sufficiently fast system, a sync takes
about the same time as downloading all contacts. To get the text-only
part of the contacts even faster, PBAP syncing can be done such that
it first syncs the text-only parts (without removing existing photos),
then in a second round adds or modifies photos. The PIM Manager uses
this incremental mode by default, in the command line it can be chose
with the SYNCEVOLUTION_PBAP_SYNC env variable.

The HTTP server became better at handling message resends when the
server is slow with processing a message. The server is able to keep a
sync session alive while loading the initial data set by sending
acknowledgement replies before the client times out.

Guido Günther provided some patches addressing problems when compiling
SyncEvolution for Maemo.

Details:

  • sync: less verbose output, shorter runtime

    For each incoming change, one INFO line with “received x[/out of y]”
    was printed, immediately followed by another line with total counts
    “added x, updated y, removed z”. For each outgoing change, a “sent
    x[/out of y]” was printed.

    In addition, these changes were forwarded to the D-Bus server where a
    “percent complete” was calculated and broadcasted to clients. All of
    that caused a very high overhead for every single change, even if the
    actual logging was off. The syncevo-dbus-server was constantly
    consuming CPU time during a sync when it should have been mostly idle.

    To avoid this overhead, the updated received/sent numbers that come
    from the Synthesis engine are now cached and only processed when done
    with a SyncML message or some other event happens (whatever happens
    first).

    To keep the implementation simple, the “added x, updated y, removed z”
    information is ignored completely and no longer appears in the output.

  • HTTP server: handle message resends

    If a client gave up waiting for the server’s response and resent its message
    while the server was still processing the message, syncing failed with
    “protocol error: already processing a message” raised by the
    syncevo-dbus-server because it wasn’t prepared to handle that situation.

    The right place to handle this is inside the syncevo-http-server, because it
    depends on the protocol (HTTP in this case) whether resending is valid or
    not. It handles that now by tracking the message that is currently in
    processing and matching it against each new message. If it matches, the new
    request replaces the obsolete one without sending the message again to
    syncevo-dbus-server. When syncevo-dbus-server replies to the old message, the
    reply is used to finish the newer request.

  • PBAP: incremental sync ((FDO #59551)[https://bugs.freedesktop.org/show_bug.cgi?id=59551])

    Depending on the SYNCEVOLUTION_PBAP_SYNC env variable, syncing reads
    all properties as configured (”all”), excludes photos (”text”) or
    first text, then all (”incremental”).

    When excluding photos, only known properties get requested. This
    avoids issues with phones which reject the request when enabling
    properties via the bit flags. This also helps with
    “databaseFormat=^PHOTO”.

  • PIM: use incremental sync for PBAP by default ((FDO #59551)[https://bugs.freedesktop.org/show_bug.cgi?id=59551])

    When doing a PBAP sync, PIM manager asks the D-Bus sync helper to set
    its SYNCEVOLUTION_PBAP_SYNC to “incremental”. If the env variable
    is already set, it does not get overwritten, which allows overriding
    this default.

  • PIM: set debug level in peer configs via env variable

    Typically the peer configs get created from scratch, in particular
    when testing with testpim.py. In that case the log level cannot be set
    in advance and doing it via the D-Bus API is also not supported.
    Therefore, for debugging, use SYNCEVOLUTION_LOGLEVEL= to create
    peers with a specific log level.

  • PIM: include pim-manager-api.txt in source distro ((FDO #62516)[https://bugs.freedesktop.org/show_bug.cgi?id=62516])

    The text file must be listed explicitly to be included by “make dist”.

  • PIM: “full name” -> “fullname” fix in documentation ((FDO #62515)[https://bugs.freedesktop.org/show_bug.cgi?id=62515])

    Make the documentation match the code. A single word without
    space makes more sense, so let’s go with what the code already
    used.

  • PIM: enhanced searching (search part of (FDO #64177)[https://bugs.freedesktop.org/show_bug.cgi?id=64177])

    Search terms now also include ‘is/contains/begins-with/ends-with’
    and they can be combined with ‘and’ and ‘or’, also recursively.

  • PIM: Pinyin sorting for zh languages (part of (FDO #64173)[https://bugs.freedesktop.org/show_bug.cgi?id=64173])

    Full interleaving of Pinyin transliterations of Chinese names with
    Western names can be done by doing an explicit Pinyin transliteration
    as part of computing the sort keys.

    This is done using ICU’s Transliteration(”Han-Latin”), which we have
    to call directly because boost::locale does not expose that API.

    We hard-code this behavior for all “zh” languages (as identified by
    boost::locale), because by default, ICU would sort Pinyin separately
    from Western names when using the “pinyin” collation.

  • PIM: new return value for SyncPeer(), new SyncProgress signal ((FDO #63417)[https://bugs.freedesktop.org/show_bug.cgi?id=63417])

    The SyncPeer() result is derived from the sync statistics. To have
    them available, the “sync done” signal must include the SyncReport.

    Start and end of a sync could already be detected; “modified” signals
    while a sync runs depends on a new signal inside the SyncContext when
    switching from one cycle to the next and at the end of the last one.

  • PIM: allow removal of data together with database removal (part of (FDO #64835)[https://bugs.freedesktop.org/show_bug.cgi?id=64835])

    There is a difference in EDS between removing the database definition
    from the ESourceRegistry (which makes the data unaccessible via EDS)
    and removing the actual database. EDS itself only removes the definition
    and leaves the data around to be garbage-collected eventually. This is
    not what we want for the PIM Manager API; the API makes a stronger
    guarantee that data is really gone.

    Fixed by introducing a new mode flag for the deleteDatabase() method
    and deleting the directory of the source directly in the EDS backend,
    if requested by the caller.

    The syncevolution command line tool will use the default mode and thus
    keep the data around, while the PIM Manager forces the removal of
    data.

  • EDS: create new databases by cloning the builtin ones ((FDO #64176)[https://bugs.freedesktop.org/show_bug.cgi?id=64176])

    Instead of hard-coding a specific “Backend Summary Setup” in
    SyncEvolution, copy the config of the system database. That way
    special flags (like the desired “Backend Summary Setup” for local
    address books) can be set on a system-wide basis and without having to
    modify or configure SyncEvolution.

    Because EDS has no APIs to clone an ESource or turn a .source file
    into a new ESource, SyncEvolution has to resort to manipulating and
    creating the keyfile directly.

  • EDS contacts: update PHOTO+GEO during slow sync, avoid rewriting PHOTO file

    If PHOTO and/or GEO were the only modified properties during a slow
    sync, the updated item was not written into local storage because
    they were marked as compare=”never” = “not relevant”.

    For PHOTO this was intentional in the sample config, with the
    rationale that local storages often don’t store the data exactly as
    requested. When that happens, comparing the data would lead to
    unnecessary writes. But EDS and probably all other local SyncEvolution
    storages (KDE, file) store the photo exactly as requested, so not
    considering changes had the undesirable effect of not always writing
    new photo data.

    For GEO, ignoring it was accidental.

  • EDS contacts: avoid unnecessary DB writes during slow sync

    Traditionally, contacts were modified shortly before writing into EDS
    to match with Evolution expectations (must have N, only one CELL TEL,
    VOICE flag must be set). During a slow sync, the engine compare the
    modified contacts with the unmodified, incoming one. This led to
    mismatches and/or merge operations which end up not changing anything
    in the DB because the only difference would be removed again before
    writing.

  • EDS contacts: read-ahead cache

    Performance is improved by requesting multiple contacts at once and
    overlapping reading with processing. On a fast system (SSD, CPU fast
    enough to not be the limiting factor), testpim.py’s testSync takes 8
    seconds for a “match” sync where 1000 contacts get loaded and compared
    against the same set of contacts. Read-ahead with only 1 contact per
    query speeds that up to 6.7s due to overlapping IO and
    processing. Read-ahead with the default 50 contacts per query takes
    5.5s. It does not get much faster with larger queries.

  • command line: execute –export and –print-items while the source is still reading

    Instead of reading all item IDs, then iterating over them, process
    each new ID as soon as it is available. With sources that support
    incremental reading (only the PBAP source at the moment) that provides
    output sooner and is a bit more memory efficient.

  • WebDAV: avoid segfault during collection lookup

    Avoid referencing pathProps->second when the set of paths that
    PROPFINDs returns is empty. Apparently this can happen in combination
    with Calypso.

  • engine: prevent timeouts in HTTP server mode

    HTTP SyncML clients give up after a certain timeout (SyncEvolution
    after RetryDuration = 5 minutes by default, Nokia e51 after 15
    minutes) when the server fails to respond.

    This can happen with SyncEvolution as server when it uses a slow
    storage with many items, for example via WebDAV. In the case of slow
    session startup, multithreading is now used to run the storage
    initializing in parallel to sending regular “keep-alive” SyncML
    replies to the client.

    By default, these replies are sent every 2 minutes. This can be
    configured with another extensions of the SyncMLVersion property:
    SyncMLVersion = REQUESTMAXTIME=5m

    Other modes do not use multithreading by default, but it can be
    enabled by setting REQUESTMAXTIME explicitly. It can be disabled
    by setting the time to zero.

    The new feature depends on a libsynthesis with multithreading enabled
    and glib >= 2.32.0, which is necessary to make SyncEvolution itself
    thread-safe. With an older glib, multithreading is disabled, but can
    be enabled as a stop-gap measure by setting REQUESTMAXTIME explicitly.

  • Various testing and stability enhancements. SyncEvolution had to
    be made thread-safe for the HTTP timeout prevention.

Source, Installation, Further information

Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is the git repositories.

i386, lpia and amd64 binaries for Debian-based distributions are
available via the “unstable” syncevolution.org repository. Add the
following entry to your /apt/source.list:

Then install “syncevolution-evolution”, “syncevolution-kde” and/or
“syncevolution-activesync”.

These binaries include the “sync-ui” GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for “syncevolution-activesync” which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.

After installation, follow the getting started steps. More specific HOWTOs can be found in the Wiki.