Skip to content

{ Monthly Archives } October 2011

State of the union, version 1.2

With SyncEvolution 1.2 released and work on 1.3 under way it is a good time to take a small break and reflect on the state of the SyncEvolution project.

Features

This is meant to give an idea of what can be done with SyncEvolution already. For a more complete list of features, see the documentation about backends, frontends and engine.

This section introduces the key ideas, some of the supported protocols + backends and how they work in combination with specific peers.

Data Synchronization

The main purpose of SyncEvolution is the synchronization of Personal Information Management (PIM) data (contacts, events, tasks, notes). This intentionally does not include email and messages in general because those are sufficiently different to require different solutions. SyncEvolution provides real synchronization of PIM data:

  • Items (= contact, event, …) may have different properties and representations on both sides of a sync (heterogeneous environment).
  • Items can be modified at any time, even while offline (more than just online access to a server).
  • Also works between devices, without involving a server.
  • A database (= a set of items) can be synchronized to one or more peers (other devices, servers). For example, the same address book on a desktop can be synchronized with a mobile phone and a laptop.
  • SyncEvolution supports arbitrary number of databases. However, many peers (in particular SyncML servers) are more limited and only provide access to one addressbook, calendar, etc.

For each peer, SyncEvolution needs to remember the previous state so that the next sync can be limited to only those items which have changed. The underlying assumption is that there is no cycle in the connections between multiple sync peers. For example, “phone <-> laptop, laptop <-> server, phone <-> server” is bad because a new item created on the phone will go to the laptop, from there to the server and then come back as new item to the phone, resulting in a duplicate which then repeats the process ad infinitum. Each SyncEvolution instance only knows the peers it is set up to talk with and thus cannot detect the problem.

To break this cycle reliably, each item must have a unique ID that is assigned to it once when the item gets created. For ad-hoc synchronization to work without constraints on the topology, this ID must be supported by all involved peers. Unfortunately, calendar and contact data either doesn’t have such an ID or it is not supported. Without such an ID, each request to add a new item must be checked against all existing items based on some key properties to find out whether it is a duplicate. This is slow and error-prone (”Is ‘Doe, John” the same person as ‘John Doe’, or is one the father and the other a child or namesake?”).

TODOad-hoc synchronization between SyncEvolution instances”:

  • Teach Evolution Data Server to create and preserve a unique ID for contacts (the vCard 3.0 UID gets overwritten at the moment).
  • Improve the SyncEvolution<->SyncEvolution sync such that
    • it uses the unique ID to speed up sync between peers which sync against each other for the first time and
    • checks the ID to find duplicates in following syncs.

TODOad-hoc synchronization between arbitrary peers”:

  • Do the duplicate detection based on item content for each add request.

Testing

SyncEvolution has an extensive test suite which is run regularly (used to be nightly, currently triggered manually). It contains unit tests for various aspects of SyncEvolution and does real interoperability testing with different SyncML, CalDAV and CardDAV servers. See the SyncEvolution 1.2 test report for an example. More CalDAV/CardDAV servers were added in the 1.3 branch.

Keeping this testing going and analyzing/reporting problems is an on-going activity. It is needed to avoid regressions and achieve higher quality in the releases, which are built as part of these test runs. There are additional ideas.

TODO “improve nightly testing”

SyncML + Engine

The Synthesis engine which provides SyncML is one of the best in the industry. It has extensive support for data modeling/conversion and supports suspending a session and resuming it later.

The same engine is also used to synchronize between two backends internally. This is how synchronization was added for CalDAV/CardDAV, protocols which themselves only provide online access. This works reasonably well, but there are also quite a few limitations.

TODOlibvxx = refactoring of the Synthesis code base

  • make data conversion available outside of a sync session
  • decouple sync engine from SyncML and SyncML message encoding/decoding
  • support items which are a set of items: important for CalDAV + ActiveSync, because those combine all VEVENTs with the same UID in one item; certain transformations cannot be done in the engine at the moment because they depend on access to all related items at once and that is not how they are handled at the moment
  • more flexible session handling: instead sending changes in one direction, then back and then stopping, allow the session to continue until both sides are in sync; required for CalDAV where storing an event might lead to further changes that have to be sent back

TODOpush sync“:

  • react to local or remote changes immediately (instead of polling at long intervals) and/or
  • make sync sessions without changes more efficient (in particular when polling)

TODObetter credential handling

  • When creating multiple configurations which need the same credentials (Google CalDAV and SyncML, for example), the username/password needs to be set separately. Replace with a mechanism where both configs only contain a pointer to shared credentials.
  • Also support an external system component which does the authentication without ever returning username/password to SyncEvolution. Depends on having such a component.

This could be used to use a CalDAV/CardDAV server as backend for a SyncML server.

TODOuse CalDAV/CardDAV inside SyncML server“:

  • In the backend, check databaseuser/password before falling back to the context’s username/password.
  • Support shared credentials, to avoid having to configure them in each source.

At the moment there are situations where SyncEvolution cannot determine what the right resolution for a failed synchronization is. It has a built-in backup mechanism and can ask the user for assistance, but ultimately it would be better to not bother the user. This can be achieved in some cases by simplifying the problem:

  • The peer (most likely a service on the Internet) must be able to store all data.
  • There is a separate local database for each database on the peer.

In that case it is acceptable to wipe out the local data and restart with the data stored on the peer.

TODOautomatic error recovery

  • Implement the necessary policy in SyncEvolution (”peer wins”).
  • Check whether this interferes with libfolks (local IDs will change, data added by libfolks to a contact might get lost).

CalDAV/CardDAV

Works reasonable well and passes automated testing against a variety of servers (Apple Calendar Server, DAViCal, Google Calendar, Yahoo). But there are some know limitations, like meeting invitations being sent by Evolution and the CalDAV server.

TODObetter support for meeting invitations

  • Find a way to suppress sending of meeting invitations on the CalDAV server when Evolution already sent one, or
  • suppress sending of meeting invitations in Evolution for calendars which are mirrored in a CalDAV server.
  • When storing a meeting and/or the server does not return an ETag, retrieve the possibly modified item from the server and store the modified item locally. Right now the more possibly automatically modified data on the server gets ignored. Depends on the engine improvements mentioned above.

TODOhandle concurrent changes

  • Use ETags to avoid modifying more recent data on the server.

TODOCalDAV attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them and perhaps even support the more efficient “managed attachments” that are currently being discussed by the CalConnect consortium.

TODOWebDAV: use sync extension

  • Each sync session must list the entire collection (= retrieve path names and ETag) to determine new and modified items. This can be done more efficiently by using the sync extension defined and implemented in the Apple Calendar Server.

TODOCalDAV tasks = VTODO

  • The CalDAV backend is limited to exchanging VEVENTs. It might also not handle collections well (or at all) that contain both VEVENTs and VTODOs.

ActiveSync

This was not included in SyncEvolution 1.2 although it was already quite usable. Work on it is still going on. The main issue with ActiveSync is the limited data model specified as part of the protocol.

Contacts are only allowed to have a fixed number of certain phone numbers and addresses, which is a limitation that neither Evolution nor Google Contacts have. There is no good way to handle these limitations except educating the user about them, or enforcing the same constraints locally by modifying the app which creates and modifies contacts.

The ActiveSync calendar format does not support detached recurrences properly (”you are invited to a specific instance of a meeting series”), although Exchange internally does.

TODO “finish initial ActiveSync support”

  • detached recurrences without parent:
    • investigate receiving multiple detached recurrences in multiple items with the same UID in each (the Exchange workaround for the ActiveSync calendar format limitation): might break SyncEvolution
    • implement the “stand-alone detached recurrence” support (either do it like Exchange does or better, create a fake parent event)
  • test concurrent item changes while a sync runs (supposed to work, but without a test case it is hard to be sure)

[updated] TODO “ActiveSync performance improvement”

  • Writing changes to the server is done one change at a time. Could be improved considerably by batching changes. Depends on core engine improvements.

TODOActiveSync push sync

  • enhance activesyncd and SyncEvolution (see engine above) to react to server-side changes with minimal delay

TODOActiveSync calendar attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them. Not sure how to do it efficiently.

Google Calendar

Overall Google Calendar syncing works well with CalDAV (I’m using it myself) as long as one does all meeting scheduling in the Google web interface. But there are some known issues, most of them on the server side:

Google Contacts

Works via SyncML. Google’s support for SyncML is very incomplete (many properties not supported, for example birthday).

TODO “better Google Contacts support”

Apple Calendar Server, DAViCal

Done via CalDAV/CardDAV. No know issues with these peers.

Yahoo Calendar + Contacts

Works well, when it works; unfortunately the number of requests per 24 hour period is so limited that the tests cannot complete without running into a 503 “Service Unavailable” error.

TODOYahoo token authentication

  • Currently SyncEvolutions uses normal HTTP authentication. Yahoo also supports another, token based authentication mechanism for approved apps. Get SyncEvolution approved and implement that other authentication mechanism. Hopefully that’ll avoid the 503 error.

Phones

SyncEvolution can synchronize against phones if those phones support SyncML via Bluetooth. This is a common feature among older feature phones but most (all?) Android phones and iPhones don’t support it. Testing of this feature is limited and there is no list of phones which are guaranteed to work.

[updated] Synchronization with Android/iOS is possible by installing a third-party SyncML client (like the ones from Synthesis) and configuring the phones to use a SyncEvolution HTTP SyncML server.

TODOcontact sync via PBAP

  • Write a backend using PBAP, the only (?) protocol supported by Android and iOS for address book access. Not very good for real syncing, but at least one-way sync should be possible.

GTK UI

The GTK “sync UI” is the main user interface for SyncEvolution in MeeGo and Linux in general. It is included in the source and binary distribution archives. It supports configuring syncing against SyncML services (extensible via configuration templates) and phones which support SyncML. Configuring the latter is integrated into the GNOME Bluetooth applet, which invokes the sync UI. The UI also has “emergency recovery” support which allows the user to restore from the automatic backup and/or choose between different recovery operations after a failed sync:

  • continue with local data
  • continue with remote data
  • try to merge both (”slow sync”)

TODOport to GTK3

  • The sync UI is currently using GTK2. It needs to be ported to GTK3. The goal is to put the GTK2 version into maintenance and continue with the GTK3 version.

TODOconfigure local sync, choose databases

  • CalDAV/CardDAV and ActiveSync cannot be configured yet with the UI. In this context it becomes important to let the user choose local and remote databases. Would be useful to have, although there is a certain overlap with the Evolution integration.

Evolution

This happens to be the main backend for storing data locally. Other backends could be supported just as well (there’s nothing technical which favors Evolution) if there were developers motivated enough to implement and test them. KDE/Akonadi has come a long way, but seems less active (see KDE/Akonadi and Community below).

TODO “Evolution backend improvements”

TODOintegrate SyncEvolution into Evolution

  • Evolution typically supports offline read access with some of its backends. It does not support write access. SyncEvolution addresses that, but needs to be configured and invoked separately. It would be nice to have seamless and transparent syncing from inside Evolution. ActiveSync is getting integrated like that at the moment -> extend that to CalDAV/CardDAV?

File backend

There is a generic file backend with a 1:1 mapping between a single item and a file in a directory. The format of the local data can be configured. vCalendar 1.0, iCalendar 2.0, vCard 2.1/3.0 and plain text notes are supported already.

TODOiCalendar 2.0 .ics file

  • Some people synchronize an .ics file by pointing the Evolution backend to it (no longer works after EDS removed support for the file:// URI) or manipulating the ~/.evolution/calendar data (a hack which only works when being very careful). Write a SyncEvolution backend which reads/writes an iCalendar 2.0 file using libical.

KDE/Akonadi

The Akonadi backend is available in the source code. Support for KWallet instead of GNOME keyring is also there. A KDE GUI is in development.

TODOtest and release binaries

  • nightly builds needs to be reconfigured to enable building the KDE support
  • nightly testing needs to include KDE backends, both for unit testing the backends and real combinations with peers; it depends on the KDE developers to do something if issues are found in those tests (if any are found)

Manipulating PIM data

SyncEvolution also is a very capable tool for manipulating databases via the command line. It can list, import/export and delete items. Converting between formats (for example, mirror Evolution contacts as files in vCard 2.1 format) is possible by setting up synchronization with a file backend as peer.

TODO “improve item manipulation”

Community

Lots of users. The feedback from users is often very helpful for improving the software.

Not so many developers, though, except those paid to work on SyncEvolution. Ove Kaaven (port+UI for Maemo/N900 and MeeGo Harmattan/N9/N950) and Frederik Elwert (Genesis UI) are the notable exceptions. Sascha Peilicke/Dinesh/Rohan Garg have done some work for KDE/Akonadi, but are not very active in the SyncEvolution project itself.

I can only speculate about the reasons for the lack of external contributors:

  • The number of people who care about PIM sync and storage is small to start with. It’s not one of those sexy areas that gets a lot of attention, although it is arguably very important.
  • OpenSync still seems to be considered “the” open source sync solution, despite not having a stable release available or even in sight anytime soon.
  • Perhaps developers also get the impression that problems will be solved anyway, without having to get involved. That is true for some aspects, but definitely not for all.
  • SyncEvolution development has a certain learning curve (although Ove and Franz Knipp managed to write their backends with very little assistance) and often happens at a rapid pace, which makes it hard for people to contribute small improvements.
  • SyncEvolution was focused on Evolution and SyncML initially, which might make it seem too limited in scope for some use cases. But the scope has already increased and will increase further, so now would be a good time to check it out again.

Many of the TODOs above will not get worked on unless some external developer picks up the mantle and contributes patches. I’d be more than happy to help someone get started. If there is interest, I can also tag bugs in Bugzilla as “easy fixes”. Right now I am not doing that because it would create additional work which would not be justified if there is no-one interested – I know, chicken and egg…

State of the union, version 1.2

With SyncEvolution 1.2 released and work on 1.3 under way it is a good time to take a small break and reflect on the state of the SyncEvolution project.

Features

This is meant to give an idea of what can be done with SyncEvolution already. For a more complete list of features, see the documentation about backends, frontends and engine.

This section introduces the key ideas, some of the supported protocols + backends and how they work in combination with specific peers.

Data Synchronization

The main purpose of SyncEvolution is the synchronization of Personal Information Management (PIM) data (contacts, events, tasks, notes). This intentionally does not include email and messages in general because those are sufficiently different to require different solutions. SyncEvolution provides real synchronization of PIM data:

  • Items (= contact, event, …) may have different properties and representations on both sides of a sync (heterogeneous environment).
  • Items can be modified at any time, even while offline (more than just online access to a server).
  • Also works between devices, without involving a server.
  • A database (= a set of items) can be synchronized to one or more peers (other devices, servers). For example, the same address book on a desktop can be synchronized with a mobile phone and a laptop.
  • SyncEvolution supports arbitrary number of databases. However, many peers (in particular SyncML servers) are more limited and only provide access to one addressbook, calendar, etc.

For each peer, SyncEvolution needs to remember the previous state so that the next sync can be limited to only those items which have changed. The underlying assumption is that there is no cycle in the connections between multiple sync peers. For example, “phone <-> laptop, laptop <-> server, phone <-> server” is bad because a new item created on the phone will go to the laptop, from there to the server and then come back as new item to the phone, resulting in a duplicate which then repeats the process ad infinitum. Each SyncEvolution instance only knows the peers it is set up to talk with and thus cannot detect the problem.

To break this cycle reliably, each item must have a unique ID that is assigned to it once when the item gets created. For ad-hoc synchronization to work without constraints on the topology, this ID must be supported by all involved peers. Unfortunately, calendar and contact data either doesn’t have such an ID or it is not supported. Without such an ID, each request to add a new item must be checked against all existing items based on some key properties to find out whether it is a duplicate. This is slow and error-prone (”Is ‘Doe, John” the same person as ‘John Doe’, or is one the father and the other a child or namesake?”).

TODOad-hoc synchronization between SyncEvolution instances”:

  • Teach Evolution Data Server to create and preserve a unique ID for contacts (the vCard 3.0 UID gets overwritten at the moment).
  • Improve the SyncEvolution<->SyncEvolution sync such that
    • it uses the unique ID to speed up sync between peers which sync against each other for the first time and
    • checks the ID to find duplicates in following syncs.

TODOad-hoc synchronization between arbitrary peers”:

  • Do the duplicate detection based on item content for each add request.

Testing

SyncEvolution has an extensive test suite which is run regularly (used to be nightly, currently triggered manually). It contains unit tests for various aspects of SyncEvolution and does real interoperability testing with different SyncML, CalDAV and CardDAV servers. See the SyncEvolution 1.2 test report for an example. More CalDAV/CardDAV servers were added in the 1.3 branch.

Keeping this testing going and analyzing/reporting problems is an on-going activity. It is needed to avoid regressions and achieve higher quality in the releases, which are built as part of these test runs. There are additional ideas.

TODO “improve nightly testing”

SyncML + Engine

The Synthesis engine which provides SyncML is one of the best in the industry. It has extensive support for data modeling/conversion and supports suspending a session and resuming it later.

The same engine is also used to synchronize between two backends internally. This is how synchronization was added for CalDAV/CardDAV, protocols which themselves only provide online access. This works reasonably well, but there are also quite a few limitations.

TODOlibvxx = refactoring of the Synthesis code base

  • make data conversion available outside of a sync session
  • decouple sync engine from SyncML and SyncML message encoding/decoding
  • support items which are a set of items: important for CalDAV + ActiveSync, because those combine all VEVENTs with the same UID in one item; certain transformations cannot be done in the engine at the moment because they depend on access to all related items at once and that is not how they are handled at the moment
  • more flexible session handling: instead sending changes in one direction, then back and then stopping, allow the session to continue until both sides are in sync; required for CalDAV where storing an event might lead to further changes that have to be sent back

TODOpush sync“:

  • react to local or remote changes immediately (instead of polling at long intervals) and/or
  • make sync sessions without changes more efficient (in particular when polling)

TODObetter credential handling

  • When creating multiple configurations which need the same credentials (Google CalDAV and SyncML, for example), the username/password needs to be set separately. Replace with a mechanism where both configs only contain a pointer to shared credentials.
  • Also support an external system component which does the authentication without ever returning username/password to SyncEvolution. Depends on having such a component.

This could be used to use a CalDAV/CardDAV server as backend for a SyncML server.

TODOuse CalDAV/CardDAV inside SyncML server“:

  • In the backend, check databaseuser/password before falling back to the context’s username/password.
  • Support shared credentials, to avoid having to configure them in each source.

At the moment there are situations where SyncEvolution cannot determine what the right resolution for a failed synchronization is. It has a built-in backup mechanism and can ask the user for assistance, but ultimately it would be better to not bother the user. This can be achieved in some cases by simplifying the problem:

  • The peer (most likely a service on the Internet) must be able to store all data.
  • There is a separate local database for each database on the peer.

In that case it is acceptable to wipe out the local data and restart with the data stored on the peer.

TODOautomatic error recovery

  • Implement the necessary policy in SyncEvolution (”peer wins”).
  • Check whether this interferes with libfolks (local IDs will change, data added by libfolks to a contact might get lost).

CalDAV/CardDAV

Works reasonable well and passes automated testing against a variety of servers (Apple Calendar Server, DAViCal, Google Calendar, Yahoo). But there are some know limitations, like meeting invitations being sent by Evolution and the CalDAV server.

TODObetter support for meeting invitations

  • Find a way to suppress sending of meeting invitations on the CalDAV server when Evolution already sent one, or
  • suppress sending of meeting invitations in Evolution for calendars which are mirrored in a CalDAV server.
  • When storing a meeting and/or the server does not return an ETag, retrieve the possibly modified item from the server and store the modified item locally. Right now the more possibly automatically modified data on the server gets ignored. Depends on the engine improvements mentioned above.

TODOhandle concurrent changes

  • Use ETags to avoid modifying more recent data on the server.

TODOCalDAV attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them and perhaps even support the more efficient “managed attachments” that are currently being discussed by the CalConnect consortium.

TODOWebDAV: use sync extension

  • Each sync session must list the entire collection (= retrieve path names and ETag) to determine new and modified items. This can be done more efficiently by using the sync extension defined and implemented in the Apple Calendar Server.

TODOCalDAV tasks = VTODO

  • The CalDAV backend is limited to exchanging VEVENTs. It might also not handle collections well (or at all) that contain both VEVENTs and VTODOs.

ActiveSync

This was not included in SyncEvolution 1.2 although it was already quite usable. Work on it is still going on. The main issue with ActiveSync is the limited data model specified as part of the protocol.

Contacts are only allowed to have a fixed number of certain phone numbers and addresses, which is a limitation that neither Evolution nor Google Contacts have. There is no good way to handle these limitations except educating the user about them, or enforcing the same constraints locally by modifying the app which creates and modifies contacts.

The ActiveSync calendar format does not support detached recurrences properly (”you are invited to a specific instance of a meeting series”), although Exchange internally does.

TODO “finish initial ActiveSync support”

  • detached recurrences without parent:
    • investigate receiving multiple detached recurrences in multiple items with the same UID in each (the Exchange workaround for the ActiveSync calendar format limitation): might break SyncEvolution
    • implement the “stand-alone detached recurrence” support (either do it like Exchange does or better, create a fake parent event)
  • test concurrent item changes while a sync runs (supposed to work, but without a test case it is hard to be sure)

[updated] TODO “ActiveSync performance improvement”

  • Writing changes to the server is done one change at a time. Could be improved considerably by batching changes. Depends on core engine improvements.

TODOActiveSync push sync

  • enhance activesyncd and SyncEvolution (see engine above) to react to server-side changes with minimal delay

TODOActiveSync calendar attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them. Not sure how to do it efficiently.

Google Calendar

Overall Google Calendar syncing works well with CalDAV (I’m using it myself) as long as one does all meeting scheduling in the Google web interface. But there are some known issues, most of them on the server side:

Google Contacts

Works via SyncML. Google’s support for SyncML is very incomplete (many properties not supported, for example birthday).

TODO “better Google Contacts support”

Apple Calendar Server, DAViCal

Done via CalDAV/CardDAV. No know issues with these peers.

Yahoo Calendar + Contacts

Works well, when it works; unfortunately the number of requests per 24 hour period is so limited that the tests cannot complete without running into a 503 “Service Unavailable” error.

TODOYahoo token authentication

  • Currently SyncEvolutions uses normal HTTP authentication. Yahoo also supports another, token based authentication mechanism for approved apps. Get SyncEvolution approved and implement that other authentication mechanism. Hopefully that’ll avoid the 503 error.

Phones

SyncEvolution can synchronize against phones if those phones support SyncML via Bluetooth. This is a common feature among older feature phones but most (all?) Android phones and iPhones don’t support it. Testing of this feature is limited and there is no list of phones which are guaranteed to work.

[updated] Synchronization with Android/iOS is possible by installing a third-party SyncML client (like the ones from Synthesis) and configuring the phones to use a SyncEvolution HTTP SyncML server.

TODOcontact sync via PBAP

  • Write a backend using PBAP, the only (?) protocol supported by Android and iOS for address book access. Not very good for real syncing, but at least one-way sync should be possible.

GTK UI

The GTK “sync UI” is the main user interface for SyncEvolution in MeeGo and Linux in general. It is included in the source and binary distribution archives. It supports configuring syncing against SyncML services (extensible via configuration templates) and phones which support SyncML. Configuring the latter is integrated into the GNOME Bluetooth applet, which invokes the sync UI. The UI also has “emergency recovery” support which allows the user to restore from the automatic backup and/or choose between different recovery operations after a failed sync:

  • continue with local data
  • continue with remote data
  • try to merge both (”slow sync”)

TODOport to GTK3

  • The sync UI is currently using GTK2. It needs to be ported to GTK3. The goal is to put the GTK2 version into maintenance and continue with the GTK3 version.

TODOconfigure local sync, choose databases

  • CalDAV/CardDAV and ActiveSync cannot be configured yet with the UI. In this context it becomes important to let the user choose local and remote databases. Would be useful to have, although there is a certain overlap with the Evolution integration.

Evolution

This happens to be the main backend for storing data locally. Other backends could be supported just as well (there’s nothing technical which favors Evolution) if there were developers motivated enough to implement and test them. KDE/Akonadi has come a long way, but seems less active (see KDE/Akonadi and Community below).

TODO “Evolution backend improvements”

TODOintegrate SyncEvolution into Evolution

  • Evolution typically supports offline read access with some of its backends. It does not support write access. SyncEvolution addresses that, but needs to be configured and invoked separately. It would be nice to have seamless and transparent syncing from inside Evolution. ActiveSync is getting integrated like that at the moment -> extend that to CalDAV/CardDAV?

File backend

There is a generic file backend with a 1:1 mapping between a single item and a file in a directory. The format of the local data can be configured. vCalendar 1.0, iCalendar 2.0, vCard 2.1/3.0 and plain text notes are supported already.

TODOiCalendar 2.0 .ics file

  • Some people synchronize an .ics file by pointing the Evolution backend to it (no longer works after EDS removed support for the file:// URI) or manipulating the ~/.evolution/calendar data (a hack which only works when being very careful). Write a SyncEvolution backend which reads/writes an iCalendar 2.0 file using libical.

KDE/Akonadi

The Akonadi backend is available in the source code. Support for KWallet instead of GNOME keyring is also there. A KDE GUI is in development.

TODOtest and release binaries

  • nightly builds needs to be reconfigured to enable building the KDE support
  • nightly testing needs to include KDE backends, both for unit testing the backends and real combinations with peers; it depends on the KDE developers to do something if issues are found in those tests (if any are found)

Manipulating PIM data

SyncEvolution also is a very capable tool for manipulating databases via the command line. It can list, import/export and delete items. Converting between formats (for example, mirror Evolution contacts as files in vCard 2.1 format) is possible by setting up synchronization with a file backend as peer.

TODO “improve item manipulation”

Community

Lots of users. The feedback from users is often very helpful for improving the software.

Not so many developers, though, except those paid to work on SyncEvolution. Ove Kaaven (port+UI for Maemo/N900 and MeeGo Harmattan/N9/N950) and Frederik Elwert (Genesis UI) are the notable exceptions. Sascha Peilicke/Dinesh/Rohan Garg have done some work for KDE/Akonadi, but are not very active in the SyncEvolution project itself.

I can only speculate about the reasons for the lack of external contributors:

  • The number of people who care about PIM sync and storage is small to start with. It’s not one of those sexy areas that gets a lot of attention, although it is arguably very important.
  • OpenSync still seems to be considered “the” open source sync solution, despite not having a stable release available or even in sight anytime soon.
  • Perhaps developers also get the impression that problems will be solved anyway, without having to get involved. That is true for some aspects, but definitely not for all.
  • SyncEvolution development has a certain learning curve (although Ove and Franz Knipp managed to write their backends with very little assistance) and often happens at a rapid pace, which makes it hard for people to contribute small improvements.
  • SyncEvolution was focused on Evolution and SyncML initially, which might make it seem too limited in scope for some use cases. But the scope has already increased and will increase further, so now would be a good time to check it out again.

Many of the TODOs above will not get worked on unless some external developer picks up the mantle and contributes patches. I’d be more than happy to help someone get started. If there is interest, I can also tag bugs in Bugzilla as “easy fixes”. Right now I am not doing that because it would create additional work which would not be justified if there is no-one interested – I know, chicken and egg…

State of the union, version 1.2

With SyncEvolution 1.2 released and work on 1.3 under way it is a good time to take a small break and reflect on the state of the SyncEvolution project.

Features

This is meant to give an idea of what can be done with SyncEvolution already. For a more complete list of features, see the documentation about backends, frontends and engine.

This section introduces the key ideas, some of the supported protocols + backends and how they work in combination with specific peers.

Data Synchronization

The main purpose of SyncEvolution is the synchronization of Personal Information Management (PIM) data (contacts, events, tasks, notes). This intentionally does not include email and messages in general because those are sufficiently different to require different solutions. SyncEvolution provides real synchronization of PIM data:

  • Items (= contact, event, …) may have different properties and representations on both sides of a sync (heterogeneous environment).
  • Items can be modified at any time, even while offline (more than just online access to a server).
  • Also works between devices, without involving a server.
  • A database (= a set of items) can be synchronized to one or more peers (other devices, servers). For example, the same address book on a desktop can be synchronized with a mobile phone and a laptop.
  • SyncEvolution supports arbitrary number of databases. However, many peers (in particular SyncML servers) are more limited and only provide access to one addressbook, calendar, etc.

For each peer, SyncEvolution needs to remember the previous state so that the next sync can be limited to only those items which have changed. The underlying assumption is that there is no cycle in the connections between multiple sync peers. For example, “phone <-> laptop, laptop <-> server, phone <-> server” is bad because a new item created on the phone will go to the laptop, from there to the server and then come back as new item to the phone, resulting in a duplicate which then repeats the process ad infinitum. Each SyncEvolution instance only knows the peers it is set up to talk with and thus cannot detect the problem.

To break this cycle reliably, each item must have a unique ID that is assigned to it once when the item gets created. For ad-hoc synchronization to work without constraints on the topology, this ID must be supported by all involved peers. Unfortunately, calendar and contact data either doesn’t have such an ID or it is not supported. Without such an ID, each request to add a new item must be checked against all existing items based on some key properties to find out whether it is a duplicate. This is slow and error-prone (”Is ‘Doe, John” the same person as ‘John Doe’, or is one the father and the other a child or namesake?”).

TODOad-hoc synchronization between SyncEvolution instances”:

  • Teach Evolution Data Server to create and preserve a unique ID for contacts (the vCard 3.0 UID gets overwritten at the moment).
  • Improve the SyncEvolution<->SyncEvolution sync such that
    • it uses the unique ID to speed up sync between peers which sync against each other for the first time and
    • checks the ID to find duplicates in following syncs.

TODOad-hoc synchronization between arbitrary peers”:

  • Do the duplicate detection based on item content for each add request.

Testing

SyncEvolution has an extensive test suite which is run regularly (used to be nightly, currently triggered manually). It contains unit tests for various aspects of SyncEvolution and does real interoperability testing with different SyncML, CalDAV and CardDAV servers. See the SyncEvolution 1.2 test report for an example. More CalDAV/CardDAV servers were added in the 1.3 branch.

Keeping this testing going and analyzing/reporting problems is an on-going activity. It is needed to avoid regressions and achieve higher quality in the releases, which are built as part of these test runs. There are additional ideas.

TODO “improve nightly testing”

SyncML + Engine

The Synthesis engine which provides SyncML is one of the best in the industry. It has extensive support for data modeling/conversion and supports suspending a session and resuming it later.

The same engine is also used to synchronize between two backends internally. This is how synchronization was added for CalDAV/CardDAV, protocols which themselves only provide online access. This works reasonably well, but there are also quite a few limitations.

TODOlibvxx = refactoring of the Synthesis code base

  • make data conversion available outside of a sync session
  • decouple sync engine from SyncML and SyncML message encoding/decoding
  • support items which are a set of items: important for CalDAV + ActiveSync, because those combine all VEVENTs with the same UID in one item; certain transformations cannot be done in the engine at the moment because they depend on access to all related items at once and that is not how they are handled at the moment
  • more flexible session handling: instead sending changes in one direction, then back and then stopping, allow the session to continue until both sides are in sync; required for CalDAV where storing an event might lead to further changes that have to be sent back

TODOpush sync“:

  • react to local or remote changes immediately (instead of polling at long intervals) and/or
  • make sync sessions without changes more efficient (in particular when polling)

TODObetter credential handling

  • When creating multiple configurations which need the same credentials (Google CalDAV and SyncML, for example), the username/password needs to be set separately. Replace with a mechanism where both configs only contain a pointer to shared credentials.
  • Also support an external system component which does the authentication without ever returning username/password to SyncEvolution. Depends on having such a component.

This could be used to use a CalDAV/CardDAV server as backend for a SyncML server.

TODOuse CalDAV/CardDAV inside SyncML server“:

  • In the backend, check databaseuser/password before falling back to the context’s username/password.
  • Support shared credentials, to avoid having to configure them in each source.

At the moment there are situations where SyncEvolution cannot determine what the right resolution for a failed synchronization is. It has a built-in backup mechanism and can ask the user for assistance, but ultimately it would be better to not bother the user. This can be achieved in some cases by simplifying the problem:

  • The peer (most likely a service on the Internet) must be able to store all data.
  • There is a separate local database for each database on the peer.

In that case it is acceptable to wipe out the local data and restart with the data stored on the peer.

TODOautomatic error recovery

  • Implement the necessary policy in SyncEvolution (”peer wins”).
  • Check whether this interferes with libfolks (local IDs will change, data added by libfolks to a contact might get lost).

CalDAV/CardDAV

Works reasonable well and passes automated testing against a variety of servers (Apple Calendar Server, DAViCal, Google Calendar, Yahoo). But there are some know limitations, like meeting invitations being sent by Evolution and the CalDAV server.

TODObetter support for meeting invitations

  • Find a way to suppress sending of meeting invitations on the CalDAV server when Evolution already sent one, or
  • suppress sending of meeting invitations in Evolution for calendars which are mirrored in a CalDAV server.
  • When storing a meeting and/or the server does not return an ETag, retrieve the possibly modified item from the server and store the modified item locally. Right now the more possibly automatically modified data on the server gets ignored. Depends on the engine improvements mentioned above.

TODOhandle concurrent changes

  • Use ETags to avoid modifying more recent data on the server.

TODOCalDAV attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them and perhaps even support the more efficient “managed attachments” that are currently being discussed by the CalConnect consortium.

TODOWebDAV: use sync extension

  • Each sync session must list the entire collection (= retrieve path names and ETag) to determine new and modified items. This can be done more efficiently by using the sync extension defined and implemented in the Apple Calendar Server.

TODOCalDAV tasks = VTODO

  • The CalDAV backend is limited to exchanging VEVENTs. It might also not handle collections well (or at all) that contain both VEVENTs and VTODOs.

ActiveSync

This was not included in SyncEvolution 1.2 although it was already quite usable. Work on it is still going on. The main issue with ActiveSync is the limited data model specified as part of the protocol.

Contacts are only allowed to have a fixed number of certain phone numbers and addresses, which is a limitation that neither Evolution nor Google Contacts have. There is no good way to handle these limitations except educating the user about them, or enforcing the same constraints locally by modifying the app which creates and modifies contacts.

The ActiveSync calendar format does not support detached recurrences properly (”you are invited to a specific instance of a meeting series”), although Exchange internally does.

TODO “finish initial ActiveSync support”

  • detached recurrences without parent:
    • investigate receiving multiple detached recurrences in multiple items with the same UID in each (the Exchange workaround for the ActiveSync calendar format limitation): might break SyncEvolution
    • implement the “stand-alone detached recurrence” support (either do it like Exchange does or better, create a fake parent event)
  • test concurrent item changes while a sync runs (supposed to work, but without a test case it is hard to be sure)

[updated] TODO “ActiveSync performance improvement”

  • Writing changes to the server is done one change at a time. Could be improved considerably by batching changes. Depends on core engine improvements.

TODOActiveSync push sync

  • enhance activesyncd and SyncEvolution (see engine above) to react to server-side changes with minimal delay

TODOActiveSync calendar attachments

  • At the moment, attachments are not supported at all and even may get lost. Need to preserve them. Not sure how to do it efficiently.

Google Calendar

Overall Google Calendar syncing works well with CalDAV (I’m using it myself) as long as one does all meeting scheduling in the Google web interface. But there are some known issues, most of them on the server side:

Google Contacts

Works via SyncML. Google’s support for SyncML is very incomplete (many properties not supported, for example birthday).

TODO “better Google Contacts support”

Apple Calendar Server, DAViCal

Done via CalDAV/CardDAV. No know issues with these peers.

Yahoo Calendar + Contacts

Works well, when it works; unfortunately the number of requests per 24 hour period is so limited that the tests cannot complete without running into a 503 “Service Unavailable” error.

TODOYahoo token authentication

  • Currently SyncEvolutions uses normal HTTP authentication. Yahoo also supports another, token based authentication mechanism for approved apps. Get SyncEvolution approved and implement that other authentication mechanism. Hopefully that’ll avoid the 503 error.

Phones

SyncEvolution can synchronize against phones if those phones support SyncML via Bluetooth. This is a common feature among older feature phones but most (all?) Android phones and iPhones don’t support it. Testing of this feature is limited and there is no list of phones which are guaranteed to work.

[updated] Synchronization with Android/iOS is possible by installing a third-party SyncML client (like the ones from Synthesis) and configuring the phones to use a SyncEvolution HTTP SyncML server.

TODOcontact sync via PBAP

  • Write a backend using PBAP, the only (?) protocol supported by Android and iOS for address book access. Not very good for real syncing, but at least one-way sync should be possible.

GTK UI

The GTK “sync UI” is the main user interface for SyncEvolution in MeeGo and Linux in general. It is included in the source and binary distribution archives. It supports configuring syncing against SyncML services (extensible via configuration templates) and phones which support SyncML. Configuring the latter is integrated into the GNOME Bluetooth applet, which invokes the sync UI. The UI also has “emergency recovery” support which allows the user to restore from the automatic backup and/or choose between different recovery operations after a failed sync:

  • continue with local data
  • continue with remote data
  • try to merge both (”slow sync”)

TODOport to GTK3

  • The sync UI is currently using GTK2. It needs to be ported to GTK3. The goal is to put the GTK2 version into maintenance and continue with the GTK3 version.

TODOconfigure local sync, choose databases

  • CalDAV/CardDAV and ActiveSync cannot be configured yet with the UI. In this context it becomes important to let the user choose local and remote databases. Would be useful to have, although there is a certain overlap with the Evolution integration.

Evolution

This happens to be the main backend for storing data locally. Other backends could be supported just as well (there’s nothing technical which favors Evolution) if there were developers motivated enough to implement and test them. KDE/Akonadi has come a long way, but seems less active (see KDE/Akonadi and Community below).

TODO “Evolution backend improvements”

TODOintegrate SyncEvolution into Evolution

  • Evolution typically supports offline read access with some of its backends. It does not support write access. SyncEvolution addresses that, but needs to be configured and invoked separately. It would be nice to have seamless and transparent syncing from inside Evolution. ActiveSync is getting integrated like that at the moment -> extend that to CalDAV/CardDAV?

File backend

There is a generic file backend with a 1:1 mapping between a single item and a file in a directory. The format of the local data can be configured. vCalendar 1.0, iCalendar 2.0, vCard 2.1/3.0 and plain text notes are supported already.

TODOiCalendar 2.0 .ics file

  • Some people synchronize an .ics file by pointing the Evolution backend to it (no longer works after EDS removed support for the file:// URI) or manipulating the ~/.evolution/calendar data (a hack which only works when being very careful). Write a SyncEvolution backend which reads/writes an iCalendar 2.0 file using libical.

KDE/Akonadi

The Akonadi backend is available in the source code. Support for KWallet instead of GNOME keyring is also there. A KDE GUI is in development.

TODOtest and release binaries

  • nightly builds needs to be reconfigured to enable building the KDE support
  • nightly testing needs to include KDE backends, both for unit testing the backends and real combinations with peers; it depends on the KDE developers to do something if issues are found in those tests (if any are found)

Manipulating PIM data

SyncEvolution also is a very capable tool for manipulating databases via the command line. It can list, import/export and delete items. Converting between formats (for example, mirror Evolution contacts as files in vCard 2.1 format) is possible by setting up synchronization with a file backend as peer.

TODO “improve item manipulation”

Community

Lots of users. The feedback from users is often very helpful for improving the software.

Not so many developers, though, except those paid to work on SyncEvolution. Ove Kaaven (port+UI for Maemo/N900 and MeeGo Harmattan/N9/N950) and Frederik Elwert (Genesis UI) are the notable exceptions. Sascha Peilicke/Dinesh/Rohan Garg have done some work for KDE/Akonadi, but are not very active in the SyncEvolution project itself.

I can only speculate about the reasons for the lack of external contributors:

  • The number of people who care about PIM sync and storage is small to start with. It’s not one of those sexy areas that gets a lot of attention, although it is arguably very important.
  • OpenSync still seems to be considered “the” open source sync solution, despite not having a stable release available or even in sight anytime soon.
  • Perhaps developers also get the impression that problems will be solved anyway, without having to get involved. That is true for some aspects, but definitely not for all.
  • SyncEvolution development has a certain learning curve (although Ove and Franz Knipp managed to write their backends with very little assistance) and often happens at a rapid pace, which makes it hard for people to contribute small improvements.
  • SyncEvolution was focused on Evolution and SyncML initially, which might make it seem too limited in scope for some use cases. But the scope has already increased and will increase further, so now would be a good time to check it out again.

Many of the TODOs above will not get worked on unless some external developer picks up the mantle and contributes patches. I’d be more than happy to help someone get started. If there is interest, I can also tag bugs in Bugzilla as “easy fixes”. Right now I am not doing that because it would create additional work which would not be justified if there is no-one interested – I know, chicken and egg…

SyncEvolution 1.2 released

The major new feature of the 1.2 release is support for non-SyncML
protocols in general and CalDAV/CardDAV in particular. ActiveSync
support is in development and will be in 1.3. These protocols are
implemented as backends which are combined with other backends by
SyncEvolution in a so called “local sync”. The GTK sync-ui does not
yet support configuring non-SyncML protocols. See the README.rst and
man page
for more information on how to use the new feature via the
command line.

Properties not supported by SyncML servers can now be preserved
locally in two-way synchronization (BMC #15030). This depends on
information about what properties a SyncML server supports (”CtCap”),
which is typically not provided by servers. SyncEvolution contains a
copy of that information for Google Contacts (BMC #15029).

Akonadi backend and KWallet support were merged. They are not included
yet in syncevolution.org binaries. To use them compile from source.

The configuration format was updated to solve a conceptual problem
inherited with the legacy property names: the “type” property had
multiple, sometimes conflicting roles. For example, setting the
preferred data format for sync with one peer might have changed the
backend selection for some other peer (BMC #1023). Now
“backend/databaseFormat/syncFormat/forceSyncFormat” replace
“type”. “type” is still accepted by the command line as alias.

Upgrading from releases before 1.2

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Release 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Other changes

  • Using the –sync-property and –source-property command line options is
    optional, just specifying the property assignment is enough.

  • syncevo-http-server was enhanced considerably. See http://syncevolution.org/wiki/http-server-howto

  • support NetworkManager API >= 0.9 (BMC #19470)

  • syncevolution.org binaries: now compatible with Debian Testing/libnotify.so.4 (BMC #22668)

    libnotify is not linked directly into syncevo-dbus-server in the
    syncevolution.org binaries. Instead libnotify.so.1 till .so.4
    (current Debian Testing) are opened opened dynamically and the
    necessary functions are looked up via dlsym(). Not finding the
    libraries or the functions silently disables this notification
    mechanism.

  • Sync mode is recorded when running in SyncML server mode (BMC #2786).

  • syncevo-dbus-server automatically stops when some of its libraries
    are updated and restarts if auto-syncing is on (BMC #14955).

  • Added code for Buteo, mKCal and QtContacts in MeeGo.

    Buteo and mKCal were removed again from MeeGo, so the code
    is obsolete. The QtContacts backend may be still be useful
    to access items via that API, but for syncing on MeeGo
    the normal EDS backend is used since MeeGo reverted back
    to EDS as PIM storage.

  • “databasePassword” source property: lookup failure in keyring (BMC #22937)

    The databasePassword also wasn’t looked up at all when doing item operations
    via the command line.

    When configuring sources for an HTTP server, the config name typically
    is just the context (@foo). When using the config in the HTTP server,
    the config name is the peer inside that context (client@foo). Because
    the GNOME keyring lookup keys for the “databasePassword” (more
    specifically, the object name) contained the full config name which
    was different in both cases, looking up the saved password failed.

    The solution is to normalize the config name (to accomodate for
    different ways of spelling it) and use only the context, with @ as
    before. This will break existing setups where the object name in the
    keyring (incorrectly) includes the full config name. In that case just
    configure the source again to set the password anew.

  • Evolution Calendar: fixed detached recurrence support (BMC #22940)

    When manipulating a meeting series with more than one detached
    recurrence certain sequences of operations could incorrectly fail
    with “UID already exists”.

  • iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940)

    EXDATE has a VALUE parameter, which wasn’t defined in the XML
    profile. Didn’t seem to matter at all in practice, but wasn’t
    standard-compliant.

  • GTK sync-ui: wrap sync service descriptions (BMC #7199)

    Descriptions of different sync services are not fully visible unless
    word-wrapping gets enabled.

  • CalDAV/CardDAV + local storage: avoid empty properties

    The main motivation for this change is that a recent Apple Calendar
    server rejects vCards with empty BDAY property. Another reason is that
    keeping the data as small as possible is desirable by itself.

    Sending an empty property serves as a hint for the peer that the
    property is supported. This is not necessary when storing an item in a
    backend. Therefore this commit disables empty properties for all
    backends which do not themselves set the m_backendRule Synthesis info
    value.

  • Google Contacts: ensure that first/middle/name are set when storing in EDS (BMC #20864)

    Evolution and the MeeGo UX assume that first/middle/last name are set.
    That is not the case when a contact is created in the Google Contacts
    web interface. Such contacts are sent by Google without the N
    property.

    SyncEvolution now tries to recreate the name components from the FN
    string, by splitting at word boundaries and assuming ”
    ” or “, ” format. Obviously this
    heuristic fails for some locales.

  • Evolution Calendar: fixed error handling for broken TZIDs

  • Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414)

    Passing invalid UTF-8 strings into libecal caused glib to
    abort syncevo-dbus-server.

  • auto sync: show all failed syncs except for temporary network errors (BMC #21888)

    Notifications were meant to be shown for all errors except temporary
    ones. This has never been implemented correctly since the feature was
    introduced: instead of hiding known temporary errors, all errors except
    500 (fatal error) were suppressed.

  • vCard: inline local photo data (BMC #19661)

    Some platforms (Maemo, MeeGo) store photos in separate files. Now SyncEvolution
    efficiently includes that photo data in the generated vCard right before sending
    it to a peer; previously it sent a useless local file:// URI. The Maemo port
    has a less efficient workaround for that which now should be obsolete.

  • syncevo-dbus-server: online status wrong without Network Manager or ConnMan (BMC #21543)

    When neither Network Manager nor ConnMan are running, network presence was “not
    online”. This prevented running automatic syncs.

For developers:

  • modified backend API

    • ClientTestConfig modernized
    • InsertItemResult::m_merged turned from boolean to enum
  • testing and compilation changes; for example, the minimum version of
    libsynthesis is now checked at configure time instead of failing at
    runtime due to missing features in the Synthesis engine

SyncEvolution 1.1.99.7 -> 1.2, 13.10.2011

Some more bug fixes and testing improvements.

  • fixed potential invalid memory access in add<->add conflict handling
  • fixed memory leak in workaround for EDS bug
  • CalDAV/CardDAV: handle ETags without quotation marks (eGroupware)
  • updated README: warning about sync direction moved to –sync option

Source, Installation, Further information

Source snapshots are in
http://downloads.syncevolution.org/syncevolution/sources

i386, lpia and amd64 binaries for Debian-based distributions are available via the “stable” syncevolution.org repository. Add the following entry to your /apt/source.list, then install “syncevolution-evolution”:

These binaries include the “sync-ui” GTK GUI and were compiled for Ubuntu 8.04 LTS (Hardy). Older distributions like Debian 4.0 (Etch) can no longer be supported with precompiled binaries because of missing libraries, but the source still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in the download directories. In contrast to 0.8.x archives, the 1.0 .tar.gz archives have to be unpacked and the content must be moved to /usr, because several files would not be found otherwise.

After installation, follow the getting started steps.

SyncEvolution 1.2 released

The major new feature of the 1.2 release is support for non-SyncML
protocols in general and CalDAV/CardDAV in particular. ActiveSync
support is in development and will be in 1.3. These protocols are
implemented as backends which are combined with other backends by
SyncEvolution in a so called “local sync”. The GTK sync-ui does not
yet support configuring non-SyncML protocols. See the README.rst and
man page
for more information on how to use the new feature via the
command line.

Properties not supported by SyncML servers can now be preserved
locally in two-way synchronization (BMC #15030). This depends on
information about what properties a SyncML server supports (”CtCap”),
which is typically not provided by servers. SyncEvolution contains a
copy of that information for Google Contacts (BMC #15029).

Akonadi backend and KWallet support were merged. They are not included
yet in syncevolution.org binaries. To use them compile from source.

The configuration format was updated to solve a conceptual problem
inherited with the legacy property names: the “type” property had
multiple, sometimes conflicting roles. For example, setting the
preferred data format for sync with one peer might have changed the
backend selection for some other peer (BMC #1023). Now
“backend/databaseFormat/syncFormat/forceSyncFormat” replace
“type”. “type” is still accepted by the command line as alias.

Upgrading from releases before 1.2

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Release 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Other changes

  • Using the –sync-property and –source-property command line options is
    optional, just specifying the property assignment is enough.

  • syncevo-http-server was enhanced considerably. See http://syncevolution.org/wiki/http-server-howto

  • support NetworkManager API >= 0.9 (BMC #19470)

  • syncevolution.org binaries: now compatible with Debian Testing/libnotify.so.4 (BMC #22668)

    libnotify is not linked directly into syncevo-dbus-server in the
    syncevolution.org binaries. Instead libnotify.so.1 till .so.4
    (current Debian Testing) are opened opened dynamically and the
    necessary functions are looked up via dlsym(). Not finding the
    libraries or the functions silently disables this notification
    mechanism.

  • Sync mode is recorded when running in SyncML server mode (BMC #2786).

  • syncevo-dbus-server automatically stops when some of its libraries
    are updated and restarts if auto-syncing is on (BMC #14955).

  • Added code for Buteo, mKCal and QtContacts in MeeGo.

    Buteo and mKCal were removed again from MeeGo, so the code
    is obsolete. The QtContacts backend may be still be useful
    to access items via that API, but for syncing on MeeGo
    the normal EDS backend is used since MeeGo reverted back
    to EDS as PIM storage.

  • “databasePassword” source property: lookup failure in keyring (BMC #22937)

    The databasePassword also wasn’t looked up at all when doing item operations
    via the command line.

    When configuring sources for an HTTP server, the config name typically
    is just the context (@foo). When using the config in the HTTP server,
    the config name is the peer inside that context (client@foo). Because
    the GNOME keyring lookup keys for the “databasePassword” (more
    specifically, the object name) contained the full config name which
    was different in both cases, looking up the saved password failed.

    The solution is to normalize the config name (to accomodate for
    different ways of spelling it) and use only the context, with @ as
    before. This will break existing setups where the object name in the
    keyring (incorrectly) includes the full config name. In that case just
    configure the source again to set the password anew.

  • Evolution Calendar: fixed detached recurrence support (BMC #22940)

    When manipulating a meeting series with more than one detached
    recurrence certain sequences of operations could incorrectly fail
    with “UID already exists”.

  • iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940)

    EXDATE has a VALUE parameter, which wasn’t defined in the XML
    profile. Didn’t seem to matter at all in practice, but wasn’t
    standard-compliant.

  • GTK sync-ui: wrap sync service descriptions (BMC #7199)

    Descriptions of different sync services are not fully visible unless
    word-wrapping gets enabled.

  • CalDAV/CardDAV + local storage: avoid empty properties

    The main motivation for this change is that a recent Apple Calendar
    server rejects vCards with empty BDAY property. Another reason is that
    keeping the data as small as possible is desirable by itself.

    Sending an empty property serves as a hint for the peer that the
    property is supported. This is not necessary when storing an item in a
    backend. Therefore this commit disables empty properties for all
    backends which do not themselves set the m_backendRule Synthesis info
    value.

  • Google Contacts: ensure that first/middle/name are set when storing in EDS (BMC #20864)

    Evolution and the MeeGo UX assume that first/middle/last name are set.
    That is not the case when a contact is created in the Google Contacts
    web interface. Such contacts are sent by Google without the N
    property.

    SyncEvolution now tries to recreate the name components from the FN
    string, by splitting at word boundaries and assuming ”
    ” or “, ” format. Obviously this
    heuristic fails for some locales.

  • Evolution Calendar: fixed error handling for broken TZIDs

  • Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414)

    Passing invalid UTF-8 strings into libecal caused glib to
    abort syncevo-dbus-server.

  • auto sync: show all failed syncs except for temporary network errors (BMC #21888)

    Notifications were meant to be shown for all errors except temporary
    ones. This has never been implemented correctly since the feature was
    introduced: instead of hiding known temporary errors, all errors except
    500 (fatal error) were suppressed.

  • vCard: inline local photo data (BMC #19661)

    Some platforms (Maemo, MeeGo) store photos in separate files. Now SyncEvolution
    efficiently includes that photo data in the generated vCard right before sending
    it to a peer; previously it sent a useless local file:// URI. The Maemo port
    has a less efficient workaround for that which now should be obsolete.

  • syncevo-dbus-server: online status wrong without Network Manager or ConnMan (BMC #21543)

    When neither Network Manager nor ConnMan are running, network presence was “not
    online”. This prevented running automatic syncs.

For developers:

  • modified backend API

    • ClientTestConfig modernized
    • InsertItemResult::m_merged turned from boolean to enum
  • testing and compilation changes; for example, the minimum version of
    libsynthesis is now checked at configure time instead of failing at
    runtime due to missing features in the Synthesis engine

SyncEvolution 1.1.99.7 -> 1.2, 13.10.2011

Some more bug fixes and testing improvements.

  • fixed potential invalid memory access in add<->add conflict handling
  • fixed memory leak in workaround for EDS bug
  • CalDAV/CardDAV: handle ETags without quotation marks (eGroupware)
  • updated README: warning about sync direction moved to –sync option

Source, Installation, Further information

Source snapshots are in
http://downloads.syncevolution.org/syncevolution/sources

i386, lpia and amd64 binaries for Debian-based distributions are available via the “stable” syncevolution.org repository. Add the following entry to your /apt/source.list, then install “syncevolution-evolution”:

These binaries include the “sync-ui” GTK GUI and were compiled for Ubuntu 8.04 LTS (Hardy). Older distributions like Debian 4.0 (Etch) can no longer be supported with precompiled binaries because of missing libraries, but the source still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in the download directories. In contrast to 0.8.x archives, the 1.0 .tar.gz archives have to be unpacked and the content must be moved to /usr, because several files would not be found otherwise.

After installation, follow the getting started steps.

SyncEvolution 1.2 released

The major new feature of the 1.2 release is support for non-SyncML
protocols in general and CalDAV/CardDAV in particular. ActiveSync
support is in development and will be in 1.3. These protocols are
implemented as backends which are combined with other backends by
SyncEvolution in a so called “local sync”. The GTK sync-ui does not
yet support configuring non-SyncML protocols. See the README.rst and
man page
for more information on how to use the new feature via the
command line.

Properties not supported by SyncML servers can now be preserved
locally in two-way synchronization (BMC #15030). This depends on
information about what properties a SyncML server supports (”CtCap”),
which is typically not provided by servers. SyncEvolution contains a
copy of that information for Google Contacts (BMC #15029).

Akonadi backend and KWallet support were merged. They are not included
yet in syncevolution.org binaries. To use them compile from source.

The configuration format was updated to solve a conceptual problem
inherited with the legacy property names: the “type” property had
multiple, sometimes conflicting roles. For example, setting the
preferred data format for sync with one peer might have changed the
backend selection for some other peer (BMC #1023). Now
“backend/databaseFormat/syncFormat/forceSyncFormat” replace
“type”. “type” is still accepted by the command line as alias.

Upgrading from releases before 1.2

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Release 1.2
automatically migrates configurations. The old configurations
will still be available (see “syncevolution –print-configs”) but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Other changes

  • Using the –sync-property and –source-property command line options is
    optional, just specifying the property assignment is enough.

  • syncevo-http-server was enhanced considerably. See http://syncevolution.org/wiki/http-server-howto

  • support NetworkManager API >= 0.9 (BMC #19470)

  • syncevolution.org binaries: now compatible with Debian Testing/libnotify.so.4 (BMC #22668)

    libnotify is not linked directly into syncevo-dbus-server in the
    syncevolution.org binaries. Instead libnotify.so.1 till .so.4
    (current Debian Testing) are opened opened dynamically and the
    necessary functions are looked up via dlsym(). Not finding the
    libraries or the functions silently disables this notification
    mechanism.

  • Sync mode is recorded when running in SyncML server mode (BMC #2786).

  • syncevo-dbus-server automatically stops when some of its libraries
    are updated and restarts if auto-syncing is on (BMC #14955).

  • Added code for Buteo, mKCal and QtContacts in MeeGo.

    Buteo and mKCal were removed again from MeeGo, so the code
    is obsolete. The QtContacts backend may be still be useful
    to access items via that API, but for syncing on MeeGo
    the normal EDS backend is used since MeeGo reverted back
    to EDS as PIM storage.

  • “databasePassword” source property: lookup failure in keyring (BMC #22937)

    The databasePassword also wasn’t looked up at all when doing item operations
    via the command line.

    When configuring sources for an HTTP server, the config name typically
    is just the context (@foo). When using the config in the HTTP server,
    the config name is the peer inside that context (client@foo). Because
    the GNOME keyring lookup keys for the “databasePassword” (more
    specifically, the object name) contained the full config name which
    was different in both cases, looking up the saved password failed.

    The solution is to normalize the config name (to accomodate for
    different ways of spelling it) and use only the context, with @ as
    before. This will break existing setups where the object name in the
    keyring (incorrectly) includes the full config name. In that case just
    configure the source again to set the password anew.

  • Evolution Calendar: fixed detached recurrence support (BMC #22940)

    When manipulating a meeting series with more than one detached
    recurrence certain sequences of operations could incorrectly fail
    with “UID already exists”.

  • iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940)

    EXDATE has a VALUE parameter, which wasn’t defined in the XML
    profile. Didn’t seem to matter at all in practice, but wasn’t
    standard-compliant.

  • GTK sync-ui: wrap sync service descriptions (BMC #7199)

    Descriptions of different sync services are not fully visible unless
    word-wrapping gets enabled.

  • CalDAV/CardDAV + local storage: avoid empty properties

    The main motivation for this change is that a recent Apple Calendar
    server rejects vCards with empty BDAY property. Another reason is that
    keeping the data as small as possible is desirable by itself.

    Sending an empty property serves as a hint for the peer that the
    property is supported. This is not necessary when storing an item in a
    backend. Therefore this commit disables empty properties for all
    backends which do not themselves set the m_backendRule Synthesis info
    value.

  • Google Contacts: ensure that first/middle/name are set when storing in EDS (BMC #20864)

    Evolution and the MeeGo UX assume that first/middle/last name are set.
    That is not the case when a contact is created in the Google Contacts
    web interface. Such contacts are sent by Google without the N
    property.

    SyncEvolution now tries to recreate the name components from the FN
    string, by splitting at word boundaries and assuming ”
    ” or “, ” format. Obviously this
    heuristic fails for some locales.

  • Evolution Calendar: fixed error handling for broken TZIDs

  • Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414)

    Passing invalid UTF-8 strings into libecal caused glib to
    abort syncevo-dbus-server.

  • auto sync: show all failed syncs except for temporary network errors (BMC #21888)

    Notifications were meant to be shown for all errors except temporary
    ones. This has never been implemented correctly since the feature was
    introduced: instead of hiding known temporary errors, all errors except
    500 (fatal error) were suppressed.

  • vCard: inline local photo data (BMC #19661)

    Some platforms (Maemo, MeeGo) store photos in separate files. Now SyncEvolution
    efficiently includes that photo data in the generated vCard right before sending
    it to a peer; previously it sent a useless local file:// URI. The Maemo port
    has a less efficient workaround for that which now should be obsolete.

  • syncevo-dbus-server: online status wrong without Network Manager or ConnMan (BMC #21543)

    When neither Network Manager nor ConnMan are running, network presence was “not
    online”. This prevented running automatic syncs.

For developers:

  • modified backend API

    • ClientTestConfig modernized
    • InsertItemResult::m_merged turned from boolean to enum
  • testing and compilation changes; for example, the minimum version of
    libsynthesis is now checked at configure time instead of failing at
    runtime due to missing features in the Synthesis engine

SyncEvolution 1.1.99.7 -> 1.2, 13.10.2011

Some more bug fixes and testing improvements.

  • fixed potential invalid memory access in add<->add conflict handling
  • fixed memory leak in workaround for EDS bug
  • CalDAV/CardDAV: handle ETags without quotation marks (eGroupware)
  • updated README: warning about sync direction moved to –sync option

Source, Installation, Further information

Source snapshots are in
http://downloads.syncevolution.org/syncevolution/sources

i386, lpia and amd64 binaries for Debian-based distributions are available via the “stable” syncevolution.org repository. Add the following entry to your /apt/source.list, then install “syncevolution-evolution”:

These binaries include the “sync-ui” GTK GUI and were compiled for Ubuntu 8.04 LTS (Hardy). Older distributions like Debian 4.0 (Etch) can no longer be supported with precompiled binaries because of missing libraries, but the source still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in the download directories. In contrast to 0.8.x archives, the 1.0 .tar.gz archives have to be unpacked and the content must be moved to /usr, because several files would not be found otherwise.

After installation, follow the getting started steps.

SyncEvolution + non-recursive Automake

Krzesimir Nowak wrote about his work on converting SyncEvolution from an autotools project with recursive make to non-recursive make. Definitely worth a read for anyone interested in autotools. The current SyncEvolution master branch (post 1.2) uses that new build system.

His conclusion is that the new system is not necessarily easier to understand than the one before (autotools with some preprocessing shell scripts). Partly that’s because SyncEvolution tries to achieve certain things not supported well by autotools (automatically generated version number, avoid listing all files explicitly, backends which can add to the global configure script), partly it is because non-recursive Automake introduces additional constraints (like having to avoid clobbering variables and rules). Helper scripts are still needed, the only difference is how they get called.

Either way, what sold me on the idea of a non-recursive make is that on a machine with many cores, like the SyncEvolution nightly test server, compilation is considerably faster because parallel make can spawn more jobs in parallel. Recursive make often had to wait for the completion of compilation in a sub-directory. If memory serves me right, it was more than twice as fast.

SyncEvolution + non-recursive Automake

Krzesimir Nowak wrote about his work on converting SyncEvolution from an autotools project with recursive make to non-recursive make. Definitely worth a read for anyone interested in autotools. The current SyncEvolution master branch (post 1.2) uses that new build system.

His conclusion is that the new system is not necessarily easier to understand than the one before (autotools with some preprocessing shell scripts). Partly that’s because SyncEvolution tries to achieve certain things not supported well by autotools (automatically generated version number, avoid listing all files explicitly, backends which can add to the global configure script), partly it is because non-recursive Automake introduces additional constraints (like having to avoid clobbering variables and rules). Helper scripts are still needed, the only difference is how they get called.

Either way, what sold me on the idea of a non-recursive make is that on a machine with many cores, like the SyncEvolution nightly test server, compilation is considerably faster because parallel make can spawn more jobs in parallel. Recursive make often had to wait for the completion of compilation in a sub-directory. If memory serves me right, it was more than twice as fast.

SyncEvolution + non-recursive Automake

Krzesimir Nowak wrote about his work on converting SyncEvolution from an autotools project with recursive make to non-recursive make. Definitely worth a read for anyone interested in autotools. The current SyncEvolution master branch (post 1.2) uses that new build system.

His conclusion is that the new system is not necessarily easier to understand than the one before (autotools with some preprocessing shell scripts). Partly that’s because SyncEvolution tries to achieve certain things not supported well by autotools (automatically generated version number, avoid listing all files explicitly, backends which can add to the global configure script), partly it is because non-recursive Automake introduces additional constraints (like having to avoid clobbering variables and rules). Helper scripts are still needed, the only difference is how they get called.

Either way, what sold me on the idea of a non-recursive make is that on a machine with many cores, like the SyncEvolution nightly test server, compilation is considerably faster because parallel make can spawn more jobs in parallel. Recursive make often had to wait for the completion of compilation in a sub-directory. If memory serves me right, it was more than twice as fast.