Therefore it is still as hard as before to add calendar synchronization. In the meantime Funambol has started to integrate the SyncEvolution code into their GUI for the iPhone and will continue to work on it there. I’m happy with that – less work for me that way So I’ll stop working on the iPhone port of SyncEvolution and leave it to Funambol to cover that platform.

• I noticed problems with the background thread which watched the Evolution Data Server and therefore disabled it. This means that SyncEvolution will hang again when EDS dies unexpectedly.
• I fixed an incompatibility of the Funambol C++ client library with the Synthesis server; as it turned out later, this had also been fixed by Funambol on the head revision without updating the release branch. Synchronizing notes with that server now works fine.

As I mentioned before, as part of the work for the iPhone SyncEvolution was first ported to Mac OS X. The 0.7 release on SourceForge now also has a precompiled package for that platform. It should run on PPC and x86, but was only tested on x86. This is considered a proof-of-concept: the packaging is very primitive and unless there is demand I won’t compile future versions for Mac OS X.

Compared to 0.6 there are several reasons to update, so don’t hesitate, get your update in time to unwrap, ahem, unpack it under the Christmas tree…

Because SyncEvolution is a standards-compliant SyncML client, synchronization is not limited to myFUNAMBOL: it also works with other SyncML servers (I regularly test with Synthesis and ScheduleWorld; user’s have had success with eGroupware and more). The GUI only allows to configure the server’s URL and account data. For everything else it uses the configuration files for the “funambol” server. So to change the URI, edit the /var/root/.sync4j/evolution/funambol/spds/sources/addressbook/config.txt file.

The blog also concludes that “As both methods do the same thing I guess the iPhone web app would be the more preferred option”. I personally find it much more useful to have my contacts on the device itself because incoming calls are matched against the stored numbers and then displayed with the name of the person calling me.

Traditionally SyncEvolution did not have many command line options: you could select the server to synchronize against in case you have configured more than one (which is entirely possible). Then you could select a subset of data sources to synchronize. Both is not such a big difference compared to a GUI. The advantages are in the output that you get. The iPhone plugin keeps things simple and just presents success or failure. But SyncEvolution offers much more than that.

Comparison of Database Changes

For me Evolution is the main storage location for my precious contact data. Whenever I sync, I absolutely want to be sure that no unintentional changes are applied to that data. Therefore starting with the very first release SyncEvolution would do a before and after dump of the data and print a comparison of those two dumps. Whenever some unwanted change is made, falling back to the previous state is as easy as reimporting the “before” data dump.

Several people have pointed out that changes applied to the server are equally important and that the output of SyncEvolution about “changes during synchronization” was confusing because it didn’t include the changes on the server. I’m a reasonable person (okay, usually), so I listened to those remarks and added a new feature to SyncEvolution 0.7 pre2: it also does another comparison between the data dumps after the previous sync and before the current one, i.e. this comparison lists the changes applied locally. They will be sent to the server in the next sync. This feature depends on having the previous data dump still around, either in /tmp or (and that is the recommended solution because /tmp is likely to be wiped out during a reboot) in the logdir configured in the syncml/config.txt.

Note that logdir must be set manually, it is not set in the example configs

.

Here is an example:

syncevolution scheduleworld
Local changes to be applied to server during synchronization:
*** vcard30 ***
no changes
*** ical20 ***
no changes
*** itodo20 ***
no changes
*** text ***
                     after last sync | current data
             removed since last sync <
                                     > added since last sync
-------------------------------------------------------------------------------
BEGIN:VCALENDAR                          BEGIN:VCALENDAR
VERSION:2.0                              VERSION:2.0
BEGIN:VJOURNAL                           BEGIN:VJOURNAL
                                     >   SUMMARY:New memo.
                                     >   CATEGORIES:TEST
                                     >   CLASS:PRIVATE
                                     >   DESCRIPTION:New memo.\nThis needs t
                                     >    o be sent to the server.\n
                                     >   END:VJOURNAL
                                     > END:VCALENDAR
-------------------------------------------------------------------------------
                                     > BEGIN:VCALENDAR
                                     > VERSION:2.0
                                     >   BEGIN:VJOURNAL
SUMMARY:Summary Modified                 SUMMARY:Summary Modified
DESCRIPTION:Summary Modified\nBody       DESCRIPTION:Summary Modified\nBody
text                                     text
END:VJOURNAL                             END:VJOURNAL
END:VCALENDAR                            END:VCALENDAR

A completely new memo is listed here as part of the local changes.

-------------------------------------------------------------------------------
15:05:49 GMT +0100 [INFO] - Synchronization URL: http://sync.scheduleworld.com/funambol/ds
15:05:49 GMT +0100 [INFO] - Preparing synchronization of vcard30...
15:05:49 GMT +0100 [INFO] - Preparing synchronization of ical20...
15:05:49 GMT +0100 [INFO] - Preparing synchronization of itodo20...
15:05:49 GMT +0100 [INFO] - Preparing synchronization of text...
15:05:50 GMT +0100 [INFO] - vcard30: sync mode is 'two-way'
15:05:51 GMT +0100 [INFO] - ical20: sync mode is 'two-way'
15:05:52 GMT +0100 [INFO] - itodo20: sync mode is 'two-way'
15:05:52 GMT +0100 [INFO] - text: sync mode is 'two-way'
15:05:52 GMT +0100 [INFO] - text: 20071110T140450Z-5351-727-1-65@gollum: extracting from EV
15:05:54 GMT +0100 [INFO] - vcard30: John Doe, pas-id-4733548A0000000C, EV Doe, John: update
Synchronization successful.

The following table summarizes all changes transmitted during a synchronization. Sometimes changes might
be transmitted which did not modify data (e.g. when making a change and later reverting it, an update would
still be sent).

Changes applied during synchronization:
+-------------------|-------ON CLIENT-------|-------ON SERVER-------|
|                   |   successful / total  |   successful / total  |
|            Source |  NEW  |  MOD  |  DEL  |  NEW  |  MOD  |  DEL  |
+-------------------+-------+-------+-------+-------+-------+-------+
|           vcard30 |  0/0  |  1/1  |  0/0  |  0/0  |  0/0  |  0/0  |
|            ical20 |  0/0  |  0/0  |  0/0  |  0/0  |  0/0  |  0/0  |
|           itodo20 |  0/0  |  0/0  |  0/0  |  0/0  |  0/0  |  0/0  |
|              text |  0/0  |  0/0  |  0/0  |  1/1  |  0/0  |  0/0  |
+-------------------+-------+-------+-------+-------+-------+-------+

The following changes are again based on a comparison of the actual local data:

Changes applied to client during synchronization:
*** vcard30 ***
                           before sync | after sync
                   removed during sync <
                                       > added during sync
-------------------------------------------------------------------------------
BEGIN:VCARD                              BEGIN:VCARD
N:Doe;John                               N:Doe;John
ADR;TYPE=HOME:Test Box #1;;Test Drive    ADR;TYPE=HOME:Test Box #1;;Test Drive
1;Test Village;Lower Test County;123     1;Test Village;Lower Test County;123
45;Testovia                              45;Testovia
ADR;TYPE=HOME:Test Box #3;;Test Drive    ADR;TYPE=HOME:Test Box #3;;Test Drive
3;Test Megacity;Test County;12347;Ne     3;Test Megacity;Test County;12347;Ne
w Testonia                               w Testonia
ADR;TYPE=WORK:Test Box #2;;Test Drive    ADR;TYPE=WORK:Test Box #2;;Test Drive
2;Test Town;Upper Test County;12346;     2;Test Town;Upper Test County;12346;
Old Testovia                             Old Testovia
BDAY:2006-01-08                          BDAY:2006-01-08
CATEGORIES:TEST                          CATEGORIES:TEST
EMAIL:john.doe@home.priv                 EMAIL:john.doe@home.priv
EMAIL:john.doe@other.world               EMAIL:john.doe@other.world
EMAIL:john.doe@work.com                  EMAIL:john.doe@work.com
NICKNAME:user1                           NICKNAME:user1
NOTE:This is a test case which uses a    NOTE:This is a test case which uses a
lmost all Evolution fields.              lmost all Evolution fields.
ORG:Test Inc.;Testing                    ORG:Test Inc.;Testing
ROLE:professional test case              ROLE:professional test case
TEL;TYPE=CAR:car 7                       TEL;TYPE=CAR:car 7
TEL;TYPE=CELL:mobile 3                   TEL;TYPE=CELL:mobile 3
TEL;TYPE=FAX;TYPE=HOME:homefax 5         TEL;TYPE=FAX;TYPE=HOME:homefax 5
TEL;TYPE=FAX;TYPE=WORK:businessfax 4     TEL;TYPE=FAX;TYPE=WORK:businessfax 4
TEL;TYPE=HOME:home 2                     TEL;TYPE=HOME:home 2
TEL;TYPE=PAGER:pager 6                   TEL;TYPE=PAGER:pager 6
TEL;TYPE=PREF:primary 8                  TEL;TYPE=PREF:primary 8
TEL;TYPE=WORK:business 1                 TEL;TYPE=WORK:business 1
TITLE:Manager                          | TITLE:Senior Manager

Only one line was changed: John Doe has been promoted…

URL:http://john.doe.com                 URL:http://john.doe.com
VERSION:3.0                              VERSION:3.0
END:VCARD                                END:VCARD
-------------------------------------------------------------------------------
*** ical20 ***
no changes
*** itodo20 ***
no changes
*** text ***
no changes

New Command Line Options in 0.7 pre2

The check for local changes makes sense even without a sync, for example to figure out whether there are any local changes which would get lost during a “refresh-from-server” sync. Doing such a “refresh-from-server” once before continuing with normal “two-way” synchronization is another example for something which is now possible with new the command line options:

syncevolution
syncevolution [<options>] <server> [<source> ...]
syncevolution --help|-h
syncevolution --version

Options:
  --sync|-s <mode>
    Temporarily synchronize the active sources in that mode. Useful
    for a 'refresh-from-server' or 'refresh-from-client' sync which
    clears all data at one end and copies all items from the other.

  --status|-t
    The changes made to local data since the last synchronization are
    shown without starting a new one. This can be used to see in advance
    whether the local data needs to be synchronized with the server.

  --quiet|-q
    Suppresses most of the normal output during a synchronization. The
    log file still contains all the information.

  --help|-h
    Prints usage information.

  --version
    Prints the SyncEvolution version.

Running Regularly

Like any other command line application, SyncEvolution can be run regularly for example by making it part of a cron job.

When you do that, please choose a long delay between each sync!

It is easy to generate a high load on a SyncML server even without actually exchanging data.

Not available (yet)

Due to limitations in the logging of the SyncML library, the INFO messages which are printed at the end of a sync cannot be printed to the screen at the same time as they are written into the log file. That would be more natural as it provides some feedback while the sync is still running. Even more important would be an estimate how much of the sync is already completed. The SyncML library provides some support for that, it just isn’t used in SyncEvolution yet.

Starting with Evolution 2.12 SyncEvolution crashed the Evolution Data Server when synchronizing calendars.

Once I managed to compile Evolution 2.12 on my Debian Etch desktop and also solved some linking problems, it turned out to be reproducible with the SyncEvolution test suite. Lesson one: test against Evolution development builds regularly.

After Matthew Barnes pointed me in the right direction, stepping through the code in the debugger showed that a change in e_cal_create_object() caused the problem: it failed because the event (identified by its UID) already existed. In contrast to previous versions, Evolution 2.12 removes the offending UID, so when SyncEvolution called e_cal_update_object() that call then failed even worse by crashing the back end due to an unchecked NULL UID pointer. Reconstructing the UID was an easy solution for this, once the reason was found. Lesson two (not really a new one, but always good to remember): don’t depend on unspecified behavior, it can change. If only it was more obvious which behavior is intentional and which isn’t ;-/

SyncEvolution hangs when the Evolution Data Server dies.

When the Evolution Data Server died, SyncEvolution remained stuck in the blocking EDS API calls because they do not return. The Evolution API documents a "backend-died" signal. Trying to catch that signal in SyncEvolution did not work at first; more investigations showed that the signal is only delivered to “normal” GNOME/glib applications which run an event loop. As a command line tool SyncEvolution did not have that and therefore the signal was never delivered. Now a background thread drives the event loop and SyncEvolution detects the problem. Because the API calls still don’t return with an error, all it can do is print an error and abort, but that’s still better than just keeping the user waiting… forever.

SyncEvolution segfault after updating to Ubuntu

Two users reported this. Stack back traces reported by the users pointed to an invalid free() in libcurl, but it never became clear what caused the memory corruption. There is no real solution, only a workaround. A configuration change seemed to help: if you follow an old Ubuntu forum post for compiling SyncEvolution from source (like one of the users did), then the configuration created as described in that forum does not enable a message size limit and the splitting of large objects. This was added in 0.5 and made the default in 0.6. Because upgrading to a more recent Evolution does not update the user’s .sync4j configuration files, these changes must be applied manually – the NEWS file will describe such changes. In this case the spds/syncml/config.txt should contain the following entries:

# The maximum size of each message can be set (maxMsgSize) and the
# server can be told to never sent items larger than a certain
# threshold (maxObjSize). Presumably the server has to truncate or
# skip larger items. Finally the client and server may be given the
# permission to transmit large items in multiple messages (loSupport =
# large object support).
maxMsgSize = 8192
maxObjSize = 500000
loSupport = 1

Note that SyncEvolution 0.7 pre1 is available as .deb in an apt repository that works for Debian and Ubuntu. You’ll only need to compile from source if you want it for 64 bit or some other platform.

Synchronization of photos with iPhone

Sending photos had been tested, but apparently not importing them – as one user reported, it just crashed. To cut a long story short, the API for photo import/export is different on the iPhone than on Mac OS X: the iPhone stores photos in different resolutions so that thumbnails load quicker. The API was extended to support this functionality and takes additional parameter that SyncEvolution did not pass. Now SyncEvolution calls the functions correctly, but it does not yet produce thumbnails when importing photos. If you can, scale down photos before adding them to contacts – it helps to keep the load on SyncML servers small and complete syncs faster. Note that despite the possibility to “move and scale” a photo on the iPhone before setting it, the photo will be stored in its original size! SyncEvolution will send the original size because otherwise the original photo might get lost when reimporting the contact. I try to avoid data loss under all circumstances, but am less sure about this case. If you think that the iPhone client should reduce the amount of network communication by only sending smaller photos, then please let me know.

ScheduleWorld and iPhone: vCard 3.0 import did not map phone numbers correctly

The preferred format of ScheduleWorld is vCard 3.0. Clients should use the card3 URI to get contacts in that format and also send them. The example configs of SyncEvolution 0.7 pre1 had card3 but contacts where always sent as vCard 2.1 because myFUNAMBOL is more happy with that format. Even worse, there was a bug in SyncEvolution’s vCard 3.0 parser which did not correctly detect the type of incoming phone numbers, therefore mapping most of them to “other”. The next release fixes that and also sends contacts as vCard 3.0 when configured for card3 (a hack, but there is no better way to detect the intended format at the moment). After the next update it is recommended to do a “refresh-from-server” sync to fix the broken types. Currently this has to be done by setting “sync = refresh-from-server” in the source’s config.txt, sync, reload the file and set “two-way” again. I’ll try to add a command line option to pre2 which will temporarily override the synchronization mode – I’m just not sure yet whether I should delay the update because of that.

My current goal is to have a stable SyncEvolution 0.7 release out before even considering taking on more work. I believe users deserve stable, tested releases instead of constantly changing “beta” software. 0.7 is not done yet: there has been one report of a crash on the iPhone. On the Evolution side users switching to 2.12 have reported problems. There are also some features and workarounds that I wanted to add during this release cycle.

Beyond that I haven’t decided yet. The biggest problem with calendars, memos and tasks on the iPhone is that there is no documentation how the API to access them works. For contacts at least something could be learned from the Mac OS X documentation although the iPhone was sufficiently different to make the port very tedious. One could try to bypass the API and access the data in the sqlite databases, but most likely this will confuse the iPhone GUI like it did for contacts. Perhaps the upcoming Apple SDK or the Leopard iCal API will provide more information.

Another chunk of work is the encoding and decoding of the data in formats understood by SyncML servers. For calendars and tasks the vCalendar 1.0 or iCalendar 2.0 (if time zone definitions are needed) could be used. Markup for memos could be a problem. It might have to be stripped, lest it confuses other devices.

Users also need to be aware where to report problems and how to install updates: the two components are developed independently and consist of separate binaries. It is just because of limitations in the packaging system on the iPhone that the GUI package contains the SyncEvolution executable instead of depending on the SyncEvolution package.

Fabrizio and I discussed future updates and came up with the following plan:

• the GUI will contain the SyncEvolution executable (as it is now) but under a different name: this way users can install one package and have a working sync application
• a notice during the installation reminds them that installing SyncEvolution is the best way to always have an up-to-date sync engine
• if installed, the GUI invokes the normal SyncEvolution instead of the embedded one
• SyncEvolution will be installable via the Funambol repository in addition to the one on www.estamos.de; currently getting it there is a manual process so the Funambol repository might be a bit behind

Non-authoritative answer:
sync.scheduleworld.com canonical name = ns2.scheduleworld.com.
Name: ns2.scheduleworld.com
Address: 72.51.41.97

This is one of the known problems on the iPhone. The same thing happens when libCURL in SyncEvolution tries to resolve the name. Visting the site in Safari “fixes” the problem for a while, but sooner or later it has a very unwelcome comeback. Strangely enough, it only affects sync.scheduleworld.com and not my.funambol.com.

One user reported that he had the same problem and that pinging the site helped, but I cannot reproduce that. Any other suggestions?

Update 2007-11-28: good news for users of ScheduleWorld – since November 27th ScheduleWorld moved sync.scheduleworld.com off of a CNAME and onto a dedicated A record. This was done to avoid the name server lookup problems on the iPhone. Thanks Mark!

At the beginning god created the iPhone… or so some believe when Steve Jobs announce it. I’m not a follower of that particular religion and usually react allergic to hypes, but when Fabrizio offered to send me an iPhone in return for the promise to port SyncEvolution I couldn’t resist. The goal was to have a solid implementation of address book synchronization. With all my contacts maintained in Evolution on Linux, iTunes isn’t much help in that regard.

After experimenting with directly accessing the contacts in the SQLite database that approach was dropped in favor of using the Addressbook framework:

• the internal database schema is sufficiently complex that reimplementing read/write routines would have been a lot of work and likely led to compatibility issues with the iPhone address book
• changes to the SQLite database were not detected by the iPhone address book or caused it to crash
• using the Address Book API would allow to compile the same source also for Mac OS X synchronization, a nice extra because there doesn’t seem to be a SyncML client for Mac OS X

After all, iPhone runs parts of Mac OS X and that API is documented, so how different can it be on the iPhone? Unfortunately the answer turned out to be: quite a lot. From analyzing the libraries on the iPhone it was clear that all of the functions had been renamed. Using them in a way that works on Mac OS X just crashed on the iPhone. Due to the lack of documentation a lot of guessing and trial-and-error was required to figure out all the changes and avoid bugs that still seem to be lurking in normally unused parts of the iPhone libraries.

Besides figuring out how to access contacts on the iPhone the complete conversion to and from a standard format (vCard 2.1) had to be written. On Mac OS X there are library functions for that, but a) they use vCard in a way which is unlikely to be supported well enough by SyncML server and b) these functions are missing on the iPhone. With full control over the conversion process it was possible to minimize data loss to a minimum; see the compatibility entry for the platform for details.

For software on the iPhone a single-click installation is almost a requirement. Once I started with setting up the necessary repository I felt bad about not offering the same convenience to existing users, so I also added an apt repository for Debian/Ubuntu and an .install file for the Nokia Internet tablets.

Fabrizio tested all of these early versions on his own iPhone and provided valuable feedback. When it became obvious that the command line version worked, he was eager to also offer a GUI for it to the less technically inclined users. This time I managed to resist – I was a bit more diplomatic than “I’ve got better things to do” though But there is some truth to it: for the final 0.7 release the other platforms still need a bit of love and care. It also would be nice to not spend all evenings in front of the computer, even if just for a while.

Luckily that was enough motivation for Fabrizio to do the GUI himself. Now we just need to figure out a way how users can install both easily while at the same time being able to upgrade GUI and SyncEvolution independently: currently the Funambol GUI package contains a full copy of the SyncEvolution files. Therefore installing or updating the GUI after a SyncEvolution update will fall back to an older version of it unless the packages are always in sync. Hmm.

I have dodged an obvious question which came up right after the release: what about calendars? More on that in the next post.