The 7.0.3 release for the first time uses the time zone information provided by clients and therefore promises to solve the “events shifted by some hours” problem that users of the Thunderbird and Evolution clients frequently run into. Hauke Pribnow, a Funambol user, also fixed several issues (RRULEs in iCalendars, EXDATEs have to be separated by commas) in the server’s iCalendar 2.0 support. Therefore special attention was paid to calendar synchronization.

This report was posted on the Funambol developers mailing list and only published here after the initial comments were favorable. Head over to the list to read up on the discussion of the issues mentioned here, or watch the comments: I’ll post summaries occasionally.

Status Summary and Call for Action

Unfortunately, some issues were already known to affect 7.0.3 calendar support (events with a DURATION value are not handled correctly, server sends iCalendar items with TZID and UTC date-time). The tests explained below revealed further deficiencies (all reported in the Funambol bug tracker and linked to below). Some of these deficiencies (poor support for meeting invitations and complex exceptions from meeting serieses, broken time zone support) also affect other clients, in particular in an international enterprise setup. Overall calendar synchronization still doesn’t work as well as it has to for Evolution, so better don’t rely on that combination quite yet. The same applies to tasks, which use the same data format. Contacts and notes work well, although there is room for improvement for contacts.

This recommendation against using calendar sync is admittedly rather harsh, but I prefer not having users syncing their calendars over getting reports of corrupted data or missed meetings. Remember, Funambol’s server is open source. If you care about calendar synchronization and can do some Java programming, then don’t expect Funambol to fix these issues all by themselves and start contributing to the project! I don’t have the time, but perhaps reporting the issues in the bug tracker, describing ways to reproduce the problems and documenting them here will motivate someone to take the necessary next steps.

[Update 2008-08-28] Funambol has started working on the calendar issues. They are scheduled for a 7.1 update of the server in the first quarter of 2009.

[Update 2008-10-16] The roadmap was updated and now lists the iCalendar 2.0 changes for release 8.0 in the second half of 2009.

Funambol Data Model

Before we get to the details, let’s give an overview how the Funambol server handles PIM data. All incoming items are converted into a Funambol internal representation of an event, contact, note, or task. When sending to a client, the internal representation is converted back into the format expected by that client. Funambol connectors might decide to do that conversion themselves, but they might also use the Funambol framework. Therefore what is described in this post may or may not apply to them.

The advantage of converting to an internal format is that conversion between different formats only requires one encoder/decoder for each format. The disadvantage is that any property of an item that cannot be represented by the internal Funambol format gets lost during the conversion. Because of that, the server must be as capable as the most feature-rich client that is to be synchronized via the server. Vendor specific extensions of the vCard or iCalendar standards like X-EVOLUTION-MANAGER are not preserved unless the Funambol server knows about them and maps them to some internal field. Other servers store such extensions verbatim and therefore preserve them.

Therefore the main questions are:

• How capable is the Funambol data model? Extending it can be quite some work and requires special care when installing an update.
• How good are the encoders/decoders? Fixing them is easier and might be done in a simple bug fix update.

Setup

I enabled iCalendar 2.0 in the local installation of Funambol 7.0.3 on Linux like this:

• run the Funambol Administration Tool
• connect to the server
• unfold the <localhost>/Modules/foundation/FunambolFoundationConnector/PIM Calendar SyncSource
• add an “event2″ sync source for events with type “iCalendar”
• add a “task2″ sync source for tasks with type “iCalendar”
• change the SyncEvolution source property “URI” for “calendar” resp. “todo” accordingly

For testing the PIM synchronization, I ran the SyncEvolution 0.8 beta 2 client-test testItems tests. That automates the following steps:

• import test data into client A
• send to server
• receive from server into client B
• compare the two database dumps with synccompare

Note that synccompare automatically suppresses some known differences when the server configuration name contains the string funambol. For this analysis the names were chosen so that the comparison doesn’t suppress any difference. Here’s how the testing can be reproduced inside SyncEvolution’s src directory:



make client-test
CLIENT_TEST_SERVER=local ./client-test --help
./syncevolution --configure --sync-property username=guest --sync-property password=guest \
                --sync-property syncURL=http://localhost:8080/funambol/ds local_1
./syncevolution --configure --source-property uri=event2 local_1 ical20 file_ical20
./syncevolution --configure --source-property uri=task2 local_1 itodo20 file_itodo20
./syncevolution --configure --sync-property username=guest --sync-property password=guest \
                --sync-property syncURL=http://localhost:8080/funambol/ds local_2
./syncevolution --configure --source-property uri=event2 local_2 ical20 file_ical20
./syncevolution --configure --source-property uri=task2 local_2 itodo20 file_itodo20
CLIENT_TEST_SERVER=local CLIENT_TEST_DELAY=5 CLIENT_TEST_EVOLUTION_PREFIX=file:///tmp/test/ \
         ./client-test \
                   Client::Sync::ical20::testItems \
                   Client::Sync::itodo20::testItems \
                   Client::Sync::vcard21::testItems \
                   Client::Sync::text::testItems

I used the Evolution test program because I wanted to see how item modifications affected Evolution. If you don’t have Evolution, then there are two other alternatives that can be used to reproduce most of the testing:

1. SyncEvolution compiles fine without Evolution. The file back end will always be enabled. To test with it, just use Client::Sync::file_ical20::testItems Client::Sync::file_itodo20::testItems Client::Sync::file_vcard21::testItems. The test data for vCard 2.1 will be different from the one used by the Evolution back end, though, because the Evolution back end imports the vCard 3.0 test cases and converts them to vCard 2.1 itself. There are no tests for plain text.
2. Use the Funambol C++ client library’s client-test. It is doing the same tests as the file back end in SyncEvolution. Use the latest C++ client library source code, the tests in 7.0.3 don’t work. Also beware that one has to list the enabled sources via CLIENT_TEST_SOURCES=ical20,itodo20,vcard21 explicitly.

This testing has one major shortcoming: it doesn’t check that the server actually maps the items to its internal data model correctly. A dumb server which doesn’t understand the content of items at all and just bounces the binary blob back to the second client will pass all tests with flying colors. A server which applies the same incorrect mapping both for incoming and outgoing items will also pass.

Therefore this automated testing still has to be combined with additional tests, like checking how the web interface or some other client which uses a different data format display items. I haven’t done that kind of testing this time around.

Events/Tasks

As explained in a discussion on the Funambol developers mailing list, the new time zone support basically works like this:

• Every time stamp in the server database can be associated with a time zone, identified with a string that references the time zone information built into Java. The time stamp itself is always in UTC.
• When receiving an event or task with time zone information, the server maps the time zone to the built-in ones by looking at the content of the time zone definition. This definition always has to be sent together with the item because the standard mandates that. It then converts the time stamps to UTC and stores it together with the internal time zone ID. The original definition sent by the client is dropped. If it wasn’t possible to map the time zone, then the event is stored with converted UTC time stamps and no time zone ID.
• When sending an item, then time stamps are converted back from UTC using the associated time zone ID, if one exists. At least in theory: in practice there is the problem mentioned above that the conversion back from UTC doesn’t happen, leading to invalid iCalendar 2.0 times which are in UTC and have a TZID. Evolution (I’m using 2.12.3, but other versions likely do the same) then treats the times as if they were relative to that time zone (which shifts them in time) and hangs when trying to open such a broken event. That’s a bug in Evolution, it should tolerate the broken data.

Besides that implementation issue, there are also some conceptual problems with the approach:

• Converting time stamps to UTC without the original time zone drops relevant information. Recurring events are then no longer expanded correctly. It would be better to store the time zone definition received from the original client.
• Storing in UTC can shift the event in time even if a time zone ID is stored. Suppose a time stamp is converted to UTC, using the rules of its time zone valid at the time when the event is imported. Later the time zone definition is updated (that happens!). Now the time stamp is converted back from UTC using different rules, which can lead to a result which is different from the original time stamp.
• The mapping via the time zone content maps many different time zones to the same ID. For example, “Europe/Paris”, “Europe/Rome”, “Europe/Berlin” all have the same content – at the moment. The server then uses “Europe/Berlin” for all of them. It’s mostly hypothetical in Western Europe, but in other regions time zone changes might be less coordinated. This can lead to the situation that country A and B are both mapped to country A at some point in time, but then later only country B changes its time zone rules. It’s events are then not converted back correctly. Funambol agreed to also use the TZID for time zone mapping in those cases where that is possible, but that is not in 7.0.3 yet.
• If an old event is sent with a time zone definition that was valid at the time when the event was created, but is no longer up-to-date when the event is imported, the mapping against current internal time zones can lead to incorrect results. This might not matter for one-time events (because the rules are correct at that time) but can be incorrect for recurring events.

My blog post about storing time stamps explains these issues in more detail.

Here’s a list of other issues observed during Client::Sync::ical20::testItems:

• Evolution’s definition of “Europe/Berlin” (with summer saving time) is mapped to “Africa/Lagos” (without summer saving time) which shifts the event.
• The UID of events are lost. The UID is important for clients. Without it, they cannot properly process meeting updates received via email. Meeting serieses with complex exceptions (different start time/summary/attendees, etc.) depend on the UID and the so called RECURRENCE-ID which also gets lost when synchronizing via Funambol.
• The “show event as busy” flag is not preserved. This affects scheduling of meetings when colleagues have access to the free/busy information.
• In meetings the list of attendees is lost. The organizer is stored, but only the mail address, not the name.
• In iCalendar 2.0 items sent by the server, commas in text properties are not escaped properly. As a result of that, Evolution splits descriptions or summaries at these incorrectly escaped commas and only displays one of the parts.
• All-day events are sent by the server as events starting at midnight and lasting until the last minute before the next day. A special X-FUNAMBOL-ALLDAY tells Funambol clients that this really is an all-day event. The normal iCalendar 2.0 representation of all-days events should be used instead. Evolution doesn’t know the flag and displays the event as covering the whole day in the day view. Real all-day events are shown at the top of the day view.
• Special characters are escaped using quoted-printable, which is not a valid encoding for iCalendar 2.0. Evolution then displays the escape codes instead of the real text.
• In the same example the server also sends a redundant UTF-8 character set parameter, which Evolution’s libical complains about by adding a X-LIC-ERROR property to the item.
• The alarm specification in iCalendar 2.0 items gets lost. All events imported into Evolution then have no alarm set at all.

Issues found elsewhere which are not (yet) triggered by the testItems test data:

• events with a DURATION value are not handled correctly
• [Update 2008-08-28] the server doesn’t support week days recurring events
• [Update 2008-10-16] parser error for some events generated by Outlook 2007

The results for tasks are similar because they also use the iCalendar 2.0 format and similar properties.

Note that iCalendar 2.0 is not supported for myFUNAMBOL. When trying to exchange iCalendar 2.0 with myFUNAMBOL, the server will send events as vCalendar 1.0. Evolution partly imports that, but there are character encoding issues which can later on cause errors on the server and then break synchronization completely – better don’t try it!

Contacts

The standard formats for contacts, vCard 2.1 and its later revision 3.0, are quite old and cannot represent contact properties added later on, like instant messaging information, anniversaries and related persons (spouse, manager, etc.). Funambol uses their own extension for such properties (e.g., X-FUNAMBOL-SPOUSE) which only works when both client and server are from Funambol. Evolution follows the example set by other software in some cases (X-AIM/ICQ/..., X-MOZILLA-HTML), but also adds its own extensions (X-EVOLUTION-SPOUSE). Therefore most of these extended properties cannot be synchronized between Funambol server and Evolution.

This was discussed with Funambol, leading to the agreement to change both server and SyncEvolution so that they use the vendor-neutral extensions where it makes sense – none of these changes are in the current releases, though.

The properties not synchronized during Client::Sync::vcard21::testItems have been known for a while, so this loss of information is normally ignored by synccompare when testing the Funambol server. Here’s the list of properties not synchronized, for the sake of completeness:

• second address line
• X-EVOLUTION-FILEAS (can be specified explicitly by Evolution users in addition to full name, nick name and the name components)
• the office (represented as third level in the ORG property)
• calendar URI, ordering of emails in the Evolution GUI, instant messaging fields, anniversary, assistant, manager, spouse, blog URL, video URL: all of them rely on vCard extensions
• very long notes are truncated

[Update 2008-11-01] only one email and phone number of each kind is stored (i.e., only one work email).

Notes

Using the “plain/text” note URI everything worked fine, both with the local installation as well as with the myFUNAMBOL. According to a comment on the Funambol users mailing list, the “plain/text” sync source only sends the body of a note. This might indicate that the title of a note sent by a client which uses Funambol’s XML format is not sent to clients using “plain/text”. I haven’t tested this, though.

The usual convention is that the first line of a text is used as a title or summary. One possible solution would be the following rules, which are also applied by SyncEvolution when converting between Evolution’s summary plus body and plain text: when sending plain text, the summary is inserted before the body if it is different from the first line. When receiving a plain text, the first line is used as summary without removing it from the body. That’s because it might be an integral part of the text.