Skip to content
{ 2009 07 01 }

Linux 2.6.30: Hardware Assisted Time Stamping of Network Packets

It might not be breaking news anymore, but it’s still the latest kernel release, so here we go… Linux 2.6.30 was released.

It includes my patches which add the new timestamping.txt user space API ( for hardware assisted time stamping of network packets. I also updated the igb driver for the Intel NIC 82576 and the user space PTPd, an implementation of IEEE 1588/Precision Time Protocol (PTP), so that finally there is a decent replacement for NTP in LANs and clusters. For details, see the LCI 2008 paper about that aspect.

This is my first Linux kernel patch. It would not have been possible without support by managers and colleagues at Intel and the patient code review by the Linux community, so kudos to everyone involved. Please contact the Intel Ethernet driver team if you have further questions about PTP or the Intel NIC 82576 features, I myself focus exclusively on SyncEvolution nowadays.

Unfortunately, PTPd is now officially unmaintained after a long period of no activity, so my patches for it were never looked at. Anyone interested in taking over the project?

Posted by Patrick Ohly on Wednesday, July 1, 2009, at 4:55 pm. Filed under Uncategorized. Follow any responses to this post with its comments RSS feed. You can post a comment or trackback from your blog.

{ 16 } Comments

  1. Chenthill | July 4, 2009 at 4:31 pm | Permalink

    congrats on ur kernel patch! its always very nice feeling have the first kernel patch isn it ? :)

  2. CJ | August 12, 2009 at 5:03 pm | Permalink

    Are the updated ptpd patches here capable of working with the 2.6.30.4 kernel and stock IGB drivers ?

  3. Patrick Ohly | August 12, 2009 at 5:29 pm | Permalink

    CJ, yes, the patched PTPd should work with 2.6.30 and the in-kernel IGB drivers.

    I need to add that the software TX fallback wasn’t merged into 2.6.30. It’s part of the API, but the implementation is missing and therefore TX time stamps will not be returned unless the hardware support in a NIC 82576 is used. I completely forgot about that when writing the blog post (I swear!) because I had implemented it and then handed it off to my colleagues from the driver team.

    It’s also worth pointing out that John Linville has volunteered to maintain PTPd:
    https://sourceforge.net/forum/message.php?msg_id=7554174
    http://lists.infradead.org/mailman/listinfo/ptpd

    His tree supersedes mine:
    git://git.infradead.org/ptpd.git

  4. CJ | August 12, 2009 at 6:30 pm | Permalink

    I am assuming this isnt expected ? Also what sort of granularity should we expect here, down to the nano?

    # ./ptpd -z assisted -b eth3 -c
    (ptpd debug) allocated 1296 bytes for protocol engine data
    (ptpd debug) allocated 600 bytes for foreign master data
    (ptpd debug) event POWERUP
    (ptpd debug) state PTP_INITIALIZING
    (ptpd debug) manufacturerIdentity: Kendall;1.0.0
    (ptpd debug) netInit
    (ptpd error) could not activate E1000 hardware time stamping on eth3: Operation not supported
    (ptpd error) failed to initialize timing
    (ptpd debug) state PTP_FAULTY
    (ptpd notice) self shutdown, probably due to an error
    # ethtool -i eth3
    driver: igb
    version: 1.3.16-k2
    firmware-version: 1.2-1
    bus-info: 0000:09:00.0
    # lspci | grep 82576
    09:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    09:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

  5. Patrick Ohly | August 12, 2009 at 10:26 pm | Permalink

    -z assisted uses the ioctl() method, which wasn’t included in the 2.6.30 kernel. What you need is -z linux_hw:

    -z …
    linux_hw = synchronize system time with Linux kernel assistance
    via net_tstamp API, uses NIC time stamping

    There’s also linux_sw, but that does not work at the moment due to the missing TX software fallback:

    linux_sw = synchronize system time with Linux kernel assistance
    via net_tstamp API, uses software time stamping

  6. CJ | August 13, 2009 at 12:57 pm | Permalink

    OK well I have been testing this in my lab using linux_hw and I am getting really bad clock drift, but I was running linux_sw on the preffered clock and linux_hw on the client. So that could explain why it was not working. I will try testing this a bit more today.

    Two more questions:

    * Will I be able to get down to the nanonsecond granularity here? We are considering using these cards for packet capture devices since they are relatively cheap. We are trying to get timestamping on the box as granular as possible so packet timestamps in our pcaps are very close.

    * When will the linux_sw feature be supported? Can you describe what you mean by TX fallback? I would imagine this would provide a faster interface for ptp to use for packet timestamping versus the native approach since it will be using an API directly into the kernel?

  7. Patrick Ohly | August 15, 2009 at 2:32 pm | Permalink

    Nanosecond is ambitious. One microsecond is more realistic. See the LCI paper for test results. Although the paper was about the ioctl() API, the Linux kernel implementation is very similar to the “assisted” mode and should yield similar results.

    linux_sw does time stamping of network packets inside the Linux IP stack, very much like the time stamping of incoming packets that has been in the kernel before. They main difference is the API, not so much the accuracy. It’s definitely going to be worse than hardware-assisted methods.

    I’m not sure what you mean with “the native approach”. PTPd has used time stamping inside the kernel for a while already, by asking for time stamps on incoming packets and looping outgoing ones so that they get time stamped on the way back.

  8. Patrick Ohly | August 15, 2009 at 2:33 pm | Permalink

    As to “when will the kernel support linux_sw”, that depends on who is going to propose its inclusion. Currently the Linux driver team has the patch. I suggest you get in touch with them, if you want the patch to get included.

  9. Marcus Leech | September 5, 2009 at 11:18 pm | Permalink

    How much of the driver-side interface as described in timestamping.txt is actually in 2.6.30.X kernels? For example, timestamping.txt refers to:skb_hwtstamp_check_tx_hardware()

    But I don’t find it anywhere in the 2.6.30.5 kernel code that I downloaded last night.

    I have some experimental multi-port NIC hardware (done in an FPGA) that supports both PTP and NTP clocks, and timestamps packets in both directions, and can do “fixups” on outbound NTP and PTP packets. Some test-out code has been written, and now I want to make it work nicely as Linux device, with support for the emerging hardware timestamping support.

    So, if my hardware provides a timestamp on every packet–I can convert it into whatever format is required (like timespec/ktime), how do I pass it upwards via the skb? If the interface hasn’t been configured for this, does it do any harm if I pass the information upwards anyway? What driver-API (DAPI?) calls should I use? The timestamping.txt file is useful, but it looks like only part of it is actually implemented, so I could really use some guidance.

  10. Patrick Ohly | September 8, 2009 at 1:57 pm | Permalink

    Marcus, have a look at include/linux/net_tstamp.h for the definitions and drivers/net/igb for a driver which implements the feature.

    The reference to skb_hwtstamp_check_tx_hardware() in timestamping.txt is out-dated. The approach implemented in the final patch is:

    union skb_shared_tx *shtx;
    shtx = skb_tx(skb);
    if (unlikely(shtx->hardware)) {
    /* do HW time stamping for outgoing packet */
    }

    See igb_xmit_frame_ring_adv().

    The time stamp is returned via skb_tstamp_tx(). See igb_tx_hwtstamp. If the sending socket didn’t request the send time stamp, then the app might be surprised to get a packet back via its error queue – better check on a per-packet basis.

    For received packets there’s no harm in always adding the time stamp, even if no-one later uses the information.

  11. Kate | September 15, 2009 at 12:00 am | Permalink

    I have been trying to get PTPd with hardware timestamps to work, but unsuccessfully. I am running PTPd as follows:
    ptpd -z linux_hw -b eth0

    The error I am receiving is:
    (ptpd error) net_tstamp SIOCSHWTSTAMP: Invalid argument: Invalid argument

    As far as I can tell, igb_ioctl does not get reached.
    Can anyone point me in the right direction? I am running a 2.6.29 kernel.

  12. Kate | September 15, 2009 at 4:17 pm | Permalink

    I tried running ptpd -z linux_hw -b eth0 on the 2.6.30 kernel and got a different error:
    net_tstamp SIOCSHWTSTAMP: Operation not supported

    I do have the NIC 82576 card with an igb driver on my machine. Any ideas on what’s wrong?

  13. Patrick Ohly | September 15, 2009 at 5:55 pm | Permalink

    Kate, SIOCSHWTSTAMP is not supported by the Linux kernel < 2.6.30, which explains the “Invalid argument” argument.

    I can only guess about the “Operation not supported” error with 2.6.30: do you perhaps try to use the sf.net igb driver instead of the one that comes with 2.6.30? I am not sure whether (or which) sf.net releases support the HW time stamping API. This is a question that you should bring up on the sf.net/project/e1000 mailing list. These are also the people who continue to support the Linux kernel feature.

  14. Kate | September 15, 2009 at 11:10 pm | Permalink

    Patrick, Thank you for your reply. I was using a wrong driver, just like you said. I found the correct driver and ptpd -z linux_hw works for me now.

  15. Barry | September 27, 2009 at 5:33 am | Permalink

    Patrick Ohly,
    Thank you very much for you bring-up hardware timestamp supporting in ptpd. I am using it, but unfortunately it doesn’t work.
    Case 1:
    master Sep 27 2009, slave Jan 5 1970, running ptpd -z linux_hw -g in slave, the log is:

    root:/> ptpd -z linux_hw -g

    (ptpd debug) allocated 1264 bytes for protocol engine data
    (ptpd debug) allocated 600 bytes for foreign master data
    (ptpd debug) event POWERUP
    (ptpd debug) state PTP_INITIALIZING
    (ptpd debug) manufacturerIdentity: Kendall;1.0.0
    (ptpd debug) netInit
    (ptpd debug) initData
    (ptpd debug) initTimer
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) sync message interval: 2
    (ptpd debug) clock identifier: DFLT
    (ptpd debug) 256*log2(clock variance): -4000
    (ptpd debug) clock stratum: 4
    (ptpd debug) clock preferred?: no
    (ptpd debug) bound interface name: eth0
    (ptpd debug) communication technology: 1
    (ptpd debug) uuid: 00:e0:22:fe:5c:e0
    (ptpd debug) PTP subdomain name: _DFLT
    (ptpd debug) subdomain address: e0.0.1.81
    (ptpd debug) event port address: 3f 1
    (ptpd debug) general port address: 40 1
    (ptpd debug) state PTP_LISTENING
    (ptpd debug) updateForeign: new record (0,1) 1 1 00:22:19:0a:46:7d
    (ptpd debug) state PTP_PTP_SLAVE
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) Q = 0, R = 6
    (ptpd notice) resetting system clock to 1254021169s 935610415ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: -1253605562s -863610415ns
    (ptpd debug) observed drift: 0
    (ptpd notice) resetting system clock to 2134629930s 225735894ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: -880608758s -290125894ns
    (ptpd debug) observed drift: 0
    (ptpd info) requested adj 11420 ppb => adjust system frequency by 742300 scaled ppm (11420 ppb) + 0 us/tick (0 ppb) = adj 11420 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -113078ns
    (ptpd debug) observed drift: -113
    (ptpd info) requested adj 25746 ppb => adjust system frequency by 1673490 scaled ppm (25746 ppb) + 0 us/tick (0 ppb) = adj 25746 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -253807ns
    (ptpd debug) observed drift: -366
    (ptpd info) requested adj 24677 ppb => adjust system frequency by 1604005 scaled ppm (24677 ppb) + 0 us/tick (0 ppb) = adj 24677 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -240717ns
    (ptpd debug) observed drift: -606
    (ptpd debug) Q = 0, R = 20
    (ptpd debug) handleDelayReq: self
    (ptpd info) requested adj 18520 ppb => adjust system frequency by 1203800 scaled ppm (18520 ppb) + 0 us/tick (0 ppb) = adj 18520 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -177371ns
    (ptpd debug) observed drift: -783
    (ptpd info) requested adj 15849 ppb => adjust system frequency by 1030185 scaled ppm (15849 ppb) + 0 us/tick (0 ppb) = adj 15849 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -149179ns
    (ptpd debug) observed drift: -932
    (ptpd info) requested adj 14357 ppb => adjust system frequency by 933205 scaled ppm (14357 ppb) + 0 us/tick (0 ppb) = adj 14357 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -132939ns
    (ptpd debug) observed drift: -1064
    (ptpd info) requested adj 11621 ppb => adjust system frequency by 755365 scaled ppm (11621 ppb) + 0 us/tick (0 ppb) = adj 11621 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -104531ns
    (ptpd debug) observed drift: -1168
    (ptpd info) requested adj 7668 ppb => adjust system frequency by 498420 scaled ppm (7668 ppb) + 0 us/tick (0 ppb) = adj 7668 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 0s -64364ns
    (ptpd debug) observed drift: -1232
    ^C(ptpd notice) shutdown on interrupt signal

    Then I check the time:
    root:/> date
    Sun Aug 23 08:45:48 UTC 2037
    root:/>
    The time is so much different with master’s Sep 27 2009.

    Case 2:
    After that, I start-up slave ptpd again, then the log is:
    root:/> ptpd -z linux_hw -g
    (ptpd debug) allocated 1264 bytes for protocol engine data
    (ptpd debug) allocated 600 bytes for foreign master data
    (ptpd debug) event POWERUP
    (ptpd debug) state PTP_INITIALIZING
    (ptpd debug) manufacturerIdentity: Kendall;1.0.0
    (ptpd debug) netInit
    (ptpd debug) initData
    (ptpd debug) initTimer
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) sync message interval: 2
    (ptpd debug) clock identifier: DFLT
    (ptpd debug) 256*log2(clock variance): -4000
    (ptpd debug) clock stratum: 4
    (ptpd debug) clock preferred?: no
    (ptpd debug) bound interface name: eth0
    (ptpd debug) communication technology: 1
    (ptpd debug) uuid: 00:e0:22:fe:5c:e0
    (ptpd debug) PTP subdomain name: _DFLT
    (ptpd debug) subdomain address: e0.0.1.81
    (ptpd debug) event port address: 3f 1
    (ptpd debug) general port address: 40 1
    (ptpd debug) state PTP_LISTENING
    (ptpd debug) updateForeign: new record (0,1) 1 1 00:22:19:0a:46:7d
    (ptpd debug) state PTP_PTP_SLAVE
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) Q = 0, R = 6
    (ptpd notice) resetting system clock to 766255858s 548470130ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: 1368374263s 670814870ns
    (ptpd debug) observed drift: 0
    (ptpd notice) resetting system clock to -1381227787s -451556678ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: -2147483647s -999973322ns
    (ptpd debug) observed drift: 0
    (ptpd notice) resetting system clock to -1381227785s -451539169ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: -2147483647s -999990831ns
    (ptpd debug) observed drift: 0
    (ptpd notice) resetting system clock to -1381227783s -451487406ns
    (ptpd debug) initClock
    (ptpd info) requested adj 0 ppb => adjust system frequency by 0 scaled ppm (0 ppb) + 0 us/tick (0 ppb) = adj 0 ppb (freq limit -504123/504123 ppm, tick limit -1000/1000 us*USER_HZ)
    (ptpd error) adjtimex -> time bad
    (ptpd debug) offset from master: -2147483648s -42594ns
    (ptpd debug) observed drift: 0
    ^C(ptpd notice) shutdown on interrupt signal

    Now the time comes to:
    root:/> date
    Wed Apr 13 16:51:06 UTC 1994
    The time is still so much different with master’s Sep 27 2009.

    Case 3:
    I simple changes the codes in drivers like:
    timecompare_update(&adapter->compare, ns);
    shhwtstamps.hwtstamp = ns_to_ktime(ns);
    shhwtstamps.syststamp =
    - timecompare_transform(&adapter->compare, ns);
    + ktime_get_real();
    I means replacing transformed hardware time-stamp by system time directly, then then the slave time can sync with master right.

    So I want to know whether “ptpd -z linux_hw” can really work.
    In ptpd patched by you, there are many ioctl like E1000_TSYNC_SYSTIME_IOCTL,E1000_TSYNC_ADJTIME_IOCTL for “both” and “nic” mode, but the driver codes don’t have them fulfilled. So do hardware timestamp must have them fulfilled to work?

  16. Christopher Johnston | October 21, 2009 at 12:27 am | Permalink

    Is PTPv2 supported with the client and with Intel 82576 NICs with the Kernel hardware assisted timestamping?

Post a Comment

Your email is never published nor shared. Required fields are marked *