an addendum has been posted, read it here

2023/12/02 - Hello, Hacker News! Check out these pictures of my cat: https://emma.pet

Back in February, Mikrotik announced the CCR2004-1G-2XS-PCIe, a peculiar 25gbps NIC. They’ve thrown an entire router into a PCIe NIC. But why?

Wait, why?

Mikrotik’s product listing has illustrations that suggest using it in a file server and a workstation, with a copper DAC in between. For something like a basic NIC, sure, that makes sense. But why throw in all of the routing and firewall capabilities? It all seems wildly excessive for that usecase (however it does seem to be one of the cheapest brand new 25Gbe NICs currently available).

I can’t really see it fitting into large infrastructure deployments other than being used as a cheap 25Gbps NIC in the default passthrough configuration, and seems to be targeted mostly towards small businesses.

I’m not sure if this product will ever actually take off, since it’s been over 3 months since it was announced (or launched?) and is seemingly nowhere to be seen. I can’t find any proper reviews for it, and pretty much all the information on it is just commentary on or directly parroting the official announcement (so I guess I’m first?).
The lifecycle state of it is extremely unclear: Mikrotik’s site just says “send purchase questions”, there are 34 pages total that mention it on their site (a vast majority of which are people asking why in forum threads, the rest are a handful of manuals and product matrixes), and almost all the Mikrotik authorized retailers either say “Pre-Order” or “Out of Stock”. I’m not even sure if I’m supposed to have this thing.

Regardless of the weird lifecycle void this falls into, I believe I’ve found the ideal use case for it: single server co-location.

When you lease co-location space in small enough quantities to lease in terms of “rack units” instead of “racks”, you’re going to be paying extra for every single space you use. They often provide no solution for firewalls and no customer-configurable routers and just hand you an uplink that’s wide-open to any traffic that could possibly head your way. The typical configuration for custom routing and firewalls is to attach the WAN port to a virtual machine running something like Vyos or pfSense and then creating a LAN vSwitch and attaching it to all of your VMs, which burns extra CPU cycles on all of the traffic processing.

Aside from the weird lifecycle situation this card has, there’s also an inconsistency between what’s listed on their main product page which says that it runs a 1.5GHz Annapurna Labs AL52400 CPU and the manual and spec sheet, which says it uses a 2GHz Annapurna Labs AL32400 CPU. The two different CPU speeds in addition to the different model numbers leads me to believe they’re different chips but without taking it apart I can’t confirm. I’m unable to find any actual documentation on the AL52400, so it could either be an error or something from an earlier “beta” card. Unfortunately, I’m not able to get any good diagnostic information on what it actually is from the device itself. Mikrotik’s documentation lists system resource print as also outputting a CPU field that shows what the model is, but this doesn’t output one. I’m inclined to believe that it’s the AL32400 since it’s the same one that appears in its cousin, the CCR2004-1G-12S+2XS.

[admin@MikroTik] /system/resource> print
                   uptime: 1h8m25s
                  version: 7.1.4 (stable)
               build-time: Mar/21/2022 11:23:09
         factory-software: 7.1.4
              free-memory: 3875.9MiB
             total-memory: 4032.0MiB
                cpu-count: 4
                 cpu-load: 0%
           free-hdd-space: 106.9MiB
          total-hdd-space: 129.0MiB
  write-sect-since-reboot: 174
         write-sect-total: 174
               bad-blocks: 0%
        architecture-name: arm64
               board-name: CCR2004-1G-2XS-PCIe
                 platform: MikroTik

Looks and Physical Specs

The CCR2004-1G-2XS-PCIe is a sleek, black, single-slot card with a single blower fan towards the back. It has two SFP28 cages for uplinks, and one RJ-45 connector. Mikrotik’s specs list it as 170mm x 69mm x 18mm, small enough to fit comfortably in my half-depth HP ProLiant DL20 G9. It does partially block the SATA port that’s under it, so if you want to access the DL20’s SATA port you’ll have to replace the DL20’s original SATA cable with one that has a 90° connector. It doesn’t require any power sources other than the PCIe slot it’s installed in.

Testing the Card

The manual suggests adding a PCIe initialization delay in your BIOS settings because the card needs to be up before the host system tries to initialize it. I couldn’t find any obvious ways to do that on my HP ProLiant DL20 G9 after going through every menu, but in the end it didn’t matter because the card was always up long before the host tried to initialize it. The brochure (but not the manual? Seriously, the manual has almost nothing. Its biggest section tells you how to use the reset button.) recommends using the following script to reinitialize PCIe devices in Linux systems if the host boots before the card:

echo "1" > /sys/bus/pci/devices/0000\:03\:00.0/remove # substitue your actual device ID here
sleep 2
echo "1" > /sys/bus/pci/rescan

The card is presented to the host operating system as a Qualcomm Atheros AR8151 (for the full verbose output of lspci, check here), so the system you install it in will need drivers that support it. I tested Ubuntu 22.04 (but it says anything with kernel 5.15.25 or higher should work), pfSense 2.6.0, and the latest Vyos 1.4 Rolling build that was available at the time (vyos-1.4-rolling-202205210217-amd64.iso).

I attempted to load drivers for it into VMware 7.0 Update 3d, but the community VIB file (net-atl1e-1.0.1.14-1.x86_64.vib) for the Atheros chipset isn’t supported in VMware 6.7+ and newer because VMware deprecated the ability to load legacy drivers. It may run in VMware 5.0-6.5, but I have not tested it and they’re old enough that I’m not sure why you’d want to.

In my testing, I used FS.com’s SFP+ DACs and was able to get a link up at 10Gbps to my NAS running an Intel X520-DA2 and across the SFP+ ports on my Dell Force10 S4810P. (ethtool output for the interfaces the card presented is here (ens2f0 has the SFP+ module attached; ens2f2 and ens2f3 are bridged to the management interface))

It works exactly as you’d expect, it’s just a Mikrotik router that siphons power from and can present virtual interfaces to its host, there’s not really any surprises.

I don’t have the gear to build elaborate topologies to push it to its limits and don’t have hardware to test it against that goes any faster than 10Gbps, so my performance testing was extremely limited.

In testing, I used the default MTU of 1500. Hardware-wise I used:

• HP ProLiant DL20 G9
  • Intel Xeon E3-1240 v6
  • 32 GB DDR4 ECC (2x16)
  • Mikrotik CCR2004-1G-2XS-PCIe
  • Nitrokey HSM 2
• Custom Cube NAS
  • Supermicro X11SCL-F
  • Intel Celeron G4930 (believe it or not, this actually works fantastically for a NAS)
  • 16GB DDR4 ECC (2x8)
  • Intel X520-DA2
• Dell S4180P
• FS.com CBL-10GSFP-DAC-1M Copper DACs

Bi-directional iPerf3

I ran a bidirectional iPerf3 test in the default configuration for a duration of 10 minutes, and didn’t observe any performance or speed degradations during that time. During the test the card reported a CPU load of roughly 20% for the entire duration. I’ve attached the first ten seconds below, but you can view the entire test here.

During the test the RX rate is significantly lower than the TX, which I suspect may be due to a bottleneck from the Celeron-powered NAS.

[  5][TX-C]   0.00-1.00   sec  1.06 GBytes  9.08 Gbits/sec   37   1.01 MBytes
[  7][RX-C]   0.00-1.00   sec   676 MBytes  5.67 Gbits/sec
[  5][TX-C]   1.00-2.00   sec  1.08 GBytes  9.31 Gbits/sec   10   1.37 MBytes
[  7][RX-C]   1.00-2.00   sec   661 MBytes  5.55 Gbits/sec
[  5][TX-C]   2.00-3.00   sec  1.07 GBytes  9.16 Gbits/sec   14   1.43 MBytes
[  7][RX-C]   2.00-3.00   sec   670 MBytes  5.62 Gbits/sec
[  5][TX-C]   3.00-4.00   sec  1.09 GBytes  9.36 Gbits/sec    0   1.46 MBytes
[  7][RX-C]   3.00-4.00   sec   684 MBytes  5.74 Gbits/sec
[  5][TX-C]   4.00-5.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.48 MBytes
[  7][RX-C]   4.00-5.00   sec   676 MBytes  5.67 Gbits/sec
[  5][TX-C]   5.00-6.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.49 MBytes
[  7][RX-C]   5.00-6.00   sec   675 MBytes  5.66 Gbits/sec
[  5][TX-C]   6.00-7.00   sec  1.08 GBytes  9.31 Gbits/sec    0   1.51 MBytes
[  7][RX-C]   6.00-7.00   sec   667 MBytes  5.59 Gbits/sec
[  5][TX-C]   7.00-8.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.52 MBytes
[  7][RX-C]   7.00-8.00   sec   688 MBytes  5.77 Gbits/sec
[  5][TX-C]   8.00-9.00   sec  1.08 GBytes  9.29 Gbits/sec    0   1.54 MBytes
[  7][RX-C]   8.00-9.00   sec   671 MBytes  5.63 Gbits/sec
[  5][TX-C]   9.00-10.00  sec  1.09 GBytes  9.34 Gbits/sec   20   1.57 MBytes
[  7][RX-C]   9.00-10.00  sec   676 MBytes  5.67 Gbits/sec

The low RX speed persisted in a separate unidirectional test with the NAS TXing and the DL20+Mikrotik RXing, ruling out it being due to the load from the bidirectional test:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   505 MBytes  4.24 Gbits/sec   42    617 KBytes
[  5]   1.00-2.00   sec   540 MBytes  4.53 Gbits/sec   62    609 KBytes
[  5]   2.00-3.00   sec   545 MBytes  4.57 Gbits/sec   66    658 KBytes
[  5]   3.00-4.00   sec   545 MBytes  4.57 Gbits/sec   52    609 KBytes
[  5]   4.00-5.00   sec   544 MBytes  4.56 Gbits/sec   79    629 KBytes
[  5]   5.00-6.00   sec   546 MBytes  4.58 Gbits/sec   75    626 KBytes
[  5]   6.00-7.00   sec   546 MBytes  4.58 Gbits/sec   40    683 KBytes
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec   63    624 KBytes
[  5]   8.00-9.00   sec   548 MBytes  4.59 Gbits/sec   48    598 KBytes
[  5]   9.00-10.00  sec   542 MBytes  4.55 Gbits/sec   73    508 KBytes

Final Thoughts

The CCR2004-1G-2XS-PCIe seems to be a solid card for use in small deployments, especially single-server co-location setups where something like a dedicated router may not be practical. It’s a fully featured router and is cheaper than many of the 25Gbps capable cards that are available new on the market. It has no or limited support in VMware, so if your infrastructure uses VMware it’s solidly a no-go. Drivers should be available in most of the latest releases of most major Linux distros, and it works with the current releases of Vyos, pfSense and OPNsense (if you want to put a router in your router). For the $200 price point Mikrotik has set for it, it’s hard to pass. The cheapest and most basic dual-port SFP28 25Gbps card currently available on CDW, the Supermicro AOC-S25G-b2S, starts at $313 USD, more than $100 more than Mikrotik’s CCR2004-1G-2XS-PCIe. It fits my niche perfectly, and in the next few months I’m going to be deploying my CCR2004-1G-2XS-PCIe as well as my HP ProLiant DL20 G9 to a co-location facility as part of another upcoming project.

The card also lacks SR-IOV support, so if that’s something you need, you’re unfortunately out of luck.

If you can manage to find a retailer that carries it (seriously, has this been officially released? almost no one has it) and have a use-case that it fits (or just want a really cheap 25Gbps NIC and don’t care about the routing features), it’s definitely worth it.

an addendum has been posted, read it here