Xenial - OpenvSwitch with DPDK binding to 10G NIC, not working...
Martinx - ジェームズ
thiagocmartinsc at gmail.com
Tue Mar 22 15:12:18 UTC 2016
Hi Christian,
Thank you for your support, I really appreciate it!
So, in the end of the day, even if a 10G NIC is running with a
"DPDK-Compatible Driver", this does NOT means that it will work with
OpenvSwtich!
There are only a very few limited number of NIC cards that are compatible
with DPDK + PMD, which are required for OpenvSwitch+DPDK to function, if
the DPDK Driver doesn't have PMD enabled, no deal. Am I right?
This information MUST be clear on Xenial release notes! Otherwise, this
will lead to confusion, because until now, we're saying that if a NIC is
bonded to a DPDK-Compatible driver, it WILL work with OVS+DPDK, which is
NOT the case.
Oh, thanks for replying on my bug report too! You're very kind... ^_^
Best,
Thiago
On 21 March 2016 at 04:21, Christian Ehrhardt <
christian.ehrhardt at canonical.com> wrote:
> Thanks Thiago,
> I'll look into doability to enable that - even not thinking being after FF
> there are pro's and con's to be considered.
> We will go on with communication regarding this issue on that bug then.
> Yet feel free to catch me on IRC as I suggested if you want.
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>
> On Sat, Mar 19, 2016 at 8:01 AM, Martinx - ジェームズ <
> thiagocmartinsc at gmail.com> wrote:
>
>> Hey Christian!
>>
>> I think I found the problem! I just filled a bug report:
>>
>> --
>> Many PMD driver options are disabled, including "BNX2X"!
>>
>> https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1559408
>> --
>>
>> Let me know about what do you think!
>>
>> Cheers!
>> Thiago
>>
>> On 19 March 2016 at 02:49, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> wrote:
>>
>>> Oh, I forgot to mention that my /etc/dpdk/dpdk.conf have:
>>>
>>> ---
>>> NR_1G_PAGES=2
>>> ---
>>>
>>> From what I'm seeing, my OpenvSwitch-DPDK is not using / seeing PMD.
>>>
>>> So, I'm researching about this but, I'll talk with you on Monday...
>>>
>>> Have a nice weekend Christian!
>>>
>>> Thanks,
>>> Thiago
>>>
>>> On 18 March 2016 at 13:16, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>> wrote:
>>>
>>>> Hello Christian!
>>>>
>>>> Lets make a checklist...
>>>>
>>>>
>>>> 1- Packages "dpdk and openvswitch-switch-dpdk", installed;
>>>>
>>>>
>>>> 2- /etc/default/grub have:
>>>>
>>>> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1GB
>>>> hugepagesz=1G hugepages=4"
>>>> GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 BOOTIF=01-b0-83-fe-d7-12-fc"
>>>>
>>>> Linux CMD Line:
>>>>
>>>> Command line: BOOT_IMAGE=/vmlinuz-4.4.0-13-generic
>>>> root=/dev/mapper/xenial-1--vg-root ro console=ttyS0,115200n8
>>>> BOOTIF=01-b0-83-fe-d7-12-fc quiet splash default_hugepagesz=1GB
>>>> hugepagesz=1G hugepages=4 vt.handoff=7
>>>>
>>>>
>>>> 3- /etc/dpdk/interfaces have:
>>>>
>>>> pci 0000:01:00.0 uio_pci_generic
>>>> pci 0000:01:00.1 uio_pci_generic
>>>>
>>>>
>>>> 4- /etc/default/openvswitch-switch have:
>>>>
>>>> DPDK_OPTS='--dpdk -c 0x1 -n 4 --socket-mem 1024,0'
>>>>
>>>>
>>>> 5- /etc/alternatives/ovs-vswitchd: symbolic link to
>>>> /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
>>>>
>>>>
>>>> 6- My CPU supports 1G Hugepages (it have the flag: pdpe1gb)
>>>>
>>>>
>>>> 7- I am running most recent Xenial (4.4.0-13-generic), fully upgraded
>>>> with "apt update ; apt full-upgrade ; reboot"
>>>>
>>>>
>>>> 8- Huge pages on /proc/meminfo
>>>>
>>>> # grep Huge /proc/meminfo
>>>> AnonHugePages: 26624 kB
>>>> HugePages_Total: 2
>>>> HugePages_Free: 1
>>>> HugePages_Rsvd: 0
>>>> HugePages_Surp: 0
>>>> Hugepagesize: 1048576 kB
>>>>
>>>>
>>>>
>>>> 9- OVS process is running with DPDK options:
>>>>
>>>> ---
>>>> ps xfa | grep dpdk
>>>> 2634 ? S<s 0:00 ovs-vswit... --dpdk -c 0x1 -n 4
>>>> --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer
>>>> -vsyslog:err -vfile:info --mlockall --no-chdir
>>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>>> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
>>>> 2635 ? S<Ll 0:00 \_ ovs-vswitchd --dpdk -c 0x1 -n 4
>>>> --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer
>>>> -vsyslog:err -vfile:info --mlockall --no-chdir
>>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>>> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
>>>> ---
>>>>
>>>>
>>>> 10- DPDK-Compatible Driver activated
>>>>
>>>> ---
>>>> dpdk_nic_bind --status
>>>>
>>>> Network devices using DPDK-compatible driver
>>>> ============================================
>>>> 0000:01:00.0 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
>>>> drv=uio_pci_generic unused=bnx2x
>>>> 0000:01:00.1 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
>>>> drv=uio_pci_generic unused=bnx2x
>>>> ---
>>>>
>>>>
>>>> 11- OVS log have no "port 0" / "port 1"
>>>>
>>>> # service openvswitch-switch restart
>>>> # journalctl | grep -i ovs:
>>>>
>>>> https://paste.ubuntu.com/15415651/
>>>>
>>>> Also, I'm not seeing any "PMD" lines on my OVS logs!
>>>>
>>>> NOTE: after running: "service openvswitch-switch restart", it still
>>>> have dpdk options (step 9 above still valid)
>>>>
>>>>
>>>> 12- The, OVS commands also fails...
>>>>
>>>> # ovs-vsctl show
>>>> b70c1e0a-20d7-4bdb-98db-467330b72d07
>>>> ovs_version: "2.5.0"
>>>>
>>>> # ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>>> # ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>>>> ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd
>>>> log for details.
>>>>
>>>> ---
>>>>
>>>> I just don't know what I am missing, I already checked the
>>>> configurations dozens of times... At least, I'm not alone seeing this
>>>> problems, since there are more users on Intel's forum with same issue...
>>>>
>>>> Thank you for your help! I'm going to IRC now.
>>>>
>>>> Best,
>>>> Thiago
>>>>
>>>> On 18 March 2016 at 10:22, Christian Ehrhardt <
>>>> christian.ehrhardt at canonical.com> wrote:
>>>>
>>>>> Hi Martin,
>>>>> didn't find you on IRC today.
>>>>> You might try to catch me later or on Monday if you like for faster
>>>>> checks and comparison.
>>>>>
>>>>> I once more tried the ovs-dpdk setup.
>>>>> While I'm blocked later on on other things I clearly can pass the
>>>>> stage you are hanging on.
>>>>>
>>>>> I played a bit around and could cause the error you had by:
>>>>> 1. not having any card on a dpdk compatible driver
>>>>> 2. not enabling dpdk in ovs at all
>>>>>
>>>>> Could you please check your journalctl when starting openvswitch - it
>>>>> should have a section like the following per card (ovs-)dpdk initializes:
>>>>>
>>>>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>>>>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086
>>>>> deviceID=0x1528
>>>>> horsea ovs-ctl[2155]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
>>>>> horsea ovs-ctl[2155]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd
>>>>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40204000
>>>>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40404000
>>>>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>>>>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): port 0
>>>>> vendorID=0x8086 deviceID=0x1528
>>>>>
>>>>> You see "port 0" that will be what is "dpdk0" later on.
>>>>>
>>>>> When I e.g. have two devices in ugb_pci_generic and start OVS I get
>>>>> one for port 0 and one for port 1.
>>>>> Then I can later on use dpdk0 and dpdk1, but not e.g. dpdk2
>>>>>
>>>>>
>>>>>
>>>>> Christian Ehrhardt
>>>>> Software Engineer, Ubuntu Server
>>>>> Canonical Ltd
>>>>>
>>>>> On Fri, Mar 11, 2016 at 10:17 PM, Martinx - ジェームズ <
>>>>> thiagocmartinsc at gmail.com> wrote:
>>>>>
>>>>>> Hey Christian,
>>>>>>
>>>>>> Nice job on fixing many DPDK package bugs! :-D
>>>>>>
>>>>>> Replying inline below...
>>>>>>
>>>>>> On 9 March 2016 at 04:17, Christian Ehrhardt <
>>>>>> christian.ehrhardt at canonical.com> wrote:
>>>>>>
>>>>>>> Thanks Jörg for making Martin and me aware.
>>>>>>> It is kind of a close call, as much of that are new features, but
>>>>>>> you are right - so I'll be moving this to the Ubuntu-sever mailing list.
>>>>>>>
>>>>>>> @Martin:
>>>>>>> tl;dr - I suggest the following:
>>>>>>> - you let me know if anything more comes back from the Intel forum
>>>>>>> post
>>>>>>>
>>>>>>
>>>>>> Absolutely, I'll...
>>>>>>
>>>>>>
>>>>>>> - short Term I'd ask you to reach out on IRC to me on #ubuntu-server
>>>>>>> - my nick is "cpaelzer" (maybe we find something in an interactive session)
>>>>>>>
>>>>>>
>>>>>> Sounds perfect! I'm there now, "tmartins" nick...
>>>>>>
>>>>>>
>>>>>>> - somewhen in the next weeks I'll be back at openvswitch-dpdk. I'd
>>>>>>> invite you to a IRC session or a hangout then (let me know what you would
>>>>>>> prefer and I contact you once I'm ready)
>>>>>>>
>>>>>>
>>>>>> That would be awesome!
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Details:
>>>>>>> The vfio-pci fix (and several others) are just waiting for the
>>>>>>> upload to happen, if I don't run into any unexpected running blockers that
>>>>>>> should be today.
>>>>>>>
>>>>>>
>>>>>> Verified, it is working now.
>>>>>>
>>>>>>
>>>>>>> Please let me know if you run into anything else with DPDK as I want
>>>>>>> to clear as much as possible before release.
>>>>>>>
>>>>>>
>>>>>> After latest release on Xenial, it looks very good! Both UIO and VFIO
>>>>>> works without effort.
>>>>>>
>>>>>> I can easily switch between UIO and VFIO, that DPDK shows the NICs
>>>>>> being used as "DPDK-Compatible drivers".
>>>>>>
>>>>>> Only OpenvSwitch+DPDK doesn't see any "dpdk0 | dpdk1" devices but, I
>>>>>> know you guys are working into this...
>>>>>>
>>>>>>
>>>>>>> Please feel free to just directly contact me about it - I'd suggest
>>>>>>> IRC if that work for you.
>>>>>>> We can move things to mailing lists whenever needed/applicable then.
>>>>>>>
>>>>>>
>>>>>> Sounds great!
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Since last week I took a step back and work on more "low level" dpdk
>>>>>>> testing and issues than the full setup with openvswitch-dpdk.
>>>>>>> That helps the overall quality, but means my environment is
>>>>>>> currently not useful to reproduce your case without too much
>>>>>>> reconfiguration.
>>>>>>>
>>>>>>
>>>>>> That sounds like a good plan, I totally understand that... =)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I highly appreciate to have someone in the Community working with me
>>>>>>> on it.
>>>>>>>
>>>>>>
>>>>>> You can count on me! I'm a big fan of Ubuntu...
>>>>>>
>>>>>> I met Mark in Vancouver on OpenStack summit, he is a great guy and it
>>>>>> is easier now to see why Ubuntu is so cool! A lot of amazing people working
>>>>>> here! ^_^
>>>>>>
>>>>>> So I wanted to ask if you could be patient for a few days, once my
>>>>>>> Env is working for openvswitch-dpdk again I'd contact you and we could work
>>>>>>> together comparing what break in your setup.
>>>>>>>
>>>>>>
>>>>>> Of course! I'll be the beta tester. ;-)
>>>>>>
>>>>>> Christian Ehrhardt
>>>>>>> Software Engineer, Ubuntu Server
>>>>>>> Canonical Ltd
>>>>>>>
>>>>>>
>>>>>> Just for the record, we moved this thread to Ubuntu Server, here is
>>>>>> the previous thread on Ubuntu Devel, so, people on Ubuntu Server will
>>>>>> understand what we're talking about:
>>>>>>
>>>>>>
>>>>>> https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-March/016287.html
>>>>>>
>>>>>> Cheers!
>>>>>> Thiago
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20160322/16c8527d/attachment.html>
More information about the ubuntu-server
mailing list