Quantcast

How scalable is Yarp name server?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How scalable is Yarp name server?

Truong Nghiem
Hello all,

I've been using Yarp in C++ for quite some time, and it has been working very well. Recently, when I develop a rather large-scale system, with over 400 ports (almost 100 processes, each has between 2 and dozens Yarp ports), there is an issue with removing the ports when the programs exits.  Typically, when a process ends, Yarp (the yarp::os::Network object) will remove all the ports opened by this process in its destructor. However, in my system, many processes fail to remove all of their ports when they end (some ports of a process can be removed but others can't). Specifically, the error I got from a process looked like this:
yarp: Removing output from /powernet/user13_1/y to /powernet/bus13/Vfrom1
yarp: No connection to nameserver
yarp: *** try running: yarp detect ***
yarp: Removing input from /powernet/_smn_/user13_1 to /powernet/user13_1/_gc_
yarp: Removing output from /powernet/user13_1/_gc_ to /powernet/_smn_/_gc_

As you can see, this process opened 2 ports (/powernet/user13_1/_gc_ and /powernet/user13_1/y) but only the first could be removed. The second couldn't be removed because of "no connection to nameserver" (although I checked carefully and the connection seemed fine, no problem with Yarp configuration). This problem didn't happen with smaller scale systems I tried. The problem also seems to happen randomly, in the sense that the set of ports which couldn't be removed changed every time I tried.

So I suspect the number of ports/connections may be too large for the Yarp nameserver to handle? How scalable is Yarp nameserver?
P.S. I am using Yarp stable 2.3.64.

Thank you.
Truong


------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How scalable is Yarp name server?

Truong Nghiem
Update on this issue:
I think the problem was that when all the processes exit at almost the same time, the number of requests to close ports/connections sent to the nameserver was too large in a very short duration, overloading the nameserver (like a DOS attack). I made a temporary fix for the issue by: a) turning the Yarp verbosity level to -1; b) insert a random delay before exit into each process, so they won't all request to close ports at the same time. This is to reduce the load on the nameserver. The fix seems to work fine so far.

My question is then whether there is a nicer, more stable way to fix this issue. For example, when the Network object is destroyed and closes ports, can it try several times if the connection to the nameserver fails (with a small delay between trials)?

Thanks.
Truong



On Thursday, September 3, 2015 6:35 PM, Truong Nghiem <[hidden email]> wrote:


Hello all,

I've been using Yarp in C++ for quite some time, and it has been working very well. Recently, when I develop a rather large-scale system, with over 400 ports (almost 100 processes, each has between 2 and dozens Yarp ports), there is an issue with removing the ports when the programs exits.  Typically, when a process ends, Yarp (the yarp::os::Network object) will remove all the ports opened by this process in its destructor. However, in my system, many processes fail to remove all of their ports when they end (some ports of a process can be removed but others can't). Specifically, the error I got from a process looked like this:
yarp: Removing output from /powernet/user13_1/y to /powernet/bus13/Vfrom1
yarp: No connection to nameserver
yarp: *** try running: yarp detect ***
yarp: Removing input from /powernet/_smn_/user13_1 to /powernet/user13_1/_gc_
yarp: Removing output from /powernet/user13_1/_gc_ to /powernet/_smn_/_gc_

As you can see, this process opened 2 ports (/powernet/user13_1/_gc_ and /powernet/user13_1/y) but only the first could be removed. The second couldn't be removed because of "no connection to nameserver" (although I checked carefully and the connection seemed fine, no problem with Yarp configuration). This problem didn't happen with smaller scale systems I tried. The problem also seems to happen randomly, in the sense that the set of ports which couldn't be removed changed every time I tried.

So I suspect the number of ports/connections may be too large for the Yarp nameserver to handle? How scalable is Yarp nameserver?
P.S. I am using Yarp stable 2.3.64.

Thank you.
Truong




------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How scalable is Yarp name server?

Lorenzo Natale-2
In reply to this post by Truong Nghiem

Dear Truong,

From your description I suspect there is a glitch in how you close the ports.

 

Background:

Ports names are unregistered by the ports themselves and not by the yarp::os::Network. Name unregistration happen explicitly when you call Port::close() or implicitly in the destructor.

 

My hypothesis:

The problem you describe can happen for example if the yarp::os::Network goes out of scope (or is explicitly uninitialized) before ports call close() (or go out of scope). Are you sure the lifecycle of all these objects is such that destructions of all ports (or explicit calls to close) happen before yarp::os::Network goes out of scope?

 

Lorenzo

 

From: Truong Nghiem [mailto:[hidden email]]
Sent: giovedì 3 settembre 2015 18:35
To: [hidden email]
Subject: [rc-hackers] How scalable is Yarp name server?

 

Hello all,

 

I've been using Yarp in C++ for quite some time, and it has been working very well. Recently, when I develop a rather large-scale system, with over 400 ports (almost 100 processes, each has between 2 and dozens Yarp ports), there is an issue with removing the ports when the programs exits.  Typically, when a process ends, Yarp (the yarp::os::Network object) will remove all the ports opened by this process in its destructor. However, in my system, many processes fail to remove all of their ports when they end (some ports of a process can be removed but others can't). Specifically, the error I got from a process looked like this:

yarp: Removing output from /powernet/user13_1/y to /powernet/bus13/Vfrom1
yarp: No connection to nameserver
yarp: *** try running: yarp detect ***
yarp: Removing input from /powernet/_smn_/user13_1 to /powernet/user13_1/_gc_
yarp: Removing output from /powernet/user13_1/_gc_ to /powernet/_smn_/_gc_

 

As you can see, this process opened 2 ports (/powernet/user13_1/_gc_ and /powernet/user13_1/y) but only the first could be removed. The second couldn't be removed because of "no connection to nameserver" (although I checked carefully and the connection seemed fine, no problem with Yarp configuration). This problem didn't happen with smaller scale systems I tried. The problem also seems to happen randomly, in the sense that the set of ports which couldn't be removed changed every time I tried.

 

So I suspect the number of ports/connections may be too large for the Yarp nameserver to handle? How scalable is Yarp nameserver?

P.S. I am using Yarp stable 2.3.64.

 

Thank you.

Truong

 


------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How scalable is Yarp name server?

Lorenzo Natale-2
In reply to this post by Truong Nghiem

Sorry I should have read this update before replying. Yes your suggestion is plausible; we can investigate it further. Can you open an issue on github?

Lorenzo

 

From: Truong Nghiem [mailto:[hidden email]]
Sent: venerdì 4 settembre 2015 10:37
To: [hidden email]
Subject: Re: [rc-hackers] How scalable is Yarp name server?

 

Update on this issue:

I think the problem was that when all the processes exit at almost the same time, the number of requests to close ports/connections sent to the nameserver was too large in a very short duration, overloading the nameserver (like a DOS attack). I made a temporary fix for the issue by: a) turning the Yarp verbosity level to -1; b) insert a random delay before exit into each process, so they won't all request to close ports at the same time. This is to reduce the load on the nameserver. The fix seems to work fine so far.



My question is then whether there is a nicer, more stable way to fix this issue. For example, when the Network object is destroyed and closes ports, can it try several times if the connection to the nameserver fails (with a small delay between trials)?



Thanks.

Truong

 

 

On Thursday, September 3, 2015 6:35 PM, Truong Nghiem <[hidden email]> wrote:

 

Hello all,

 

I've been using Yarp in C++ for quite some time, and it has been working very well. Recently, when I develop a rather large-scale system, with over 400 ports (almost 100 processes, each has between 2 and dozens Yarp ports), there is an issue with removing the ports when the programs exits.  Typically, when a process ends, Yarp (the yarp::os::Network object) will remove all the ports opened by this process in its destructor. However, in my system, many processes fail to remove all of their ports when they end (some ports of a process can be removed but others can't). Specifically, the error I got from a process looked like this:

yarp: Removing output from /powernet/user13_1/y to /powernet/bus13/Vfrom1
yarp: No connection to nameserver
yarp: *** try running: yarp detect ***
yarp: Removing input from /powernet/_smn_/user13_1 to /powernet/user13_1/_gc_
yarp: Removing output from /powernet/user13_1/_gc_ to /powernet/_smn_/_gc_

 

As you can see, this process opened 2 ports (/powernet/user13_1/_gc_ and /powernet/user13_1/y) but only the first could be removed. The second couldn't be removed because of "no connection to nameserver" (although I checked carefully and the connection seemed fine, no problem with Yarp configuration). This problem didn't happen with smaller scale systems I tried. The problem also seems to happen randomly, in the sense that the set of ports which couldn't be removed changed every time I tried.

 

So I suspect the number of ports/connections may be too large for the Yarp nameserver to handle? How scalable is Yarp nameserver?

P.S. I am using Yarp stable 2.3.64.

 

Thank you.

Truong

 

 


------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How scalable is Yarp name server?

Truong Nghiem
Thank you for the response.  I opened a new issue on github.  I also added a comment to that issue for a related problem I faced this morning. My system has hundreds of ports. At some point, when I request to connect two ports, the request always fails even though both ports exist. The error message is the same: No connection to nameserver (though the nameserver is still running fine). This error happened to the same ports every time I tried. A verbose message from Yarp can be seen in the comment on github.

A practical question: is there a limit on the number of ports or connections a nameserver can manage? For example does the nameserver set a maximum number of ports / connections on its lists?

Thanks,
Truong



On Friday, September 4, 2015 11:14 AM, Lorenzo Natale <[hidden email]> wrote:


Sorry I should have read this update before replying. Yes your suggestion is plausible; we can investigate it further. Can you open an issue on github?
Lorenzo
From: Truong Nghiem [mailto:[hidden email]]
Sent: venerdì 4 settembre 2015 10:37
To: [hidden email]
Subject: Re: [rc-hackers] How scalable is Yarp name server?
 
Update on this issue:
I think the problem was that when all the processes exit at almost the same time, the number of requests to close ports/connections sent to the nameserver was too large in a very short duration, overloading the nameserver (like a DOS attack). I made a temporary fix for the issue by: a) turning the Yarp verbosity level to -1; b) insert a random delay before exit into each process, so they won't all request to close ports at the same time. This is to reduce the load on the nameserver. The fix seems to work fine so far.


My question is then whether there is a nicer, more stable way to fix this issue. For example, when the Network object is destroyed and closes ports, can it try several times if the connection to the nameserver fails (with a small delay between trials)?


Thanks.
Truong
 
 
On Thursday, September 3, 2015 6:35 PM, Truong Nghiem <[hidden email]> wrote:
 
Hello all,
 
I've been using Yarp in C++ for quite some time, and it has been working very well. Recently, when I develop a rather large-scale system, with over 400 ports (almost 100 processes, each has between 2 and dozens Yarp ports), there is an issue with removing the ports when the programs exits.  Typically, when a process ends, Yarp (the yarp::os::Network object) will remove all the ports opened by this process in its destructor. However, in my system, many processes fail to remove all of their ports when they end (some ports of a process can be removed but others can't). Specifically, the error I got from a process looked like this:
yarp: Removing output from /powernet/user13_1/y to /powernet/bus13/Vfrom1
yarp: No connection to nameserver
yarp: *** try running: yarp detect ***
yarp: Removing input from /powernet/_smn_/user13_1 to /powernet/user13_1/_gc_
yarp: Removing output from /powernet/user13_1/_gc_ to /powernet/_smn_/_gc_
 
As you can see, this process opened 2 ports (/powernet/user13_1/_gc_ and /powernet/user13_1/y) but only the first could be removed. The second couldn't be removed because of "no connection to nameserver" (although I checked carefully and the connection seemed fine, no problem with Yarp configuration). This problem didn't happen with smaller scale systems I tried. The problem also seems to happen randomly, in the sense that the set of ports which couldn't be removed changed every time I tried.
 
So I suspect the number of ports/connections may be too large for the Yarp nameserver to handle? How scalable is Yarp nameserver?
P.S. I am using Yarp stable 2.3.64.
 
Thank you.
Truong
 
 



------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Loading...