[SDL_net] Limits |
[SDL_net] Limits |
Simon
Guest
|
That's interesting. My application suffers from the fact many connections may arrive at once on a host. I guess it could be easy to hack SDL_net's source to increase this limit to reach at least the OS's limit.
On TCP, packets do get dropped, but it is the OS's job to retransmit them. This is called congestion. On a very busy host, when lots of packets arrive in disorder, the host has to hold all that unordered data until the missing pieces arrive so the server can reorder the stream. The amount of data the server can hold unordered is the TCP window. But because the window is set for "the perfect world" and in our world windows do get full and use a lot of resources. Many protocols have implemented a congestion-window on the sender side, that is to send data at the maximum rate until packets start being dropped (still talking about TCP here) then the congestion-window is lowered. This just means the client understands the server window adverstised may not equal the real-time window on the server.
Exactly, but my application needs to be aware of all this, I will certainly have to use UDP to catch those problems and deal with them my way.
My problem is the server of my application will get lots to process every frame. I may expect some situations (maybe not so rare) where every frame will take several seconds, maybe even more. During this time, lots of trouble can happen as you may understand. Design-wise, there is not much we can do about this except balance all that load onto a different server, but this is just "distributing" the problem and not solving it.
I guess this is done OS-dependantly. To my application, it doesn't matter much, the fact is: 1) packets will get dropped, no matter which, 2) those dropped packets will be resent 3) packets will be resent until succesfully received. Basically, my application will be compiled for different OS and I will not be able to avoid the OS-dependent characteristics. The best I can do is to make the application aware of the limits of its system. Thanks, Simon |
|||||||||||||||||||||
|
[SDL_net] Limits |
Josiah Manson
Guest
|
Why not have seperate threads dealing with connections and read/writes? They could put data into your own buffers, and you wouldn't have to worry about OS limits. |
|||||||||||||||
|
[SDL_net] Limits |
Nuage
Guest
|
Hello,
SDL is geat for all IO management. But when I had to design a complex client-server network system, I did just rewrote my own one. Today all OSs understand the "unix-netowrk-C-functions", which are actualy used in SDL_net, except max os classic... Well, as far as I know: -win32 -liunx -maxosx do. And it's enough for me. I love OpenBSD, and I guess it would work with it too, but haven't tried. So I would simply advise you to use the direct unix C calls. There is just with win32, you have to call some init at the beginning, but that's far from complex. Well, I did rewrote my network abstraction because I improve some code "cleaness" because I could do it into C++ too. At the end I found it did worth. Maybe if you write C it doesn't worth. Just my toughts. Nuage Simon wrote:
|
|||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
Simon <simon.xhz at gmail.com> wrote:
i solved this once and for all for me with a little socket class. i spawn a thread for each connection, each with its own receive buffer. the network protocoll consists of an 4 byte header and an up to 4GB body. the socket is opened in blocking mode. first i read the 4 byte header, then realloc the buffer if needed and finally read the body. when the whole packet has been read, i call back the application through a virtual method. (can also be implemeted via a normal c callback function, does not matter.) i saved myself from all the packet dropping/disordering by using TCP though. this design scales very well with minimum latency. best regards ... clemens |
|||||||||||||||
|
[SDL_net] Limits |
Torsten Giebl
Guest
|
Hello !
Would it be possible to post this class here ? CU |
|||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
"Torsten Giebl" <wizard at syntheticsw.com> wrote:
my very first C version is available under GPL already at www.1541.org it's called SNL (Simple Network Layer). the C++ redesign, which is somehow saver to use and has simple UDP support, is part of a library i did for my current employer. i will try to set the whole library free anyway, but i have to check with him first. i will rip the socket class from the complete lib though, as it is based on SNL anyway. give me a few minutes to package it ... clemens |
|||||||||||||||
|
[SDL_net] Limits |
Ryan C. Gordon
Guest
|
Just to play devil's advocate (and before this gets too far from discussion of SDL_net): I assume that four-byte header is an unsigned int that reports the amount of data that will be forthcoming, so you can realloc if needed. What happens when some jerk sends you a packet that starts with 0x4000000, and then feeds you a gigabyte of data? If he does this with multiple connections at once, does the server just sort of fall over when malloc() eventually fails, or does it bring the whole box to its knees swapping before that happens? You might want to put an upper limit on that currently-4GB maximum, and assume it's malicious if it's more than, say, a kilobyte, and drop the connection. (If a kilobyte is too small, fine, 100 kilobytes, whatever.) Network programming has so many nasty what-if situations. :) --ryan. |
|||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
"Torsten Giebl" <wizard at syntheticsw.com> wrote:
i have uploaded the socket class to my site: http://www.1541.org/public/socket-0.0.1.tar.gz the archive contains the socket.h and socket.cpp files and basic demonstration server and client code (also for UDP). the socket.h file is doxygen documented, so you can generate the API documentation if you have doxygen installed. after untaring, just cd into the socket dir and build with make. have fun ... clemens |
|||||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
"Ryan C. Gordon" <icculus at icculus.org> wrote:
right, for the time being, i'm even unsure if its int or unsigned. 2GB is already more i could handle anyway.
i thought about this (of course ), but as this beast was used internally only so far, and clients had to authenticate with predistributed keys anyway, there was no need to handle this. authentication and encryption is thus missing in the archive i uploaded today. if there is demand for it and i will get permission to release the rest of the lib this will be addressed.
for now this is left as an exercise to the fellow reader. ;-)
indeed. thank you very much for pointing this out. i should have put a warning in the readme that the socket class "as is" could be a target for DOS attacks. anyway, currently i'm not even 100% sure the lib is without a race. it has proven itself stable in all tests and never failed so far, though. best regards ... clemens |
|||||||||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
You are avoiding the problem with this solution. I would like to know the problem and deal with it. Besides, using thread in high demand servers is not recommended. Thanks, Simon |
|||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
Can you tell me how you can handle more than 2000 connections to you
server, without the need for maxing out the amount of RAM in the PC?
|
|||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
True, that's why I was asking about the limits in the first place... it seems nobody ever tried to find vulnerabilities in SDL-based code, or are there anybody who tried? Anyway, I will be working on a first draft soon and I will possibly try to crash it all ways possible and find the limits the hard way. I wonder how the base library is strong against buffer overflows, etc... All this to avoid DoS. Thanks, Simon |
|||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
Thanks Nuage, That's very interesting... do those libs support more than SDL_net? I guess so. Would I be able to do ICMP communications too? Thanks, Simon |
|||||||||||||
|
[SDL_net] Limits |
IraqiGeek
Guest
|
On Sunday, January 08, 2006 1:38 AM GMT,
Simon <simon.xhz at gmail.com> wrote:
forget about maxing memory, how the heck is he able to handle up to 4GB body the MTU is 1500 bytes on LAN (8KB if you use jumbo frames on GbE), not to count all the TCP overhead, and still maintain "minimum latency"
Regards, IraqiGeek www.iraqigeek.com |
|||||||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
Simon <simon.xhz at gmail.com> wrote:
i don't think there are any, at least not in the linux/win32 codepath. SDL_net is a rather thin layer around bind/accept/connect/send/receive so it does no buffering or threading on its own which is most error prone in network code.
see above. best regards ... clemens |
|||||||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
"IraqiGeek" <geek at IraqiGeek.com> wrote:
i don't see why 2000 should be the limit in any way. yes, if you expect to have really many simulatanious connections to the server all the time (tens or hundreds of thousends), another solution might be better because each connection means an additional thread with its seperate process context, stack aso.
how much data you send is up to you you, it's just that the library does not force any unnecessary low limits on you. minimum latency means your application gets the data as soon as it is available (completely). of course there are other ways to implement this (select) but i prefered to use the blocking thread method. having only thread is nice but then, if you have high traffic on your server and packets get bigger (fragmented) and thus read() does not return the whole packet (of your higher level protokoll) at once (what you never can rely on anyway), things start to get complicated. i wanted two things from my network lib: packets should always be complete when deliverd and receive buffers should not be memcopied. best regards ... clemens |
|||||||||||||||||
|
[SDL_net] Limits |
Ryan C. Gordon
Guest
|
I doubt it, but it's really a very thin wrapper over BSD sockets (unless you count the OpenTransport code for MacOS 9)...frankly, if this is a huge concern for you, I'd use BSD sockets directly. --ryan. |
|||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
Security is one of my main concerns in this project, but it's not prioritary for the development. I'll probably keep working with SDL_net and may switch over to those BSD sockets. They seem to do all I need more directly. Thanks, Simon |
|||||||||||||
|
[SDL_net] Limits |
Simon
Guest
|
IraqiGeek wrote:
I agree, this large buffer can never work on a high end server. I've checked a couple of protocols and the best way I found to do my project, is to use a combination of ICMP, TCP and UDP. All of them will have small buffers since I prefer to drop information rather than being DoS or maybe have to ignore more connections. Thanks, Simon |
|||||||||||||||
|
[SDL_net] Limits |
Clemens Kirchgatterer
Guest
|
Simon <simon.xhz at gmail.com> wrote:
without wanting to sound rude, i don't think you have understod, how this works. initial buffer size is 4kb. if you do not send larger packets than 4kb, the buffer size will never increase to more. even if somebody sends a broken header, there is no problem yet, because mallocing a large buffer without using it, will not claim more memory. putting an upper limit of buffer size, depending on your high level protocol is trivial.
sounds really complicated and i can not imaging how ICMP could be useful here, but who knows. clemens |
|||||||||||||||||||
|